Release - Support Azure with Kubernetes Configuration

Azure team has successfully run GCZ in Local Profile, and are now trying the Kubernetes profile. They are running into challenges with the Transformer/Ignite connection in a Kubernetes environment, and are seeking assistance on next steps.

They have followed all instructions on GCZ Wiki and from official Apache Ignite / Azure documentation. At first they were having trouble getting the Ignite Cluster to start within Kubernetes, but they found patial success after changing the docker image to gridgain/community:latest.

Latest error suggests a potential issue with the Kubernetes service token, but this is not an issue we've encountered in previous implementations with IBM.

Error:

[ERROR] 2023-03-15 11:09:28.874 [tcp-client-disco-msg-worker-#4-#44] TcpDiscoverySpi - Failed to get registered addresses from IP finder (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries) [maxTimeout=0]org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:77) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:2016) ~[ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1954) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ClientImpl.joinTopology(ClientImpl.java:589) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ClientImpl.access$1000(ClientImpl.java:148) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.tryJoin(ClientImpl.java:2109) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1752) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:315) [ignite-core-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:57) [ignite-core-8.8.11.jar!/:8.8.11]Caused by: org.apache.ignite.IgniteException: Failed to retrieve Ignite pods IP addresses.        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:120) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:73) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        ... 9 moreCaused by: org.apache.ignite.IgniteException: Failed to load services account token [setAccountToken= /var/run/secrets/kubernetes.io/serviceaccount/token]        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.serviceAccountToken(KubernetesServiceAddressResolver.java:191) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:103) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:73) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        ... 9 moreCaused by: java.nio.file.NoSuchFileException: \var\run\secrets\kubernetes.io\serviceaccount\token        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:85) ~[?:?]        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) ~[?:?]        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) ~[?:?]        at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:235) ~[?:?]        at java.nio.file.Files.newByteChannel(Files.java:371) ~[?:?]        at java.nio.file.Files.newByteChannel(Files.java:422) ~[?:?]        at java.nio.file.Files.readAllBytes(Files.java:3206) ~[?:?]        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.serviceAccountToken(KubernetesServiceAddressResolver.java:189) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.internal.kubernetes.connection.KubernetesServiceAddressResolver.getServiceAddresses(KubernetesServiceAddressResolver.java:103) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:73) ~[ignite-kubernetes-8.8.11.jar!/:8.8.11]

Acceptance Criteria:

• Kubernetes Environment issue is resolved, and EPAM/Azure are able to run GCZ successfully in their environment
• Detailed steps and background are documented to provide knowledge foundation for future CSP-Kubernetes integration efforts

Update as of 4/3/2023:

• Both GCZ team and EPAM/Azure team are reaching out to IBM for clarification on their procedure for Kubernetes deployment, but correspondence from IBM has been delayed.
• Further investigation is being made into the Gridgain Image Version, and whichever other configuration modifications may be required to ensure connection is possible between Transformer and Ignite in a Kubernetes environment.

cc @nareshj @omprakash_epam @shivani_karipe