Interesting ActiveMQ Behaviour for 3-node Cluster Deployment in AKS

Latest response

Hello all,

I'm looking to get a bit of assistance here with respect to ActiveMQ behaviour. I'm seeing something odd during deployment of my app to an AKS cluster hosted on Azure. The version of Kubernetes is 1.25.15. I am using JBoss EAP 7.3.10.

Issue:
In short, I am seeing some inconsistent connection-related behaviour between my pods 0, 1, and 2 during a deployment. I'm deploying my pods as a StatefulSet.

The first pod that comes up is depl-0 (master node). After depl-0 has passed the health checks, depl-1 spins up. This process does not produce any connection errors. depl-0 happily waits for depl-1 to spin up, and complete it's health check, and complete the node rebalancing. The issue is when depl-2 spins up. When that occurs, depl-0 and depl-1 start screaming about connections issue to depl-2. ONLY after depl-2 is up and fully running do these errors stop coming in.

I don't believe the issue is related to the AKS or Azure related because my thought process is we'd be seeing the same connection error be thrown while depl-0 waits for depl-1.

The errors are ActiveMQ related. I'm a bit lost on why the wait between depl-0 and depl-1 is fine but with 2 nodes in the cluster, waiting for the 3rd node to come up and rebalance, there are connection issues.

 [exec] 2023-11-28 03:26:08.825+0000 WARN  [org.apache.activemq.artemis.core.server:1041] {} (Thread-43 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@391a1a77)) AMQ224091: Bridge ClusterConnectionBridge@5d68abf7 [name=$.artemis.internal.sf.avatodemo-avato.5142c47c-8d89-11ee-a3a3-ca07b34b3ff4, queue=QueueImpl[name=$.artemis.internal.sf.avatodemo-avato.5142c47c-8d89-11ee-a3a3-ca07b34b3ff4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=f176d734-8d88-11ee-a20c-a29b786158af], temp=false]@53af4e2a targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@5d68abf7 [name=$.artemis.internal.sf.avatodemo-avato.5142c47c-8d89-11ee-a3a3-ca07b34b3ff4, queue=QueueImpl[name=$.artemis.internal.sf.avatodemo-avato.5142c47c-8d89-11ee-a3a3-ca07b34b3ff4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=f176d734-8d88-11ee-a20c-a29b786158af], temp=false]@53af4e2a targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=demo-avato-depl-2], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1854543031[nodeUUID=f176d734-8d88-11ee-a20c-a29b786158af, connector=TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=demo-avato-depl-0, address=jms, server=ActiveMQServerImpl::serverUUID=f176d734-8d88-11ee-a20c-a29b786158af])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?httpUpgradeEndpoint=http-acceptor&activemqServerName=default&httpUpgradeEnabled=true&port=8080&host=demo-avato-depl-2], discoveryGroupConfiguration=null]] is unable to connect to destination. Retrying
     [exec] 2023-11-28 03:26:09.327+0000 ERROR [org.apache.activemq.artemis.core.client:877] {} (Thread-62 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@391a1a77)) AMQ214016: Failed to create netty connection: java.net.UnknownHostException: demo-avato-depl-2
     [exec]     at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764)
     [exec]     at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
     [exec]     at java.net.InetAddress.getAllByName(InetAddress.java:1144)
     [exec]     at java.net.InetAddress.getAllByName(InetAddress.java:1065)
     [exec]     at java.net.InetAddress.getByName(InetAddress.java:1015)

Any advice/suggestions would be appreciated.

Thanks!

DS

Responses