High CPU usage in ActiveMQ broker when network connector fails

Solution Verified - Updated -

Environment

  • JBoss A-MQ 6.2.0

Issue

Our ActiveMQ broker is configured with a network connector, as in one of these examples:

<networkConnectors>
  <networkConnector name="toRemote" uri="static:(failover:(tcp://otherBroker:61616)?maxReconnectAttempts=0)" />
  <networkConnector name="toRemote2" uri="masterslave:(tcp://otherMaster:61616,tcp://otherSlave:61616)" />
</networkConnectors>

Note: Only one of the above variants would be needed to run into the problem described in this article.

If the remote broker - to which the network bridge tries to connect to - is not available, the local broker will use a lot of CPU soon after startup (i.e. within a few minutes).

Resolution

Update to JBoss A-MQ 6.2.1 to get bug ENTMQ-1204 resolved.

Root Cause

Bug ENTMQ-1204.

Diagnostic Steps

Every time the broker logs an unsuccessful attempt to connect to the remote broker as in this line

ERROR | Failed to connect to [tcp://otherMaster:61616, tcp://otherSlave:61616] after: 1 attempt(s)

the CPU usage will increase.
When taking a thread dump after a few minutes, there will be multiple threads named triggerStartAsyncNetworkBridgeCreation where there should be only one per network connector definition.

These threads typically have the following call stack:

[1]
 "triggerStartAsyncNetworkBridgeCreation: remoteBroker=unconnected, localBroker= vm://localhost#14" daemon prio=10 tid=0x00007f6b78010800 nid=0x7c92 waiting on condition [0x00007f6be4738000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007818ab7d0> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
    at org.apache.activemq.network.DemandForwardingBridgeSupport.safeWaitUntilStarted(DemandForwardingBridgeSupport.java:1405)
    at org.apache.activemq.network.DemandForwardingBridgeSupport.doStartLocalAndRemoteBridges(DemandForwardingBridgeSupport.java:435)
    at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:106)
    at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:336)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

    Locked ownable synchronizers:
    - <0x00000007818ab4c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.