Deadlock in A-MQ broker under load, when timed message expiry is in use

Solution Verified - Updated -

Issue

Under conditions of high load, in a set-up that uses timed message expiry, the broker stops producing or consuming messages. A JVM thread dump shows the following two characteristic threads, where one is waiting store a message, and the other is waiting to expire a message:

"ActiveMQ Transport: tcp:///10.80.85.225:43386@61616" daemon prio=10 tid=0x000000001c3df800 nid=0x50b4 waiting for monitor entry [0x00002b53888ad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
    - waiting to lock <0x00000000a3d1de28> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
    at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)
    - locked <0x00000000a3d20c10> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
    at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1795)
        ...
"ActiveMQ BrokerService.worker.1":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000a3d20bc0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        ...
    at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1781)
    at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
        ... 

Environment

  • Red Hat JBoss A-MQ
    • 6.2.1 before Rollup 4
    • 6.3 before Rollup 3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content