Deadlock in A-MQ broker under load, when timed message expiry is in use

Solution Verified - Updated -

Issue

Under conditions of high load, in a set-up that uses timed message expiry, the broker stops producing or consuming messages. A JVM thread dump shows the following two characteristic threads, where one is waiting store a message, and the other is waiting to expire a message:

"ActiveMQ Transport: tcp:///10.80.85.225:43386@61616" daemon prio=10 tid=0x000000001c3df800 nid=0x50b4 waiting for monitor entry [0x00002b53888ad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
    - waiting to lock <0x00000000a3d1de28> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
    at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)
    - locked <0x00000000a3d20c10> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
    at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1795)
        ...
"ActiveMQ BrokerService.worker.1":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000000a3d20bc0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        ...
    at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1781)
    at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
        ... 

Environment

  • Red Hat JBoss A-MQ
    • 6.2.1 before Rollup 4
    • 6.3 before Rollup 3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.