Deadlock in A-MQ broker under load, when timed message expiry is in use
Issue
Under conditions of high load, in a set-up that uses timed message expiry, the broker stops producing or consuming messages. A JVM thread dump shows the following two characteristic threads, where one is waiting store a message, and the other is waiting to expire a message:
"ActiveMQ Transport: tcp:///10.80.85.225:43386@61616" daemon prio=10 tid=0x000000001c3df800 nid=0x50b4 waiting for monitor entry [0x00002b53888ad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
- waiting to lock <0x00000000a3d1de28> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)
- locked <0x00000000a3d20c10> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1795)
...
"ActiveMQ BrokerService.worker.1":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000a3d20bc0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
...
at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1781)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
...
Environment
- Red Hat JBoss A-MQ
- 6.2.1 before Rollup 4
- 6.3 before Rollup 3
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
