Deadlock in A-MQ broker under load, when timed message expiry is in use
Issue
Under conditions of high load, in a set-up that uses timed message expiry, the broker stops producing or consuming messages. A JVM thread dump shows the following two characteristic threads, where one is waiting store a message, and the other is waiting to expire a message:
"ActiveMQ Transport: tcp:///10.80.85.225:43386@61616" daemon prio=10 tid=0x000000001c3df800 nid=0x50b4 waiting for monitor entry [0x00002b53888ad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
- waiting to lock <0x00000000a3d1de28> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)
- locked <0x00000000a3d20c10> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1795)
...
"ActiveMQ BrokerService.worker.1":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000a3d20bc0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
...
at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1781)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
...
Environment
- Red Hat JBoss A-MQ
- 6.2.1 before Rollup 4
- 6.3 before Rollup 3
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.