A-MQ, DLQ; potential deadlock when duplicates read from store for the DLQ destination

Solution Verified - Updated -

Issue

Potential for deadlock when duplicates read in from store for DLQ destination itself and another producer is trying to send messages from the DLQ destination.
It can result with threads in following state

Found one Java-level deadlock:
=============================
"ActiveMQ Transport: tcp:///127.0.0.1:59885@59875":
  waiting for ownable synchronizer 0x00000007f6188490, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "ActiveMQ BrokerService[localhost] Task-1"
"ActiveMQ BrokerService[localhost] Task-1":
  waiting for ownable synchronizer 0x00000007f6190cf8, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "ActiveMQ Transport: tcp:///127.0.0.1:59885@59875"

Java stack information for the threads listed above:
===================================================
"ActiveMQ Transport: tcp:///127.0.0.1:59885@59875":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007f6188490> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
    at org.apache.activemq.broker.region.Queue.toString(Queue.java:937)
    at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:830)
    at org.apache.activemq.broker.region.Queue.send(Queue.java:728)
    at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:390)
    at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:455)
    at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
    at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
    at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
    at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
    at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:541)
    at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:756)
    at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:334)
    at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
    at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
    at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
    at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
    at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
    at java.lang.Thread.run(Thread.java:722)
"ActiveMQ BrokerService[localhost] Task-1":
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000007f6190cf8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:894)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1221)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
    at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:828)
    at org.apache.activemq.broker.region.Queue.send(Queue.java:728)
    at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:390)
    at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:455)
    at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:297)
    at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
    at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
    at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
    at org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)
    at org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)
    at org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:762)
    at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:318)
    at org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:318)
    at org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:330)
    at org.apache.activemq.broker.region.BaseDestination.duplicateFromStore(BaseDestination.java:801)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.dealWithDuplicates(AbstractStoreCursor.java:134)
    at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:110)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:385)
    - locked <0x00000007f6188550> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
    at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:142)
    - locked <0x00000007f6188550> (a org.apache.activemq.broker.region.cursors.QueueStorePrefetch)
    at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)
    - locked <0x00000007f61885a8> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
    at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1906)
    at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2127)
    at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1608)
    - locked <0x00000007f61885f8> (a java.lang.Object)
    at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
    at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

Found 1 deadlock.

Environment

  • JBoss A-MQ 6.1
  • JBoss A-MQ 6.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.