A-MQ memory leak when using STOMP and client acknowledgement

Solution Verified - Updated -

Environment

  • Red Hat JBoss A-MQ
    • 6.1.0

Issue

  • A-MQ memory leak when using STOMP and client acknowledgement

Resolution

Update to A-MQ 6.1.0 Rollup 2.

Root Cause

This happens due to a bug on the STOMP acknowledgement tracking code in A-MQ as explain in AMQ-5423 and ENTMQ-849.

Diagnostic Steps

Looking at a head dump when memory consumption is high, verify if the classes below are consuming a great part of the heap:

Class Name                                                                      | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------
org.apache.activemq.broker.region.cursors.FilePendingMessageCursor @ 0x634a338d0|           88 | 1,872,081,960 |     33.27%
|- org.apache.activemq.broker.region.cursors.OrderedPendingList @ 0x633705ae0   |           24 | 1,872,079,296 |     33.27%
|  |- java.util.HashMap @ 0x633705af8                                           |           48 |    39,833,920 |      0.71%
|  |- org.apache.activemq.broker.region.cursors.PendingNode @ 0x687dc3e98       |           32 |         2,032 |      0.00%
----------------------------------------------------------------------------------------------------------------------------

java.util.HashMap @ 0x633705af8 contains 2,097,152 and org.apache.activemq.broker.region.cursors.OrderedPendingList @ 0x633705ae0 contains 2,947,992 instances of org.apache.activemq.broker.region.cursors.PendingNode.

This single instance from org.apache.activemq.broker.region.cursors.FilePendingMessageCursor occupying 1.8GB+ of memory shows a large number of messages pending.

Looking further there should be a large amount of org.apache.activemq.transport.stomp.ProtocolConverter$AckEntry in memory:

Class Name                                                    |    Objects | Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------
org.apache.activemq.transport.stomp.ProtocolConverter$AckEntry| 10,338,285 |  248,118,840 | 1,733,213,384
----------------------------------------------------------------------------------------------------------

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.