[AMQ 7.2.x, large Message ]Can't write records bigger than the bufferSize(501760) on the journal

Solution Verified - Updated -

Environment

RedHat AMQ 7.2.4

Issue

  • intermittently hitting the following exception.
2019-06-18 14:50:20,862 WARN  [org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile /apps/artemis/amq-broker-7.2.4/examples/features/clustered/clustered-static-discovery/target/server0/./data/journal/activemq-data-1.amq, message=Can't write records bigger than the bufferSize(501760) on the journal: java.lang.IllegalStateException: Can't write records bigger than the bufferSize(501760) on the journal
    at org.apache.activemq.artemis.core.io.buffer.TimedBuffer.checkSize(TimedBuffer.java:247) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.core.io.AbstractSequentialFile.fits(AbstractSequentialFile.java:171) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.switchFileIfNecessary(JournalImpl.java:3013) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendRecord(JournalImpl.java:2733) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl.access$200(JournalImpl.java:91) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.core.journal.impl.JournalImpl$4.run(JournalImpl.java:1003) [artemis-journal-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
    at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$2/1768127248.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_20]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_20]
    at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.6.3.redhat-00020.jar:2.6.3.redhat-00020]
  • message headers/properties are quite small so it is NOT directly related to ENTMQBR-2467

Resolution

When Message arrives, if it is just under the 501760 bytes (assuming broker is using defaults), the org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl#checkLargeMessage() does not turn the message into a Large message. However when the message is being written to the journal the size has increased slightly as some additional flags(meta data) has being included . This meta data is pushing the message size above the journal buffer size.

In AMQ 7.3 org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl#checkLargeMessage() adds another 512 bytes ( "ESTIMATE_RECORD_TRAIL") to the message size to compensate for the additional flags. If the message is nearing the LargeMessage limit, this will push the message above the limit and org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl#checkLargeMessage() will create a large message where it did not in AMQ 7.2.4. The ESTIMATE_RECORD_TRAIL was introduced as part of ARTEMIS-2274 .

This issue should not arise in AMQ 7.3.0.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments