6.3.4.2. Journal size considerations

Important

Because the size of a circular queue is fixed, it is important to make journals large enough to contain the maximum number of messages anticipated. If the journal becomes approximately 80% full, no new messages can be enqueued, and an attempt to enqueue further messages results in an enqueue threshold exception (RHM_IORES_ENQCAPTHRESH). Dequeues are still allowed, and each dequeue frees up space in the store, so enqueues can continue once sufficient space has been freed.
Because transactions can span many persistent queues, enqueue and dequeue events within a transaction are placed in a dedicated persistent queue called the TPL (Transaction Prepared List). When a transaction is committed, the associated events are written to the journal before the transaction commit is acknowledged to the message consumer and messages published in the transaction are made available to message consumers.
If a journal becomes too full to accept new messages, message publishers encounter an enqueue threshold exception (RHM_IORES_ENQCAPTHRESH) when the journal is roughly 80% full. Message consumers can still read messages, making room for new messages.
Applications that use persistent queues must either prevent enqueue threshold exceptions, or respond appropriately when the exception occurs. Here are some ways an application might respond if it encounters such exceptions:
  • Pause to allow messages to be consumed from the queue, then reconnect and resume sending.
  • Publish using a different routing key, or change bindings to route to a different queue.
  • Perform load balancing that matches the rate of publishing with the rate of consuming. Producer flow control is one mechanism that can be used to achieved this.
Enqueue threshold exceptions will only occur if messages are allowed to accumulate beyond the capacity of a given persistent queue, which usually means that the maximum number of messages on the queue at any given time is large, or the maximum size of these messages is large. In these cases, you may need to increase the size of the journals.
However, increasing the size of the journal has a cost: if the journal is very large, creating a new persistent queue results in a noticeable delay while the journal files are initialized - for journals of multiple megabytes, this delay may be up to several seconds. In some applications, this delay may not be an issue, especially if all persistent queues are created before time-critical messaging begins. In such applications, using very large journals can be a good strategy. If you need to minimize the time needed to create a persistent queue, journals should be large enough to prevent enqueue threshold exceptions, but no larger than necessary.