6.4. Determining Journal Size

Section 6.3, “Configuring the Journal” explains how to set the size of a journal for a persistent queue. This size is fixed for the life of the queue. If a journal becomes too full to accept new messages, message publishers encounter an enqueue threshold exception (RHM_IORES_ENQCAPTHRESH) when the journal is roughly 80% full. Message consumers can still read messages, making room for new messages.
Applications that use persistent queues must either prevent enqueue threshhold exceptions, or respond appropriately when the exception occurs. Here are some ways an application might respond if it encounters such exceptions:
  • Pause to allow messages to be consumed from the queue, then reconnect and resume sending.
  • Publish using a different routing key, or change bindings to route to a different queue.
  • Perform some sort of load balancing that matches the rate of publishing with the rate of consuming.
Enqueue threshhold exceptions will only occur if messages are allowed to accumulate beyond the capacity of a given persistent queue, which usually means that the maximum number of messages on the queue at any given time is large, or the maximum size of these messages is large. In these cases, you may need to increase the size of the journals.
However, increasing the size of the journal has a cost: if the journal is very large, creating a new persistent queue results in a noticeable delay while the journal files are initialized—for journals of multiple megabytes, this delay may be up to several seconds. In some applications, this delay may not be an issue, especially if all persistent queues are created before time-critical messaging begins. In such applications, using very large journals can be a good strategy. If you need to minimize the time needed to create a persistent queue, journals should be large enough to prevent enqueue threshhold exceptions, but no larger than necessary.

6.4.1. Queue Depth

Queue depth refers to the number of messages on a queue at a given time. No matter how large your journal is, it will eventually fill if messages are published to a persistent queue faster than they are consumed.
To prevent enqueue threshhold exceptions, an application must ensure that messages are consumed and acknowledged, and that messages will not exceed the capacity of the queue journal. The way this is done depends on the application. For instance, some applications may ensure that messages are processed as quickly as possible, keeping queue depth to a minimum. Other applications may process all messages from a given queue periodically, ensuring that the journal size is large enough to accomodate any messages that might conceivably accumulate in the meantime.
In some applications, the rate at which messages are produced and consumed is known. In other applications, qpid-tool can be used to monitor queue depth over time, providing an initial value that can be used to estimate maximum queue depth.