4.9.2. Disk-paged Queues

MRG 3 replaces the MRG 2 flow-to-disk queue policy with more performant paged queues. Paged queues are backed by a file, and a configurable number of pages of messages are held in memory. Paged queues balance responsive performance (by holding messages in-memory and writing pages of messages rather than individual messages to disk) with load capacity (by allowing the queue to use the file system for additional storage).
Messages are stored in pages. The size of the memory page is configurable, and should be set to greater than the largest anticipated message size, to allow the message to fit inside a page. Messages larger than the page size are rejected by the broker.
The number of pages to hold in memory is configurable. When the configured maximum number of pages of messages in-memory are filled, further messages cause a page to be swapped out of memory to the disk file to allow an empty in-memory page to receive further messages. Pages are proactively written to disk while still held in memory to optimize performance. When a message is requested from a page that is not in-memory, that page is retrieved from the disk. If the in-memory page limit has been reached, then a page in-memory is sent to disk to allow the new page to be loaded.
The upper limit of number of pages supported by the broker is determined by the system mapping limit. This is a kernel attribute and can be examined by cat /proc/sys/vm/max_map_count, and set at run-time with: 
echo 100000 >/proc/sys/vm/max_map_count
Note that this method of setting the limit is non-permanent and is erased after a restart. To configure the limit in a persistent way, use the /etc/sysctl.conf file.
The size of a paged queue can be controlled by the usual size and message count limits.
Note that the disk-based storage for paged queues is not inherently persistent. It is used at run-time to manage the queue and balance memory use, and is not persistent across broker restarts. Paged queues can be declared durable, which provides persistence to messages that request it.
Limitations of Paged Queues

  • A paged queue cannot handle a message larger than the page size, so the queue must be configured with pages at least as big as the largest anticipated message.
  • A paged queue cannot also be a LVQ or Priority queue. An exception is thrown by an attempt to create a paged queue with LVQ or Priority specified.

Creating a Paged Queue

To configure a queue as a paged queue, specify the argument qpid.paging as true when declaring the queue.

The additional configuration options are:
qpid.max_pages_loaded
Controls how many pages are allowed to be held in memory at any given time. Default value is 4.
qpid.page_factor
Controls the size of the page. Default value is 1. The value is a multiple of the platform-defined page size. On Linux the platform-defined page size can be examined using the command getconf PAGESIZE. It is typically 4k, depending on your CPU architecture.
Example

The following command line example demonstrates creation of a paged queue: 

qpid-config add queue my-paged-queue --argument qpid.paging=True --argument qpid.max_pages_loaded=100 --argument qpid.page_factor=1
The same is accomplished in code in the following manner:
Python
tx = session.sender("my-paged-queue; {create: always, node: {x-declare: {'auto-delete': True, arguments:{'qpid.page_factor': 1, 'qpid.max_pages_loaded': 100, 'qpid.paging': True}}}}")
Report a bug