Chapter 8. Message Persistence and Paging

AMQ Broker 7 provides persistence through either a message journal or a JDBC store. The method by which the broker stores messages and pages them to disk is different than AMQ 6, and the configuration properties you use to configure message persistence are changed.

8.1. Message Persistence Changes

AMQ Broker 7 uses a different type of message journal than AMQ 6, and it does not use a journal index.

AMQ 6 used KahaDB for a message store, and it maintained a message journal index to track the position of each message inside the journal. This index enabled the broker to pull paged messages from its journal in batches and place them in its cache.

By default, AMQ Broker 7 uses an in-memory message journal from which the broker can dispatch messages. Therefore, AMQ Broker 7 does not use a message journal index. If a broker instance runs out of memory, messages are paged as they arrive at the broker, but before they are queued. These message page files are stored on disk sequentially in the same order in which they arrived. Then, when memory is freed on the broker, the messages are moved from the page file to the journal on the broker. Because the journal is read sequentially, there is no need to keep an index of messages in the journal.

In addition, AMQ Broker 7 also offers a different JDBC-based message journal option that was not available in AMQ 6.

The AMQ Broker 7 message journal supports the following shared file systems:

  • NFSv4
  • GFS2

Related Information

8.2. How Message Persistence is Configured

You use the BROKER_INSTANCE_DIR/etc/broker.xml configuration file to configure the broker instance’s message journal.

The broker.xml configuration file contains the following default message journal configuration properties: 

<core>

    <name>0.0.0.0</name>

    <persistence-enabled>true</persistence-enabled>

    <journal-type>ASYNCIO</journal-type>

    <paging-directory>./data/paging</paging-directory>

    <bindings-directory>./data/bindings</bindings-directory>

    <journal-directory>./data/journal</journal-directory>

    <large-messages-directory>./data/large-messages</large-messages-directory>

    <journal-datasync>true</journal-datasync>

    <journal-min-files>2</journal-min-files>

    <journal-pool-files>-1</journal-pool-files>

    <journal-buffer-timeout>744000</journal-buffer-timeout>

    <disk-scan-period>5000</disk-scan-period>

    <max-disk-usage>90</max-disk-usage>

    <global-max-size>104857600</global-max-size>

    ...

</core>

To configure the message journal, you can change the default values for any of the journal configuration properties. You can also add additional configuration properties.

8.3. Message Persistence Configuration Property Changes

AMQ 6 and AMQ Broker 7 both offer a number of configuration properties to control how the broker persists messages. This section compares the configuration properties in the AMQ 6 KahaDB journal to the equivalent properties in the AMQ Broker 7 in-memory message journal.

For complete details on each message persistence configuration property for the in-memory message journal, see the following:

8.3.1. Journal Size and Management

The following table compares the journal size and management configuration properties in AMQ 6 to the equivalent properties in AMQ Broker 7:

To set…​In AMQ 6In AMQ Broker 7

The time interval between cleaning up data logs that are no longer used

cleanupInterval

The default is 30000 ms.

No equivalent. In AMQ Broker 7, journal files that exceed the pool size are no longer used.

The number of message store GC cycles that must be completed without cleaning up other files before compaction is triggered

compactAcksAfterNoGC

No equivalent. In AMQ Broker 7, compaction is not related to particular record types.

Whether compaction should be run when the message store is still growing, or if it should only occur when it has stopped growing

compactAcksIgnoresStoreGrowth

The default is false.

No equivalent.

The minimum number of journal files that can be stored on the broker before it will compact them

No equivalent.

<journal-compact-min-files>

The default is 10. If you set this value to 0, compaction will be deactivated.

The threshold to reach before compaction starts

No equivalent.

<journal-compact-percentage>

The default is 30%. When less than this percentage is considered to be live data, compaction will start.

The path to the top-level folder that holds the message store’s data files

directory

AMQ Broker 7 has a separate directory for each type of journal:

  • <journal-directory> - The default is /data/journal.
  • <bindings-directory> - The default is /data/bindings.
  • <paging-directory> - The default is /data/paging.
  • <large-message-directory> - The default is /data/large-messages.

Whether the bindings directory should be created automatically if it does not already exist

No equivalent.

<create-bindings-dir>

The default is true.

Whether the journal directory should be created automatically if it does not already exist

No equivalent.

<create-journal-dir>

The default is true.

Whether the message store should periodically compact older journal log files that contain only message acknowledgements

enableAckCompaction

No equivalent.

The maximum size of the data log files

journalMaxFileLength

The default is 32 MB.

<journal-file-size>

The default is 10485760 bytes (10 MiB).

The policy that the broker should use to preallocate the journal files when a new journal file is needed

preallocationStrategy

The default is sparse_file.

No equivalent. By default, preallocated journal files are typically filled with zeroes, but it can vary depending on the file system.

The policy the broker should use to preallocate the journal files

preallocationScope

The default is entire_journal.

AMQ Broker 7 automatically preallocates the journal files specified by <journal-min-files> when the broker instance is started.

The journal type (either NIO or AIO)

No equivalent.

<journal-type>

You can choose either NIO (Java NIO journal), or ASYNCIO (Linux asynchronous I/O journal).

The minimum number of files that the journal should maintain

No equivalent.

<journal-min-files>

The number of journal files the broker should keep when reclaiming files

No equivalent.

<journal-pool-files>

The default is -1, which means the broker instance will never delete files on the journal once created.

8.3.2. Write Boundaries

The following table compares the write boundary configuration properties in AMQ 6 to the equivalent properties in AMQ Broker 7:

To set…​In AMQ 6In AMQ Broker 7

The time interval between writing the metadata cache to disk

checkpointInterval

The default is 5000 ms.

No equivalent.

Whether the message store should dispatch queue messages to clients concurrently with message storage

concurrentStoreAndDispatchQueues

The default is true.

No equivalent.

Whether the message store should dispatch topic messages to interested clients concurrently with message storage

concurrentStoreAndDispatchTopics

The default is false.

No equivalent.

Whether a disk sync should be performed after each non-transactional journal write

enableJournalDiskSyncs

The default is true.

<journal-sync-transactional>
Flushes transaction data to disk whenever a transaction boundary is reached (commit, prepare, and rollback). The default is true.
<journal-sync-nontransactional>
Flushes non-transactional message data to disk (sends and acknowledgements). The default is true.

When to flush the entire journal buffer

No equivalent.

<journal-buffer-timeout>

The default for NIO is 3,333,333 nanoseconds, and the default for AIO is 500,000 nanoseconds.

The amount of data to buffer between journal disk writes

journalMaxWriteBatchSize

The default is 4000 bytes.

No equivalent.

The size of the task queue used to buffer the journal’s write requests

maxAsyncJobs

The default is 10000.

<journal-max-io>

This property controls the maximum number of write requests that can be in the I/O queue at any given point. The default for NIO is 1, and the default for AIO is 500.

Whether to use fdatasync on journal writes

No equivalent.

<journal-datasync>

The default is true.

8.3.3. Index Configuration

AMQ 6 has a number of configuration properties for configuring the journal index. Because AMQ Broker 7 does not use journal indexes, you do not need to configure any of these properties for your broker instance.

8.3.4. Journal Archival

AMQ 6 has several configuration properties for controlling which files are archived and where the archives are stored. In AMQ Broker 7, however, when old journal files are no longer needed, the broker reuses them instead of archiving them. Therefore, you do not need to configure any journal archival properties for your broker instance.

8.3.5. Journal Recovery

AMQ 6 has several configuration properties for controlling how the broker checks for corrupted journal files and what to do when it encounters a missing journal file.

In AMQ Broker 7, however, you do not need to configure any journal recovery properties for your broker instance. Journal files have a different format in AMQ Broker 7, which should prevent a corrupted entry in the journal from corrupting the entire journal file. Even if the journal is partially damaged, the broker should still be able to extract data from the undamaged entries.