Chapter 10. Configuring Persistence

10.1. About Persistence in JBoss EAP 7 Messaging

JBoss EAP messaging handles its own persistence. It ships with a high-performance journal, which is optimized for messaging.

The JBoss EAP messaging journal has a configurable file size and is append only, which improves performance by enabling single write operations. It consists of a set of files on disk, which are initially pre-created to a fixed size and filled with padding. As server operations (add message, delete message, update message, etc.) are performed, records of the operations are appended to the journal until the journal file is full, at which point the next journal file is used.

A sophisticated garbage collection algorithm determines whether journal files can be reclaimed and re-used when all of their data has been deleted. A compaction algorithm removes dead space from journal files and compresses the data.

The journal also fully supports both local and XA transactions.

The majority of the journal is written in Java, but interaction with the file system has been abstracted to allow different pluggable implementations. The two implementations shipped with JBoss EAP messaging are:

Java New I/O (NIO)
This implementation uses standard Java NIO to interface with the file system. It provides extremely good performance and runs on any platform with a Java 6 or later runtime. Note that JBoss EAP 7 requires Java 8. Using NIO is supported on any operating system that JBoss EAP supports.
Linux Asynchronous IO (ASYNCIO)

This implementation uses a native code wrapper to talk to the Linux asynchronous IO library (ASYNCIO). This implementation removes the need for explicit synchronization. ASYNCIO typically provides better performance than Java NIO.

The following file systems have been tested and are supported only on Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7 when using the libaio natives. They are not tested and are not supported on other operating systems.

  • EXT4
  • XFS
  • NFSv4
  • GFS2

The following table lists the HA shared store file systems that have been tested, both with and without the libaio natives, and whether they are supported.

Operating SystemFile SystemSupported Using libaio Natives?
(journal-type="ASYNCIO")
Supported Without Using libaio Natives?
(journal-type="NIO")

Red Hat Enterprise Linux 6

NFSv4

Yes

Yes

Red Hat Enterprise Linux 7

NFSv4

Yes

Yes

Red Hat Enterprise Linux 6

GFS2

Yes

No

Red Hat Enterprise Linux 7

GFS2

Yes

No

The standard JBoss EAP messaging core server uses the following journal instances:

Bindings journal

This journal is used to store bindings related data, including the set of queues that are deployed on the server and their attributes. It also stores data such as id sequence counters.

The bindings journal is always a NIO journal as it is typically low throughput compared to the message journal.

The files on this journal are prefixed as activemq-bindings. Each file has a bindings extension. File size is 1048576, and it is located at the bindings folder.

JMS journal

This journal instance stores all JMS related data, such as any JMS queues,topics, connection factories and any JNDI bindings for these resources.

Any JMS Resource created via the management API will be persisted to this journal. Any resource configured via configuration files will not. The JMS Journal will only be created if JMS is being used.

The files on this journal are prefixed as activemq-jms. Each file has a jms extension. File size is 1048576, and it is located at the bindings folder.

Message journal

This journal instance stores all message related data, including the message themselves and also duplicate-id caches.

By default JBoss EAP messaging will try to use an ASYNCIO journal. If ASYNCIO is not available, for example the platform is not Linux with the correct kernel version or ASYNCIO has not been installed then it will automatically fall back to using Java NIO which is available on any Java platform.

The files on this journal are prefixed as activemq-data. Each file has an amq extension. File size is by default 10485760 (configurable), and it is located at the journal folder.

For large messages, JBoss EAP messaging persists them outside the message journal. This is discussed in the section on Large Messages.

JBoss EAP messaging can also be configured to page messages to disk in low memory situations. This is discussed in the Paging section.

If no persistence is required at all, JBoss EAP messaging can also be configured not to persist any data at all to storage as discussed in the Configuring JBoss EAP Messaging for Zero Persistence section.

10.2. Configuring the Bindings and JMS Journals

Because the bindings journal shares its configuration with the JMS journal, you can read the current configuration for both by using the single management CLI command below. The output is also included to highlight default configuration.

/subsystem=messaging-activemq/server=default/path=bindings-directory:read-resource

{
    "outcome" => "success",
    "result" => {
        "path" => "activemq/bindings",
        "relative-to" => "jboss.server.data.dir"
    }
}

Note that by default the path to the journal is activemq/bindings. You can change the location for path by using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=bindings-directory:write-attribute(name=path,value=<PATH_LOCATION>)

Also note the relative-to attribute in the output above. When relative-to is used, the value of the path attribute is treated as relative to the file path specified by relative-to. By default this value is the JBoss EAP property jboss.server.data.dir. For standalone servers, jboss.server.data.dir is located at EAP_HOME/standalone/data. For domains, each server will have its own serverX/data/activemq directory located under EAP_HOME/domain/servers. You can change the value of relative-to using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=bindings-directory:write-attribute(name=relative-to,value=<RELATIVE_LOCATION>)

By default, JBoss EAP is configured to automatically create the bindings directory if it does not exist. Use the following management CLI command to toggle this behavior.

/subsystem=messaging-activemq/server=default:write-attribute(name=create-bindings-dir,value=<TRUE_FALSE>)

Setting value to true will enable automatic directory creation. Setting value to false will disable it.

10.3. Configuring the Message Journal Location

You can read the location information for the message journal by using the management CLI command below. The output is also included to highlight default configuration.

/subsystem=messaging-activemq/server=default/path=journal-directory:read-resource
{
    "outcome" => "success",
    "result" => {
        "path" => "activemq/journal",
        "relative-to" => "jboss.server.data.dir"
    }
}

Note that by default the path to the journal is activemq/journal. You can change the location for path by using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=journal-directory:write-attribute(name=path,value=<PATH_LOCATION>)
Note

For the best performance, Red Hat recommends that the journal be located on its own physical volume in order to minimize disk head movement. If the journal is on a volume which is shared with other processes which might be writing other files, such as a bindings journal, database, or transaction coordinator, then the disk head may well be moving rapidly between these files as it writes them, thus drastically reducing performance.

Also note the relative-to attribute in the output above. When relative-to is used, the value of the path attribute is treated as relative to the file path specified by relative-to. By default this value is the JBoss EAP property jboss.server.data.dir. For standalone servers, jboss.server.data.dir is located at EAP_HOME/standalone/data. For domains, each server will have its own serverX/data/activemq directory located under EAP_HOME/domain/servers. You can change the value of relative-to using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=journal-directory:write-attribute(name=relative-to,value=<RELATIVE_LOCATION>)

By default, JBoss EAP is configured to automatically create the journal directory if it does not exist. Use the following management CLI command to toggle this behavior.

/subsystem=messaging-activemq/server=default:write-attribute(name=create-journal-dir,value=<TRUE_FALSE>)

Setting value to true will enable automatic directory creation. Setting value to false will disable it.

10.4. Configuring Message Journal Attributes

The attributes listed below are all child properties of the messaging server. Therefore, the command syntax for getting and setting their values using the management CLI is the same for each.

To read the current value of a given attribute, the syntax is as follows:

/subsystem=messaging-activemq/server=default:read-attribute(name=<ATTRIBUTE_NAME>)

The syntax for writing an attribute’s value follows a corresponding pattern.

/subsystem=messaging-activemq/server=default:write-attribute(name=<ATTRIBUTE_NAME>,value=<NEW_VALUE>)
  • create-journal-dir

    If this is set to true, the journal directory will be automatically created at the location specified in journal-directory if it does not already exist. The default value is true.

  • journal-buffer-timeout

    Instead of flushing on every write that requires a flush, we maintain an internal buffer, and flush the entire buffer either when it is full, or when a timeout expires, whichever is sooner. This is used for both NIO and ASYNCIO and allows the system to scale better with many concurrent writes that require flushing.

    This parameter controls the timeout at which the buffer will be flushed if it has not filled already. ASYNCIO can typically cope with a higher flush rate than NIO, so the system maintains different defaults for both NIO and ASYNCIO. The default for NIO is 3333333 nanoseconds, or 300 times per second. The default for ASYNCIO is 500000 nanoseconds, or 2000 times per second.

    Note

    By increasing the timeout, you may be able to increase system throughput at the expense of latency, the default parameters are chosen to give a reasonable balance between throughput and latency.

  • journal-buffer-size

    The size, in bytes, of the timed buffer on ASYNCIO. Both journal-buffer-size and journal-file-size must be set larger than min-large-message-size. Otherwise, messages will not be written to the journal. See Configuring Large Messages for more information.

  • journal-compact-min-files

    The minimal number of files before we can consider compacting the journal. The compacting algorithm won’t start until you have at least journal-compact-min-files.

    Setting this to 0 will disable the feature to compact completely. This could be dangerous though as the journal could grow indefinitely. Use it wisely!

    The default for this parameter is 10

  • journal-compact-percentage

    The threshold to start compacting. When less than this percentage is considered live data, we start compacting. Note also that compacting will not kick in until you have at least journal-compact-min-files data files on the journal

    The default for this parameter is 30.

  • journal-file-size

    The size of each journal file, in bytes. The default value for this is 10485760 bytes, or 10MB. Both journal-file-size and journal-buffer-size must be set larger than min-large-message-size. Otherwise, messages will not be written to the journal. See Configuring Large Messages for more information.

  • journal-max-io

    Write requests are queued up before being submitted to the system for execution. This parameter controls the maximum number of write requests that can be in the IO queue at any one time. If the queue becomes full then writes will block until space is freed up.

    The system maintains different defaults for this parameter depending on whether it’s NIO or ASYNCIO. The default for NIO is 1, and the default for ASYNCIO is 500.

    There is a limit and the total max ASYNCIO cannot be higher than what is configured at the OS level, found at /proc/sys/fs/aio-max-nr, usually 65536.

  • journal-min-files

    The minimum number of files the journal will maintain. When JBoss EAP starts and there is no initial message data, JBoss EAP will pre-create journal-min-files number of files. The default is 2.

    Creating journal files and filling them with padding is a fairly expensive operation and we want to minimize doing this at run-time as files get filled. By pre-creating files, as one is filled the journal can immediately resume with the next one without pausing to create it.

    Depending on how much data you expect your queues to contain at steady state you should tune this number of files to match that total amount of data.

  • journal-pool-files

    The number of journal files that can be reused. ActiveMQ will create as many files as needed however when reclaiming files it will shrink back to the value. The default is -1, which means no limit.

  • journal-sync-transactional

    If this is set to true then JBoss EAP will make sure all transaction data is flushed to disk on transaction boundaries, such as a commit, prepare, or rollback. The default value is true.

  • journal-sync-non-transactional

    If this is set to true then JBoss EAP will make sure non transactional message data, such as sends and acknowledgements, are flushed to disk each time. The default value is true.

  • journal-type

    Valid values are NIO or ASYNCIO.

    Choosing NIO tells JBoss EAP to use a Java NIO journal. ASYNCIO tells it to use a Linux asynchronous IO journal. If you choose ASYNCIO but are not running Linux, or you do not have libaio installed, JBoss EAP will use a Java NIO journal.

10.5. Note on Disabling Disk Write Cache

This happens irrespective of whether you have executed a fsync() from the operating system or correctly synced data from inside a Java program!

By default many systems ship with disk write cache enabled. This means that even after syncing from the operating system there is no guarantee the data has actually made it to disk, so if a failure occurs, critical data can be lost.

Some more expensive disks have non volatile or battery backed write caches which won’t necessarily lose data on event of failure, but you need to test them!

If your disk does not have an expensive non volatile or battery backed cache and it’s not part of some kind of redundant array (e.g. RAID), and you value your data integrity you need to make sure disk write cache is disabled.

Be aware that disabling disk write cache can give you a nasty shock performance wise. If you’ve been used to using disks with write cache enabled in their default setting, unaware that your data integrity could be compromised, then disabling it will give you an idea of how fast your disk can perform when acting really reliably.

On Linux you can inspect or change your disk’s write cache settings using the tools hdparm (for IDE disks) or sdparm or sginfo (for SDSI/SATA disks)

On Windows you can check / change the setting by right clicking on the disk and clicking properties.

10.6. Installing libaio

The Java NIO journal is highly performant, but if you are running JBoss EAP messaging using Linux Kernel 2.6 or later, Red Hat highly recommends that you use the ASYNCIO journal for the very best persistence performance.

Note

JBoss EAP supports ASYNCIO only when installed on versions 6 or 7 of Red Hat Enterprise Linux and only when using the ext4, xfs, gfs2 or nfs4 file systems. It is not possible to use the ASYNCIO journal under other operating systems or earlier versions of the Linux kernel.

You will need libaio installed to use the ASYNCIO journal. You can easily install it using the following YUM command:

yum install libaio
Warning

Do not place your messaging journals on a tmpfs file system, which is used for the /tmp directory for example. JBoss EAP will fail to start if the ASYNCIO journal is using tmpfs.

10.7. Configuring the NFS Shared Store for Messaging

When using dedicated, shared store, high availability for data replication, you must configure both the live server and the backup server to use a shared directory on the NFS client. If you configure one server to use a shared directory on the NFS server and the other server to use a shared directory on the NFS client, the backup server cannot recognize when the live server starts or is running. So to work properly, both servers must specify a shared directory on the NFS client.

You must also configure the following options for the NFS client mount:

  • sync: This option specifies that all changes are immediately flushed to disk.
  • intr: This option allows NFS requests to be interrupted if the server goes down or cannot be reached.
  • noac: This option disables attribute caching and is needed to achieve attribute cache coherence among multiple clients.
  • soft: This option specifies that if the host serving the exported file system is unavailable, the error should be reported rather than waiting for the server to come back online.
  • lookupcache=none: This option disables lookup caching.

See the Shared Store section in this guide for more information about how to use a shared file system for high availability.

10.8. Configuring JBoss EAP Messaging for Zero Persistence

In some situations, zero persistence is required for a messaging system. Zero persistence means that no bindings data, message data, large message data, duplicate id caches, or paging data should be persisted.

To configure the messaging-activemq subsystem to perform zero persistence, set the persistence-enabled parameter to false.

/subsystem=messaging-activemq/server=default:write-attribute(name=persistence-enabled,value=false)
Important

Be aware that if persistence is disabled, but paging is enabled, page files continue to be stored in the location specified by the paging-directory element. Paging is enabled when the address-full-policy attribute is set to PAGE. If full zero persistence is required, be sure to configure the address-full-policy attribute of the address-setting element to use BLOCK, DROP or FAIL.

10.9. Importing and Exporting Journal Data

See the JBoss EAP 7 Migration Guide for information on importing and exporting journal data.