Chapter 10. Configuring Persistence

10.1. About Persistence in JBoss EAP 7 Messaging

JBoss EAP ships with two persistence options for storing binding data and messages:

  • You can use the default file-based journal, which is highly optimized for messaging use cases and provides great performance. This option is provided by default and is used if you do not do any additional configuration.
  • You can store the data in a JDBC data store, which uses JDBC to connect to a database of your choice. This option requires configuration of the datasources and messaging-activemq subsystems in the server configuration file.

10.2. Messaging Journal Persistence Using the Default File Journal

JBoss EAP messaging ships with a high-performance, file-based journal that is optimized for messaging.

The JBoss EAP messaging journal has a configurable file size and is append only, which improves performance by enabling single write operations. It consists of a set of files on disk, which are initially pre-created to a fixed size and filled with padding. As server operations, such as add message, delete message, update message, are performed, records of the operations are appended to the journal until the journal file is full, at which point the next journal file is used.

A sophisticated garbage collection algorithm determines whether journal files can be reclaimed and re-used when all of their data has been deleted. A compaction algorithm removes dead space from journal files and compresses the data.

The journal also fully supports both local and XA transactions.

10.2.1. Messaging Journal File System Implementations

The majority of the journal is written in Java, but interaction with the file system has been abstracted to allow different pluggable implementations. The two implementations shipped with JBoss EAP messaging are:

Java New I/O (NIO)
This implementation uses standard Java NIO to interface with the file system. It provides extremely good performance and runs on any platform with a Java 6 or later runtime. Note that JBoss EAP 7 requires Java 8. Using NIO is supported on any operating system that JBoss EAP supports.
Linux Asynchronous IO (ASYNCIO)

This implementation uses a native code wrapper to talk to the Linux asynchronous IO library (ASYNCIO). This implementation removes the need for explicit synchronization. ASYNCIO typically provides better performance than Java NIO.

To check which journal type is in use, issue the following CLI request:

/subsystem=messaging-activemq/server=default:read-attribute(name=runtime-journal-type)

The system returns one of the following values:

Table 10.1. Journal Type Return Values

Return ValueDescription

NONE

Persistence is disabled

NIO

Java NIO is in use

ASYNCIO

AsyncIO with libaio is in use

DATABASE

JDBC persistence is in use

The following file systems have been tested and are supported only on Red Hat Enterprise Linux 6, Red Hat Enterprise Linux 7, and Red Hat Enterprise Linux 8 when using the libaio natives. They are not tested and are not supported on other operating systems.

  • EXT4
  • XFS
  • NFSv4
  • GFS2

The following table lists the HA shared store file systems that have been tested, both with and without the libaio natives, and whether they are supported.

Operating SystemFile SystemSupported Using libaio Natives?
(journal-type="ASYNCIO")
Supported Without Using libaio Natives?
(journal-type="NIO")

Red Hat Enterprise Linux 6

NFSv4

Yes

Yes

Red Hat Enterprise Linux 7 and later

NFSv4

Yes

Yes

Red Hat Enterprise Linux 6

GFS2

Yes

No

Red Hat Enterprise Linux 7 and later

GFS2

Yes

No

10.2.2. Standard Messaging Journal File System Instances

The standard JBoss EAP messaging core server uses the following journal instances:

Bindings Journal

This journal is used to store bindings related data, including the set of queues that are deployed on the server and their attributes. It also stores data such as id sequence counters.

The bindings journal is always a NIO journal as it is typically low throughput compared to the message journal.

The files on this journal are prefixed as activemq-bindings. Each file has a bindings extension. File size is 1048576, and it is located at the bindings folder.

JMS Journal

This journal instance stores all JMS related data, such as any JMS queues,topics, connection factories and any JNDI bindings for these resources.

Any JMS Resource created via the management API will be persisted to this journal. Any resource configured via configuration files will not. The JMS Journal will only be created if JMS is being used.

The files on this journal are prefixed as activemq-jms. Each file has a jms extension. File size is 1048576, and it is located at the bindings folder.

Message Journal

This journal instance stores all message related data, including the message themselves and also duplicate-id caches.

By default JBoss EAP messaging will try to use an ASYNCIO journal. If ASYNCIO is not available, for example the platform is not Linux with the correct kernel version or ASYNCIO has not been installed then it will automatically fall back to using Java NIO which is available on any Java platform.

The files on this journal are prefixed as activemq-data. Each file has an amq extension. File size is by default 10485760 (configurable), and it is located at the journal folder.

For large messages, JBoss EAP messaging persists them outside the message journal. This is discussed in the section on Large Messages.

JBoss EAP messaging can also be configured to page messages to disk in low memory situations. This is discussed in the Paging section.

If no persistence is required at all, JBoss EAP messaging can also be configured not to persist any data at all to storage as discussed in the Configuring JBoss EAP Messaging for Zero Persistence section.

10.2.3. Configuring the Bindings and JMS Journals

Because the bindings journal shares its configuration with the JMS journal, you can read the current configuration for both by using the single management CLI command below. The output is also included to highlight default configuration.

/subsystem=messaging-activemq/server=default/path=bindings-directory:read-resource

{
    "outcome" => "success",
    "result" => {
        "path" => "activemq/bindings",
        "relative-to" => "jboss.server.data.dir"
    }
}

Note that by default the path to the journal is activemq/bindings. You can change the location for path by using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=bindings-directory:write-attribute(name=path,value=PATH_LOCATION)

Also note the relative-to attribute in the output above. When relative-to is used, the value of the path attribute is treated as relative to the file path specified by relative-to. By default this value is the JBoss EAP property jboss.server.data.dir. For standalone servers, jboss.server.data.dir is located at EAP_HOME/standalone/data. For domains, each server will have its own serverX/data/activemq directory located under EAP_HOME/domain/servers. You can change the value of relative-to using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=bindings-directory:write-attribute(name=relative-to,value=RELATIVE_LOCATION)

By default, JBoss EAP is configured to automatically create the bindings directory if it does not exist. Use the following management CLI command to toggle this behavior.

/subsystem=messaging-activemq/server=default:write-attribute(name=create-bindings-dir,value=TRUE/FALSE)

Setting value to true will enable automatic directory creation. Setting value to false will disable it.

10.2.4. Configuring the Message Journal Location

You can read the location information for the message journal by using the management CLI command below. The output is also included to highlight default configuration.

/subsystem=messaging-activemq/server=default/path=journal-directory:read-resource
{
    "outcome" => "success",
    "result" => {
        "path" => "activemq/journal",
        "relative-to" => "jboss.server.data.dir"
    }
}

Note that by default the path to the journal is activemq/journal. You can change the location for path by using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=journal-directory:write-attribute(name=path,value=PATH_LOCATION)
Note

For the best performance, Red Hat recommends that the journal be located on its own physical volume in order to minimize disk head movement. If the journal is on a volume which is shared with other processes which might be writing other files, such as a bindings journal, database, or transaction coordinator, then the disk head may well be moving rapidly between these files as it writes them, thus drastically reducing performance.

Also note the relative-to attribute in the output above. When relative-to is used, the value of the path attribute is treated as relative to the file path specified by relative-to. By default this value is the JBoss EAP property jboss.server.data.dir. For standalone servers, jboss.server.data.dir is located at EAP_HOME/standalone/data. For domains, each server will have its own serverX/data/activemq directory located under EAP_HOME/domain/servers. You can change the value of relative-to using the following management CLI command.

/subsystem=messaging-activemq/server=default/path=journal-directory:write-attribute(name=relative-to,value=RELATIVE_LOCATION)

By default, JBoss EAP is configured to automatically create the journal directory if it does not exist. Use the following management CLI command to toggle this behavior.

/subsystem=messaging-activemq/server=default:write-attribute(name=create-journal-dir,value=TRUE/FALSE)

Setting value to true will enable automatic directory creation. Setting value to false will disable it.

10.2.5. Configuring Message Journal Attributes

The attributes listed below are all child properties of the messaging server. Therefore, the command syntax for getting and setting their values using the management CLI is the same for each.

To read the current value of a given attribute, the syntax is as follows:

/subsystem=messaging-activemq/server=default:read-attribute(name=ATTRIBUTE_NAME)

The syntax for writing an attribute’s value follows a corresponding pattern.

/subsystem=messaging-activemq/server=default:write-attribute(name=ATTRIBUTE_NAME,value=NEW_VALUE)
  • create-journal-dir

    If this is set to true, the journal directory will be automatically created at the location specified in journal-directory if it does not already exist. The default value is true.

  • journal-file-open-timeout

    This attribute modifies the timeout value for opening a journal file. The default value is 5 seconds.

  • journal-buffer-timeout

    Instead of flushing on every write that requires a flush, we maintain an internal buffer, and flush the entire buffer either when it is full, or when a timeout expires, whichever is sooner. This is used for both NIO and ASYNCIO and allows the system to scale better with many concurrent writes that require flushing.

    This parameter controls the timeout at which the buffer will be flushed if it has not filled already. ASYNCIO can typically cope with a higher flush rate than NIO, so the system maintains different defaults for both NIO and ASYNCIO. The default for NIO is 3333333 nanoseconds, or 300 times per second. The default for ASYNCIO is 500000 nanoseconds, or 2000 times per second.

    Note

    By increasing the timeout, you may be able to increase system throughput at the expense of latency, the default parameters are chosen to give a reasonable balance between throughput and latency.

  • journal-buffer-size

    The size, in bytes, of the timed buffer on ASYNCIO. Both journal-buffer-size and journal-file-size must be set larger than min-large-message-size. Otherwise, messages will not be written to the journal. See Configuring Large Messages for more information.

  • journal-compact-min-files

    The minimal number of files before we can consider compacting the journal. The compacting algorithm won’t start until you have at least journal-compact-min-files.

    Setting this to 0 will disable the feature to compact completely. This could be dangerous though as the journal could grow indefinitely. Use it wisely!

    The default for this parameter is 10

  • journal-compact-percentage

    The threshold to start compacting. When less than this percentage is considered live data, we start compacting. Note also that compacting will not kick in until you have at least journal-compact-min-files data files on the journal

    The default for this parameter is 30.

  • journal-file-size

    The size of each journal file, in bytes. The default value for this is 10485760 bytes, or 10MB. Both journal-file-size and journal-buffer-size must be set larger than min-large-message-size. Otherwise, messages will not be written to the journal. See Configuring Large Messages for more information.

  • journal-max-io

    Write requests are queued up before being submitted to the system for execution. This parameter controls the maximum number of write requests that can be in the IO queue at any one time. If the queue becomes full then writes will block until space is freed up.

    The system maintains different defaults for this parameter depending on whether it’s NIO or ASYNCIO. The default for NIO is 1, and the default for ASYNCIO is 500.

    There is a limit and the total max ASYNCIO cannot be higher than what is configured at the OS level, found at /proc/sys/fs/aio-max-nr, usually 65536.

  • journal-min-files

    The minimum number of files the journal will maintain. When JBoss EAP starts and there is no initial message data, JBoss EAP will pre-create journal-min-files number of files. The default is 2.

    Creating journal files and filling them with padding is a fairly expensive operation and we want to minimize doing this at run-time as files get filled. By pre-creating files, as one is filled the journal can immediately resume with the next one without pausing to create it.

    Depending on how much data you expect your queues to contain at steady state you should tune this number of files to match that total amount of data.

  • journal-pool-files

    The number of journal files that can be reused. ActiveMQ will create as many files as needed however when reclaiming files it will shrink back to the value. The default is -1, which means no limit.

  • journal-sync-transactional

    If this is set to true then JBoss EAP will make sure all transaction data is flushed to disk on transaction boundaries, such as a commit, prepare, or rollback. The default value is true.

  • journal-sync-non-transactional

    If this is set to true then JBoss EAP will make sure non transactional message data, such as sends and acknowledgements, are flushed to disk each time. The default value is true.

  • journal-type

    Valid values are NIO or ASYNCIO.

    Choosing NIO tells JBoss EAP to use a Java NIO journal. ASYNCIO tells it to use a Linux asynchronous IO journal. If you choose ASYNCIO but are not running Linux, or you do not have libaio installed, JBoss EAP will use a Java NIO journal.

10.2.6. Note on Disabling Disk Write Cache

This happens irrespective of whether you have executed a fsync() from the operating system or correctly synced data from inside a Java program!

By default many systems ship with disk write cache enabled. This means that even after syncing from the operating system there is no guarantee the data has actually made it to disk, so if a failure occurs, critical data can be lost.

Some more expensive disks have non volatile or battery backed write caches which will not necessarily lose data on event of failure, but you need to test them!

If your disk does not have an expensive non volatile or battery backed cache and it’s not part of some kind of redundant array, for example RAID, and you value your data integrity you need to make sure disk write cache is disabled.

Be aware that disabling disk write cache can give you a nasty shock performance wise. If you’ve been used to using disks with write cache enabled in their default setting, unaware that your data integrity could be compromised, then disabling it will give you an idea of how fast your disk can perform when acting really reliably.

On Linux you can inspect or change your disk’s write cache settings using the tools hdparm for IDE disks, or sdparm or sginfo for SDSI/SATA disks.

On Windows, you can check and change the setting by right clicking on the disk and then clicking properties.

10.2.7. Installing libaio

The Java NIO journal is highly performant, but if you are running JBoss EAP messaging using Linux Kernel 2.6 or later, Red Hat highly recommends that you use the ASYNCIO journal for the very best persistence performance.

Note

JBoss EAP supports ASYNCIO only when installed on versions 6, 7 or 8 of Red Hat Enterprise Linux and only when using the ext4, xfs, gfs2 or nfs4 file systems. It is not possible to use the ASYNCIO journal under other operating systems or earlier versions of the Linux kernel.

You will need libaio installed to use the ASYNCIO journal. To install, use the following command:

  • For Red Hat Enterprise Linux 6 and 7:

    yum install libaio
  • For Red Hat Enterprise Linux 8:

    dnf install libaio
Warning

Do not place your messaging journals on a tmpfs file system, which is used for the /tmp directory for example. JBoss EAP will fail to start if the ASYNCIO journal is using tmpfs.

10.2.8. Configuring the NFS Shared Store for Messaging

When using dedicated, shared store, high availability for data replication, you must configure both the live server and the backup server to use a shared directory on the NFS client. If you configure one server to use a shared directory on the NFS server and the other server to use a shared directory on the NFS client, the backup server cannot recognize when the live server starts or is running. So to work properly, both servers must specify a shared directory on the NFS client.

You must also configure the following options for the NFS client mount:

  • sync: This option specifies that all changes are immediately flushed to disk.
  • intr: This option allows NFS requests to be interrupted if the server goes down or cannot be reached.
  • noac: This option disables attribute caching and is needed to achieve attribute cache coherence among multiple clients.
  • soft: This option specifies that if the host serving the exported file system is unavailable, the error should be reported rather than waiting for the server to come back online.
  • lookupcache=none: This option disables lookup caching.
  • timeo=n: The time in deciseconds (tenths of a second) the NFS client waits for a response before it retries an NFS request. For NFS over TCP, the default timeo value is 600 (60 seconds). For NFS over UDP, the client uses an adaptive algorithm to estimate an appropriate timeout value for frequently used request types, such as read and write requests.
  • retrans=n: The number of times the NFS client retries a request before it attempts further recovery action. If the retrans option is not specified, the NFS client tries each request three times.
Important

It is important to use reasonable values when you configure the timeo and retrans options. A default timeo wait time of 600 deciseconds (60 seconds) combined with a retrans value of 5 retries can result in a five minute wait for ActiveMQ Artemis to detect an NFS disconnection.

See the Shared Store section in this guide for more information about how to use a shared file system for high availability.

10.3. Messaging Journal Persistence Using a JDBC Database

You can configure JBoss EAP 7 messaging to use JDBC to persist messages and binding data to a database instead of using the default file-based journal. To do this, you must first configure the datasource element in the datasources subsystem, and then define a journal-datasource attribute on the server element in the messaging-activemq subsystem to use that datasource. The presence of the journal-datasource attribute notifies the messaging subsystem to persist the journal entries to the database instead of the file-based journal. The journal-database attribute on the server resource in the messaging-activemq subsystem defines the SQL dialect that will be used to communicate with the database. It is configured automatically using the datasource metadata.

When persisting messages to a file-based journal, the large message size is limited only by the size of the disk. However, when persisting messages to a database, the large message size is limited to the maximum size of the BLOB data type for that database.

Important

JBoss EAP 7.3 currently supports only the Oracle 12c and IBM DB2 Enterprise databases.

10.3.1. Configuring a Messaging Journal JDBC Persistence Store

Follow these steps to configure JBoss EAP 7 messaging to use JDBC to persist messages and binding data to a database.

  1. Configure a datasource in the datasources subsystem for use by the messaging-activemq subsystem. For information about how to create and configure a datasource, see Datasource Management in the JBoss EAP Configuration Guide.
  2. Configure the messaging-activemq subsystem to use the new datasource.

    /subsystem=messaging-activemq/server=default:write-attribute(name=journal-datasource,value="MessagingOracle12cDS")

    This creates the following configuration in the messaging-activemq subsystem of the server configuration file.

    <server name="default">
      <journal datasource="MessagingOracle12cDS"/>
      ...
    </server>

JBoss EAP messaging is now configured to use the database to store messaging data.

10.3.2. Configuring Messaging Journal Table Names

JBoss EAP 7 messaging uses a separate JDBC table to store binding information, messages, large messages, and paging information. The names of these tables can be configured using the journal-bindings-table, journal-jms-bindings-table, journal-messages-table, journal-large-messages-table, and journal-page-store-table attributes on the server resource in the messaging-activemq subsystem of the server configuration file.

There are some restrictions on table names.

  • JBoss EAP 7 messaging generates identifiers for paging tables using pattern TABLE_NAME + GENERATED_ID, where the GENERATED_ID can be up to 20 characters long. Because the maximum table name length in Oracle Database 12c is 30 characters, you must limit the table name to 10 characters. Otherwise, you might see the error ORA-00972: identifier is too long and paging will no longer work.
  • Table names that do not follow Schema Object Naming Rules for Oracle Database 12c must be enclosed within double quotes. Quoted identifiers can begin with any character and can contain any characters and punctuation marks as well as spaces. However, neither quoted nor nonquoted identifiers can contain double quotation marks or the null character (\0). It is important to note that quoted identifiers are case sensitive.
  • If multiple JBoss EAP server instances use the same database to persist messages and binding data, the table names must be unique for each server instance. Multiple JBoss EAP servers cannot access the same tables.

The following is an example of the management CLI command that configures the journal-page-store-table name using a quoted identifier.

/subsystem=messaging-activemq/server=default:write-attribute(name=journal-page-store-table,value="\"PAGE_DATA\"")

This creates the following configuration in the messaging-activemq subsystem of the server configuration file.

<server name="default">
  <journal datasource="MessagingOracle12cDS" journal-page-store-table="&quot;PAGED_DATA&quot;"/>
  ...
</server>

10.3.3. Configuring Messaging Journals in a Managed Domain

As mentioned previously in Configuring Messaging Journal Table Names, multiple JBoss EAP servers cannot access the same database tables when using JDBC to persist messages and binding data to a database. Because, in a managed domain, all JBoss EAP server instances in a server group share the same profile configuration, you must use expressions to configure the messaging journal names or datasources.

If all servers are configured to use the same database to store messaging data, the table names must be unique for each server instance. The following is an example of a management CLI command that creates a unique journal-page-store-table table name for each server in a server group by using an expression that includes the unique node identifier in the name.

/subsystem=messaging-activemq/server=default:write-attribute(name=journal-page-store-table,value="${env.NODE_ID}_page_store")

If each server instance accesses a different database, you can use expressions to allow the messaging configuration for each server to connect to a different datasource. The following management CLI command uses the DB_CONNECTION_URL environment variable in the connection-url to connect to a different datasource.

data-source add --name=messaging-journal --jndi-name=java:jboss/datasources/messaging-journal --driver-name=oracle12c  --connection-url=${env.DB_CONNECTION_URL}

10.3.4. Configuring the Messaging Journal Network Timeout

You can configure the maximum amount of time, in milliseconds, that the JDBC connection will wait for the database to reply a request. This is useful in the event that the network goes down or a connection between JBoss EAP messaging and the database is closed for any reason. When this occurs, clients are blocked until the timeout occurs.

You configure the timeout by updating the journal-jdbc-network-timeout attribute. The default value is 20000 milliseconds, or 20 seconds.

The following is an example of the management CLI command that sets the journal-jdbc-network-timeout attribute value to 10000 milliseconds, or 10 seconds.

/subsystem=messaging-activemq/server=default:write-attribute(name=journal-jdbc-network-timeout,value=10000)

10.3.5. Configuring HA for Messaging JDBC Persistence Store

The JBoss EAP messaging-activemq subsystem activates the JDBC HA shared store functionality when the broker is configured with a database store type. The broker then uses a shared database table to ensure that the live and backup servers coordinate actions over a shared JDBC journal store.

You can configure HA for JDBC persistence store using the following attributes:

  • journal-node-manager-store-table: Name of the JDBC database table to store the node manager.
  • journal-jdbc-lock-expiration: The time a JDBC lock is considered valid without keeping it alive. The default value is 20000 milliseconds.
  • journal-jdbc-lock-renew-period: The period of the keep alive service of a JDBC lock. The default value is 2000 milliseconds.

The default values are taken into account based on the value of the server’s ha-policy and journal-datasource attributes.

For backward compatibility, you can also specify their values using the respective Artemis-specific system properties:

  • brokerconfig.storeConfiguration.nodeManagerStoreTableName
  • brokerconfig.storeConfiguration.jdbcLockExpirationMillis
  • brokerconfig.storeConfiguration.jdbcLockRenewPeriodMillis

When configured, these system properties have precedence over the corresponding attribute’s default value.

10.4. Managing Messaging Journal Prepared Transactions

You can manage messaging journal prepared transactions using the following management CLI commands.

  • Commit a prepared transaction:

    /subsystem=messaging-activemq/server=default:commit-prepared-transaction(transaction-as-base-64=XID)
  • Roll back a prepared transaction:

    /subsystem=messaging-activemq/server=default:rollback-prepared-transaction(transaction-as-base-64=XID)
  • Show the details of all prepared transactions:

    /subsystem=messaging-activemq/server=default:list-prepared-transactions
    Note

    You can also show the prepared transaction details in HTML format using the list-prepared-transaction-details-as-html operation, or in JSON format using the list-prepared-transaction-details-as-json operation.

10.5. Configuring JBoss EAP Messaging for Zero Persistence

In some situations, zero persistence is required for a messaging system. Zero persistence means that no bindings data, message data, large message data, duplicate id caches, or paging data should be persisted.

To configure the messaging-activemq subsystem to perform zero persistence, set the persistence-enabled parameter to false.

/subsystem=messaging-activemq/server=default:write-attribute(name=persistence-enabled,value=false)
Important

Be aware that if persistence is disabled, but paging is enabled, page files continue to be stored in the location specified by the paging-directory element. Paging is enabled when the address-full-policy attribute is set to PAGE. If full zero persistence is required, be sure to configure the address-full-policy attribute of the address-setting element to use BLOCK, DROP or FAIL.

10.6. Importing and Exporting Journal Data

See the JBoss EAP 7 Migration Guide for information on importing and exporting journal data.