Chapter 8. Handling large messages

Clients might send large messages that can exceed the size of the broker’s internal buffer, causing unexpected errors. To prevent this situation, you can configure the broker to store messages as files when the messages are larger than a specified minimum value. Handling large messages in this way means that the broker does not hold the messages in memory. Instead, you specify a directory on disk or in a database table in which the broker stores large message files.

When the broker stores a message as a large message, the queue retains a reference to the file in the large messages directory or database table.

Large message handling is available for the Core Protocol, AMQP, OpenWire and STOMP protocols.

For the Core Protocol and OpenWire protocols, clients specify the minimum large message size in their connection configurations. For the AMQP and STOMP protocols, you specify the minimum large message size in the acceptor defined for each protocol in the broker configuration.

Note

It is recommended that you do not use different protocols for producing and consuming large messages. To do this, the broker might need to perform several conversions of the message. For example, say that you want to send a message using the AMQP protocol and receive it using OpenWire. In this situation, the broker must first read the entire body of the large message and convert it to use the Core protocol. Then, the broker must perform another conversion, this time to the OpenWire protocol. Message conversions such as these cause significant processing overhead on the broker.

The minimum large message size that you specify for any of the preceding protocols is affected by system resources such as the amount of disk space available, as well as the sizes of the messages. It is recommended that you run performance tests using several values to determine an appropriate size.

The procedures in this section show how to:

  • Configure the broker to store large messages
  • Configure acceptors for the AMQP and STOMP protocols for large message handling

This section also links to additional resources about configuring AMQ Core Protocol and AMQ OpenWire JMS clients to work with large messages.

8.1. Configuring the broker for large message handling

The following procedure shows how to specify a directory on disk or a database table in which the broker stores large message files.

Procedure

  1. Open the <broker-instance-dir>/etc/broker.xml configuration file.

  2. Specify where you want the broker to store large message files.

    1. If you are storing large messages on disk, add the large-messages-directory parameter within the core element and specify a file system location. For example: 

      <configuration>
  <core>
    ...
    <large-messages-directory>/path/to/my-large-messages-directory</large-messages-directory>
    ...
  </core>
</configuration>
      Note

      If you do not explicitly specify a value for large-messages-directory, the broker uses a default value of <broker-instance-dir>/data/largemessages

    2. If you are storing large messages in a database table, add the large-message-table parameter to the database-store element and specify a value. For example: 

      <store>
  <database-store>
    ...
    <large-message-table>MY_TABLE</large-message-table>
    ...
  </database-store>
</store>
      Note

      If you do not explicitly specify a value for large-message-table, the broker uses a default value of LARGE_MESSAGE_TABLE.

Additional resources

8.2. Configuring AMQP acceptors for large message handling

The following procedure shows how to configure an AMQP acceptor to handle an AMQP message larger than a specified size as a large message.

Procedure

  1. Open the <broker-instance-dir>/etc/broker.xml configuration file.

    The default AMQP acceptor in the broker configuration file looks as follows: 

    <acceptors>
    ...
    <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300</acceptor>
    ...
</acceptors>

  2. In the default AMQP acceptor (or another AMQP acceptor that you have configured), add the amqpMinLargeMessageSize property and specify a value. For example: 

    <acceptors>
    ...
    <acceptor name="amqp">tcp://0.0.0.0:5672?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=AMQP;useEpoll=true;amqpCredits=1000;amqpLowCredits=300;amqpMinLargeMessageSize=204800</acceptor>
    ...
</acceptors>

    In the preceding example, the broker is configured to accept AMQP messages on port 5672. Based on the value of amqpMinLargeMessageSize, if the acceptor receives an AMQP message with a body larger than or equal to 204800 bytes (that is, 200 kilobytes), the broker stores the message as a large message. If you do not explicitly specify a value for this property, the broker uses a default value of 102400 (that is, 100 kilobytes).

Note
  • If you set amqpMinLargeMessageSize to -1, large message handling for AMQP messages is disabled.
  • If the broker receives a persistent AMQP message that does not exceed the value of amqpMinLargeMessageSize, but which does exceed the size of the messaging journal buffer (specified using the journal-buffer-size configuration parameter), the broker converts the message to a large Core Protocol message, before storing it in the journal.

8.3. Configuring STOMP acceptors for large message handling

The following procedure shows how to configure a STOMP acceptor to handle a STOMP message larger than a specified size as a large message.

Procedure

  1. Open the <broker-instance-dir>/etc/broker.xml configuration file.

    The default AMQP acceptor in the broker configuration file looks as follows: 

    <acceptors>
    ...
    <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true</acceptor>
    ...
</acceptors>

  2. In the default STOMP acceptor (or another STOMP acceptor that you have configured), add the stompMinLargeMessageSize property and specify a value. For example: 

    <acceptors>
    ...
    <acceptor name="stomp">tcp://0.0.0.0:61613?tcpSendBufferSize=1048576;tcpReceiveBufferSize=1048576;protocols=STOMP;useEpoll=true;stompMinLargeMessageSize=204800</acceptor>
    ...
</acceptors>

In the preceding example, the broker is configured to accept STOMP messages on port 61613. Based on the value of stompMinLargeMessageSize, if the acceptor receives a STOMP message with a body larger than or equal to 204800 bytes (that is, 200 kilobytes), the broker stores the message as a large message. If you do not explicitly specify a value for this property, the broker uses a default value of 102400 (that is, 100 kilobytes).

Note

To deliver a large message to a STOMP consumer, the broker automatically converts the message from a large message to a normal message before sending it to the client. If a large message is compressed, the broker decompresses it before sending it to STOMP clients.

8.4. Large messages and Java clients

There are two options available to Java developers who are writing clients that use large messages.

One option is to use instances of InputStream and OutputStream. For example, a FileInputStream can be used to send a message taken from a large file on a physical disk. A FileOutputStream can then be used by the receiver to stream the message to a location on its local file system.

Another option is to stream a JMS BytesMessage or StreamMessage directly. For example: 

BytesMessage rm = (BytesMessage)cons.receive(10000);
byte data[] = new byte[1024];
for (int i = 0; i < rm.getBodyLength(); i += 1024)
{
   int numberOfBytes = rm.readBytes(data);
   // Do whatever you want with the data
}

Additional resources