4.2. Flow Control

Overview

Memory limits, when configured, prevent the broker from running out of resources. The default behavior, when a limit is reached, is to block the sending thread in the broker, which blocks the destination and the connection.
Producer flow control is a mechanism that pushes the blocking behavior onto the client, so that the producer thread blocks if the broker has no space. With producer flow control, the producer has a send window that is dependent on broker memory. When the send window is full, it blocks on the client.

Flow control enabled

Figure 4.1, “Broker with Flow Control Enabled” gives an overview of what happens to a messaging application when flow control is enabled.

Figure 4.1. Broker with Flow Control Enabled

Broker with Flow Control Enabled
If a consumer is very slow at acknowledging messages (or stops acknowledging messages altogether), the broker continues to dispatch messages to the consumer until it reaches the prefetch limit, after which the messages start to back up on the broker. Assuming the producer continues to produce lots of messages and the consumer continues to be very slow, the broker will start to run short of memory resources as it holds on to pending messages for the consumer.
When the consumed memory resources start to approach their limit (as defined either by the per-destination memory limit or the per-broker memory limit), the flow control mechanism activates automatically in order to protect the broker resources. The broker sends a message to the producer asking it either to slow down or to stop sending messages to the broker. This protects the broker from running out of memory (and other) resources.
Note
There are some differences in behavior between a persistent broker and a non-persistent broker. If a broker is persistent, pending messages are stored to disk, but flow control can still be triggered if the amount of memory used by a cursor approaches its limit (see Section 6.3, “vmCursor on Destination” for more details about cursors).

Flow control disabled

While it is generally a good idea to enable flow control in a broker, there are some scenarios for which it is unsuitable. Consider the scenario where a producer dispatches messages that are consumed by multiple consumers (for example, consuming from a topic), but one of the consumers could fail without the broker becoming aware of it right away. This scenario is shown in Figure 4.2, “Broker with Flow Control Disabled”.

Figure 4.2. Broker with Flow Control Disabled

Broker with Flow Control Disabled
Because the slow consumer remains blocked for a very long time (possibly indefinitely), after flow control kicks in, the producer also ceases producing messages for a very long time (possibly indefinitely). This is an undesirable outcome, because there are other active consumers interested in the messages coming from the producer and they are now being unfairly deprived of those messages.
In this case, it is better to turn off flow control in the broker, so that the producer can continue sending messages to the other interested consumers. The broker now resorts to an alternative strategy to avoid running out of memory: the broker writes any pending messages for the slow consumer to a temporary file. Ultimately, this scenario is resolved in one of two ways: either the slow consumer revives again and consumes all of the messages from the temporary file; or the broker determines that the slow consumer has died and the temporary file can be discarded.

Discarding messages

By default, when flow control is disabled and the relevant memory limit is reached, the slow consumer's messages are backed up in a temporary file. An alternative strategy for coping with the excess messages, however, is simply to discard the slow consumer's messages when they exceed a certain limit (where the oldest messages are discarded first). This strategy avoids the overhead of writing to a temporary file.
For example, if the slow consumer is receiving a feed of real-time stock quotes, it might be acceptable to discard older, undelivered stock quotes, because the information becomes stale.
To enable discarding of messages, define a pending message limit strategy in the broker configuration. For example, to specify that the backlog of messages stored in the broker (not including the prefetched messages) cannot exceed 10 for any topics that match the PRICES.> pattern (that is, topic names prefixed by PRICES.), configure the broker as follows: 
<beans ... >
  <broker ...>
    <!--  lets define the dispatch policy -->
    <destinationPolicy>
      <policyMap>
        <policyEntries>
          <policyEntry topic="PRICES.>">

            <!-- lets force old messages to be discarded for slow consumers -->
            <pendingMessageLimitStrategy>
              <constantPendingMessageLimitStrategy limit="10"/>
            </pendingMessageLimitStrategy>

          </policyEntry>
          ...
        </policyEntries>
      </policyMap>
    </destinationPolicy>
  </broker>
</beans>
For more details about how to configure pending message limit strategies, see http://activemq.apache.org/slow-consumer-handling.html.

How to turn off flow control

Flow control can be turned off by setting a destination policy in the broker's configuration. In particular, flow control can be enabled or disabled on individual destinations or groups of destinations (using wildcards). To disable flow control, set the producerFlowControl attribute to false on a policyEntry element.
For example, to configure a broker to disable flow control for all topic destinations starting with FOO., insert a policy entry like the following into the broker's configuration: 
<broker ... >
  ...
  <destinationPolicy>
    <policyMap>
      <policyEntries>
        <policyEntry topic="FOO.>" producerFlowControl="false"/>
        ...
      </policyEntries>
    </policyMap>
  </destinationPolicy>
  ...
</broker>

Defining the memory limits

When flow control is enabled, the point at which flow control activates depends on the defined memory limits, which can be specified at either of the following levels of granularity:
  • Per-broker—to set global memory limits on a broker, define a systemUsage element as a child of the broker element, as follows: 
    <broker>
  ...
  <systemUsage>
    <systemUsage>
      <memoryUsage>
        <memoryUsage limit="64 mb" />
      </memoryUsage>
      <storeUsage>
        <storeUsage limit="100 gb" />
      </storeUsage>
      <tempUsage>
        <tempUsage limit="10 gb" />
      </tempUsage>
    </systemUsage>
  </systemUsage>
  ...
</broker>
    Where the preceding sample specifies three distinct memory limits, as follows:
    • memoryUsage—specifies the maximum amount of memory allocated to the broker.
    • storeUsage—for persistent messages, specifies the maximum disk storage for the messages.
      Note
      In certain scenarios, the actual disk storage used by JBoss A-MQ can exceed the specified limit. For this reason, it is recommended that you set storeUsage to about 70% of the intended maximum disk storage.
    • tempUsage—for temporary messages, specifies the maximum amount of memory.
    The values shown in the preceding example are the defaults.
  • Per-destination—to set a memory limit on a destination, set the memoryLimit attribute on the policyEntry element. The value of memoryLimit can be a string, such as 10 MB or 512 KB. For example, to limit the amount of memory on the FOO.BAR queue to 10 MB, define a policy entry like the following: 
    <policyEntry queue="FOO.BAR" memoryLimit="10 MB"/>

Making a producer aware of flow control

When a producer is subject to flow control, the default behavior is for the send() operation to block, until enough memory is freed up in the broker for the producer to resume sending messages. If you want the producer to be made aware of the fact that the send() operation is blocked due to flow control, you can enable either of the following attributes on the systemUsage element:
sendFailIfNoSpace
If true, the broker immediately returns an error when flow control is preventing producer send() operations; otherwise, revert to default behavior.
sendFailIfNoSpaceAfterTimeout
Specifies a timeout in units of milliseconds. When flow control is preventing producer send() operations, the broker returns an error, after the specified timeout has elapsed.
The following example shows how to configure the broker to return an error to the producer immediately, whenever flow control is blocking the producer send() operations: 
<broker>
  ...
  <systemUsage>
    <systemUsage sendFailIfNoSpace="true">
      <memoryUsage>
        <memoryUsage limit="64 mb" />
      </memoryUsage>
      ...
    </systemUsage>
  </systemUsage>
  ...
</broker>