Chapter 13. Duplicate Message Detection

AMQ Broker includes automatic duplicate message detection, which filters out any duplicate messages it receives so you do not have to code your own duplicate detection logic.

Without duplicate detection, a client cannot determine whether a message it sent was successful whenever the target broker or the connection to it fails. For example, if the broker or connection fails before the message was received and processed by the broker, the message never arrives at its address, and the client does not receive a response from the broker due to the failure. On the other hand, if the broker or connection failed after a message was received and processed by the broker, the message is routed correctly, but the client still does not receive a response.

Moreover, using a transaction to determine success does not help in these cases. If the broker or connection fails while the transaction commit is being processed, for example, the client is still unable to determine whether it successfully sent the message.

If the client resends the last message in an effort to correct the assumed failure, the result could be a duplicate message being sent to the address, which could negatively impact your system. Sending a duplicate message could mean that a purchase order is fulfilled twice, for example. Fortunately, {AMQ Broker} provides automatic duplicate messages detection as a way to prevent these kind of issues from happening.

13.1. Using the Duplicate ID Message Property

To enable duplicate message detection provide a unique value for the message property _AMQ_DUPL_ID. When a broker receives a message, it checks if _AMQ_DUPL_ID has a value. If it does, the broker then checks in its memory cache to see if it has already received a message with that value. If a message with the same value is found, the incoming message is ignored.

Procedure

The examples below illustrate how to set the duplicate detection property using a Core JMS Client. Note that for convenience, the clients use the value of the constant org.apache.activemq.artemis.api.core.Message.HDR_DUPLICATE_DETECTION_ID for the name of the duplicate ID property, _AMQ_DUPL_ID.

  • Set the value for _AMQ_DUPL_ID to a unique String.

    Message jmsMessage = session.createMessage();
    String myUniqueID = "This is my unique id";
    message.setStringProperty(HDR_DUPLICATE_DETECTION_ID.toString(), myUniqueID);

13.2. Configuring the Duplicate ID Cache

The broker maintains caches of received values of the _AMQ_DUPL_ID property. Each address has its own distinct cache. The cache is circular and fixed. New entries replace the oldest ones as cache space demands.

Note

Be sure to size the cache appropriately. If a previous message arrived more than id-cache-size messages before the arrival of a new message with the same _AMQ_DUPL_ID, the broker cannot detect the duplicate. This results in both messages being processed by the broker.

Procedure

The example configuration below illustrates how to configure the ID cache by adding elements to BROKER_INSTANCE_DIR/etc/broker.xml.

<configuration>
  <core>
    ...
    <id-cache-size>5000</id-cache-size> 1
    <persist-id-cache>false</persist-id-cache> 2
  </core>
</configuration>
1
The maximum size of the cache is configured by the parameter id-cache-size. The default value is 20000 entries. In the example above, the cache size is set to 5000 entries.
2
Set persist-id-cache to true to have each ID persisted to disk as they are received. The default value is true. In the example above, persistence is disabled by setting the value to false.

13.3. Duplicate Detection and Transactions

Using duplicate detection to move messages between brokers can give you the same once and only once delivery guarantees as using an XA transaction to consume messages, but with less overhead and much easier configuration than using XA.

If you are sending messages in a transaction, you do not have to set _AMQ_DUPL_ID for every message in the transaction, but only in one of them. If the broker detects a duplicate message for any message in the transaction, it ignores the entire transaction.

13.4. Duplicate Detection and Cluster Connections

You can configure cluster connections to insert a duplicate ID for each message they move across the cluster.

Procedure

  • Add the element use-duplicate-detection to the configuration of the desired cluster connection found in BROKER_INSTANCE_DIR/etc/broker.xml. Note that the default value for this parameter is true. In the example below, the element is added to the configuration for the cluster connection my-cluster.

    <configuration>
      <core>
        ...
        <cluster-connection>
           <cluster-connection name="my-cluster"> 2
             <use-duplicate-detection>true</use-duplicate-detection>
             ...
           </cluster-connection>
           ...
        </cluster-connections>
      </core>
    </configuration>

Related Information

For more information about broker clusters, see Section 16.2, “Creating a broker cluster”.