18.12. Duplicate Message Detection

Duplicate message detection allows filtering of duplicate messages without the need of coding the duplicate detection logic within the application. You can configure duplicate message detection in HornetQ.

When a sender(client/server) sends a message to another server there can be a situation where the target server(receiver) or the connection fails after sending the message but before sending a response to the sender indicating that the process was successful. In such situations, it is very difficult for the sender(client) to determine if the message was sent successfully to the intended receiver.

The message send may or may not be successful depending on when the target receiver or connection failed (before or after sending the message). If the sender (client/server) decides to resend the last message it can result in a duplicate message being sent to the address.

HornetQ provides duplicate message detection for messages sent to addresses.

To enable duplicate message detection for sent messages you need to set a special property on the message to a unique value. You can create this value the way you wish but this value must be unique.

When the target server receives this message, it checks if the special property is set. If the property is set then the target server checks its memory cache for a received message with that value of the header. If the server finds any message with the same value of the header it ignores the message sent by a client.

If you are sending messages in a transaction then you do not have to set the property for every message you send in that transaction; you only need to set it once in the transaction. If the server detects a duplicate message for any message in the transaction, then it will ignore the entire transaction.

The name of the property that you set is given by the value of org.hornetq.api.core.HDR_DUPLICATE_DETECTION_ID, which is _HQ_DUPL_ID. The value of this property can be of type byte[] or SimpleString for core API. For Java Messaging Service (JMS) clients, it must be of the type String with a unique value. An easy way of generating a unique id is by generating a UUID.

The following example shows how to set the property for core API:

...     

ClientMessage message = session.createMessage(true);

SimpleString myUniqueID = "This is my unique id";   // Can use a UUID for this

message.setStringProperty(HDR_DUPLICATE_DETECTION_ID, myUniqueID);

...

The following example shows how to set the property for JMS clients:

...     

Message jmsMessage = session.createMessage();

String myUniqueID = "This is my unique id";   // Could use a UUID for this

message.setStringProperty(HDR_DUPLICATE_DETECTION_ID.toString(), myUniqueID);

...

The server maintains caches of received values of the org.hornetq.core.message.impl.HDR_DUPLICATE_DETECTION_ID property sent to each address. Each address maintains its own address cache.

The cache is fixed in terms of size. The maximum size of cache is configured using the parameter id-cache-size in server configuration files (standalone.xml and domain.xml). The default value of this parameter is 2000 elements. If the cache has a maximum size of n elements, then the (n + 1)th ID stored will overwrite the 0th element in the cache.

The caches can also be configured to persist to disk or not. This can be configured using the parameter persist-id-cache in server configuration files (standalone.xml and domain.xml). If this value is set "true" then each ID will be persisted to permanent storage as they are received. The default value for this parameter is true.

Core bridges can be configured to automatically add a unique duplicate ID value (if there isn't already one in the message) before forwarding the message to the target. To configure a core bridge for duplication message detection set the property use-duplicate-detection to "true" in server configuration files (standalone.xml and domain.xml). The default value of this parameter is "true".

Cluster connections internally use core bridges to move messages between nodes of the cluster. To configure a cluster connection for duplicate message detection set the property use-duplicate-detection to "true" in server configuration files (standalone.xml and domain.xml). The default value of this parameter is "true".