Chapter 2. Client Failover

Abstract

Red Hat JBoss A-MQ provides two simple mechanisms for clients to failover to an alternate broker if its active connection fails. The failover protocol relies on a hard coded list of alternative brokers. The discovery protocol relies on discovery agents to provide a list of alternative brokers.

2.1. Failover Protocol

Abstract

The failover protocol provides a simple mechanism for clients to failover to a secondary broker if the primary connection fails. The failover protocol does not require that the brokers be configured as a network of brokers. However, when paired with a network of brokers it can be configured to use dynamic failover locations.
The failover protocol facilitates quick recovery from network failures. When a recoverable network error occurs the protocol catches the error and automatically attempts to reestablish the connection to an alternate broker endpoint without the need to recreate all of the objects associated with the connection. The failover URI is composed of one or more URIs that represent different broker endpoints. By default, the protocol randomly chooses a URI from the list and attempts to establish a network connection to it. If it does not succeed, or if it subsequently fails, a new network connection is established to one of the other URIs in the list.
You can set up failover in one of the following ways:
  • Static—the client is configured with a static list of available URIs
  • Dynamic—the brokers push information about the available broker connections

2.1.1. Static Failover

Overview

In static failover a client is configured to use a failover URI that lists the URIs of the broker connections the client can use. When establishing a connection, the client randomly chooses a URI from the list and attempts to establish a connection to it. If the connection does not succeed, the client chooses a new URI from the list and tries again. The client will continue cycling through the list until a connection attempt succeeds.
If a client's connection to a broker fails after it has been established, the client will attempt to reconnect to a different broker in the list. Once a connection to a new broker is established, the client will continue to use the new broker until the connection to the new broker is severed.

Failover URI

A failover URI is a composite URI that uses one of the following syntaxes:
  • failover:uri1,...,uriN
  • failover:(uri1,...,uriN)?TransportOptions
The URI list(uri1,...,uriN) is a comma-separated list containing the list of broker endpoint URIs to which the client can connect. The transport options(?TransportOptions) specified in the form of a query list, allow you to configure some of the failoiver behaviors.

Transport options

The failover protocol supports the transport options described in Table 2.1, “Failover Transport Options”.

Table 2.1. Failover Transport Options

OptionDefaultDescription
initialReconnectDelay 10Specifies the number of milliseconds to wait before the first reconnect attempt.
maxReconnectDelay 30000Specifies the maximum amount of time, in milliseconds, to wait between reconnect attempts.
useExponentialBackOff trueSpecifies whether to use an exponential back-off between reconnect attempts.
backOffMultiplier 2Specifies the exponent used in the exponential back-off algorithm.
maxReconnectAttempts -1Specifies the maximum number of reconnect attempts before an error is returned to the client. -1 specifies unlimited attempts. 0 specifies that an initial connection attempt is made at start-up, but no attempts to fail over to a secondary broker will be made.
startupMaxReconnectAttempts -1Specifies the maximum number of reconnect attempts before an error is returned to the client on the first attempt by the client to start a connection. -1 specifies unlimited attempts and 0 specifies no retry attempts.
randomize trueSpecifies if a URI is chosen at random from the list. Otherwise, the list is traversed from left to right.
backup falseSpecifies if the protocol initializes and holds a second transport connection to enable fast failover.
timeout -1Specifies the amount of time, in milliseconds, to wait before sending an error if a new connection is not established. -1 specifies an infinite timeout value.
trackMessages falseSpecifies if the protocol keeps a cache of in-flight messages that are flushed to a broker on reconnect.
maxCacheSize 131072Specifies the size, in bytes, used for the cache used to track messages.
updateURIsSupported trueSpecifies whether the client accepts updates to its list of known URIs from the connected broker. Setting this to false inhibits the client's ability to use dynamic failover. See Section 2.1.2, “Dynamic Failover”.
updateURIsURL  Specifies a URL locating a text file that contains a comma-separated list of URIs to use for reconnect in the case of failure. See Section 2.1.2, “Dynamic Failover”.
nested.*  Specifies extra options that can be added to the nested URLs. see Example 2.4, “Adding Options for Nested Failover URL”
warnAfterReconnectAttempts 10Specifies to log a warning for no connection after every N number of attemps to reconnect. To disable the feature, set the value to <=0

Example

Example 2.1, “Simple Failover URI” shows a failover URI that can connect to one of two message brokers.

Example 2.1. Simple Failover URI

failover:(tcp://localhost:61616,tcp://remotehost:61616)?initialReconnectDelay=100

2.1.2. Dynamic Failover

Abstract

Dynamic failover combines the failover protocol and a network of brokers to allow a broker to supply its clients with a list of broker connections to which the clients can failover.

Overview

Dynamic failover combines the failover protocol and a network of brokers to allow a broker to supply its clients with a list of broker connections to which the clients can failover. Clients use a failover URI to connect to a broker and the broker dynamically updates the clients' list of available URIs. The broker updates its clients' failover lists with the URIs of the other brokers in its network of brokers that are currently running. As new brokers join, or exit, the network of brokers, the broker will adjust its clients' failover lists.
From a connectivity point of view, dynamic failover works the same as static failover. A client randomly chooses a URI from the list provided in its failover URI. Once that connection is established, the list of available brokers is updated. If the original connection fails, the client will randomly select a new URI from its dynamically generated list of brokers. If the new broker is configured for to supply a failover list, the new broker will update the client's list.

Set-up

To use dynamic failover you must configure both the clients and brokers used by your application. The following must be configured:
  • The client's must be configured to use the failover protocol when connecting with its broker.
  • The client must be configured to accept URI lists from a broker.
  • The brokers must be configured to form a network of brokers.
  • The broker's transport connector must set the failover properties needed to update its consumers.

Client-side configuration

The client-side configuration for using dynamic failover is nearly identical to the client-side configuration for using static failover. The differences include:
  • The failover URI can consist of a single broker URI.
  • The updateURIsSupported option must be set to true.
  • The updateURIsURL option should be set so that the transport can failover to a new broker when none of the broker's in the dynamically supplied list are available.
See the section called “Failover URI” and the section called “Transport options” for more information about using failover URIs.

Broker-side configuration

Important
Brokers should never use a failover URI to configure a transport connector. The failover protocol does not support listening for incoming messages.
Configuring a broker to participate in dynamic failover requires two things:
  • The broker must be configured to participate in a network of brokers that can be available for failovers.
    See "Using Networks of Brokers" for information about setting up a network of brokers.
  • The broker's transport connector must set the failover properties needed to update its consumers.
Table 2.2, “Broker-side Failover Properties” describes the broker-side properties that can be used to configure a failover cluster. These properties are attributes on the broker's transportConnector element.

Table 2.2. Broker-side Failover Properties

PropertyDefaultDescription
updateClusterClients falseSpecifies if the broker passes information to connected clients about changes in the topology of the broker cluster.
updateClusterClientsOnRemovefalseSpecifies if clients are updated when a broker is removed from the cluster.
rebalanceClusterClients falseSpecifies if connected clients are asked to rebalance across the cluster whenever a new broker joins.
updateClusterFilter Specifies a comma-separated list of regular expression filters, which match against broker names to select the brokers that belong to the failover cluster.

Example

Example 2.2, “Broker for Dynamic Failover” shows the configuration for a broker that participates in dynamic failover.

Example 2.2. Broker for Dynamic Failover

<beans ... >
  <broker>
    ...
    <networkConnectors>
1      <networkConnector uri="multicast://default" />
    </networkConnectors>
    ...
    <transportConnectors>
      <transportConnector name="openwire"
  uri="tcp://0.0.0.0:61616"
2         discoveryUri="multicast://default"
3         updateClusterClients="true"
4         updateClusterFilter="A.*,B.*" />  
    </transportConnectors>
    ...
  </broker>
</beans>
The configuration in Example 2.2, “Broker for Dynamic Failover” does the following:
1
Creates a network connector that connects to any discoverable broker that uses the multicast transport.
2
Makes the broker discoverable by other brokers over the multicast protocol.
3
Makes the broker update the list of available brokers for clients that connect using the failover protocol.
Note
Clients will only be updated when new brokers join the cluster, not when a broker leaves the cluster.
4
Creates a filter so that only those brokers whose names start with the letter A or the letter B are considered to belong to the failover cluster.
Example 2.3, “Failover URI for Connecting to a Failover Cluster” shows the URI for a client that uses the failover protocol to connect to the broker and its cluster.

Example 2.3. Failover URI for Connecting to a Failover Cluster

failover:(tcp://0.0.0.0:61616)?initialReconnectDelay=100
Example 2.4, “Adding Options for Nested Failover URL” shows the options that are passed to a nested URL to detect dead connections in the failover protocol.

Example 2.4. Adding Options for Nested Failover URL

failover:(tcp://host01:61616,tcp://host02:61616,tcp://host03:61616)?nested.wireFormat.maxInactivityDuration=1000

2.1.3. Priority Backup

The priority backup feature allows clients to automatically reconnect to local urls. If the setup has brokers in both local and remote networks, you would want the client to connect to the local broker. In this scenario the priority backup feature is useful.

Example 2.5. Priority backup

	failover:(tcp://local:61616,tcp://remote:61616)?randomize=false&priorityBackup=true
If the above url is used for the client, the client tries to connect to the local broker. If local broker fails, the client will fail over to the remote one. But as the priorityBackup parameter is used, the client will constantly try to reconnect to the local broker and the client switches to the local broker without any intervention.

Example 2.6. Priority URI

In case of multiple local urls the priority is assigned to the url by using priorityURIs parameter
	failover:(tcp://local1:61616,tcp://local2:61616,tcp://remote:61616)?randomize=false&priorityBackup=true&priorityURIs=tcp://local1:61616,tcp://local2:61616
The client prioritizes the brokers in order (first local1 then local2) and reconnects to them in that order according to their availability.