LibraryPrintFeedback

Fault Tolerant Messaging

Version 7.1

December 2012
Trademark Disclaimer
Third Party Acknowledgements

Updated: 08 Jan 2014

Table of Contents

1. Introduction
2. Client Failover
Failover Protocol
Static Failover
Dynamic Failover
Discovery Protocol
Discovery URI
Discovery Agents
Fuse Fabric Discovery Agent
Static Discovery Agent
Multicast Discovery Agent
Zeroconf Discovery Agent
3. Master/Slave
Shared File System Master/Slave
Shared JDBC Master/Slave
4. Master/Slave and Broker Networks
Index

List of Figures

3.1. Shared File System Initial State
3.2. Shared File System after Master Failure
3.3. Shared File System after Master Restart
3.4. JDBC Master/Slave Initial State
3.5. JDBC Master/Slave after Master Failure
3.6. JDBC Master/Slave after Master Restart
4.1. Master/Slave Groups on Two Host Machines
4.2. Broker Network Consisting of Host Pairs

List of Tables

2.1. Failover Transport Options
2.2. Broker-side Failover Properties
2.3. Dynamic Discovery Protocol Options

List of Examples

2.1. Simple Failover URI
2.2. Broker for Dynamic Failover
2.3. Failover URI for Connecting to a Failover Cluster
2.4. Discovery URI
2.5. Discovery Protocol URI
2.6. Injecting Transport Options into a Discovered Transport
2.7. Enabling a Discovery Agent on a Broker
2.8. Fuse Fabric Discovery Agent URI Format
2.9. Client Connection URL using Fuse Fabric Discovery
2.10. Static Discovery Agent URI Format
2.11. Discovery URI using the Static Discovery Agent
2.12. Multicast Discovery Agent URI Format
2.13. Enabling a Multicast Discovery Agent on a Broker
2.14. Client Connection URL using Multicast Discovery
2.15. Zeroconf Discovery Agent URI Format
2.16. Enabling a Multicast Discovery Agent on a Broker
2.17. Client Connection URL using Zeroconf Discovery
3.1. Shared File System Broker Configuration
3.2. Client URL for a Shared File System Master/Slave Group
3.3. JDBC Master/Slave Broker Configuration
3.4. Client URL for a Shared JDBC Master/Slave Group
4.1. Network Connector to a Master/Slave Group

The failover protocol facilitates quick recovery from network failures. When a recoverable network error occurs the protocol catches the error and automatically attempts to reestablish the connection to an alternate broker endpoint without the need to recreate all of the objects associated with the connection. The failover URI is composed of one or more URIs that represent different broker endpoints. By default, the protocol randomly chooses a URI from the list and attempts to establish a network connection to it. If it does not succeed, or if it subsequently fails, a new network connection is established to one of the other URIs in the list.

You can set up failover in one of the following ways:

  • Static—the client is configured with a static list of available URIs

  • Dynamic—the brokers push information about the available broker connections

The failover protocol supports the transport options described in Table 2.1.

Table 2.1. Failover Transport Options

OptionDefaultDescription
initialReconnectDelay 10Specifies the number of milliseconds to wait before the first reconnect attempt.
maxReconnectDelay 30000Specifies the maximum amount of time, in milliseconds, to wait between reconnect attempts.
useExponentialBackOff trueSpecifies whether to use an exponential back-off between reconnect attempts.
backOffMultiplier 2Specifies the exponent used in the exponential back-off algorithm.
maxReconnectAttempts -1Specifies the maximum number of reconnect attempts before an error is returned to the client. -1 specifies unlimited attempts. 0 specifies that an initial connection attempt is made at start-up, but no attempts to failover over to a secondary broker will be made.
startupMaxReconnectAttempts 0Specifies the maximum number of reconnect attempts before an error is returned to the client on the first attempt by the client to start a connection. 0 specifies unlimited attempts.
randomize trueSpecifies if a URI is chosen at random from the list. Otherwise, the list is traversed from left to right.
backup falseSpecifies if the protocol initializes and holds a second transport connection to enable fast failover.
timeout -1Specifies the amount of time, in milliseconds, to wait before sending an error if a new connection is not established. -1 specifies an infinite timeout value.
trackMessages falseSpecifies if the protocol keeps a cache of in-flight messages that are flushed to a broker on reconnect.
maxCacheSize 131072Specifies the size, in bytes, used for the cache used to track messages.
updateURIsSupported trueSpecifies whether the client accepts updates to its list of known URIs from the connected broker. Setting this to false inhibits the client's ability to use dynamic failover. See Dynamic Failover.
updateURIsURL  Specifies a URL locating a text file that contains a comma-separated list of URIs to use for reconnect in the case of failure. See Dynamic Failover.

Example 2.2 shows the configuration for a broker that participates in dynamic failover.

Example 2.2. Broker for Dynamic Failover

<beans ... >
  <broker>
    ...
    <networkConnectors>
1      <networkConnector uri="multicast://default" />
    </networkConnectors>
    ...
    <transportConnectors>
      <transportConnector name="openwire"
          uri="tcp://0.0.0.0:61616"
2         discoveryUri="multicast://default"
3         updateClusterClients="true"
4         updateClusterFilter="*A*,*B*" />  
    </transportConnectors>
    ...
  </broker>
</beans>

The configuration in Example 2.2 does the following:

1

Creates a network connector that connects to any discoverable broker that uses the multicast transport.

2

Makes the broker discoverable by other brokers over the multicast protocol.

3

Makes the broker update the list of available brokers for clients that connect using the failover protocol.

[Note]Note

Clients will only be updated when new brokers join the cluster, not when a broker leaves the cluster.

4

Creates a filter so that only those brokers whose names start with the letter A or the letter B are considered to belong to the failover cluster.

Example 2.3 shows the URI for a client that uses the failover protocol to connect to the broker and its cluster.


The failover protocol provides a lot of control over the brokers to which a client can connect. Using dynamic failover adds some ability to make the broker list more transparent. However, it has weaknesses. It requires that you know the address of at least one broker and that an initial broker is active when the client starts up. Using dynamic failover also requires that all of the brokers being used for failover are configured in a network of brokers.

Fuse MQ Enterprise's discovery protocol offers an alternative method for dynamically generating a list of brokers that are available for client failover. The protocol allows brokers to advertise their availability and for clients to dynamically discover them. This is accomplished using two pieces:

Example 2.4 shows the syntax for a discovery URI.


DiscoveryAgentUri is URI for the discovery agent used to build up the list of available brokers. Discovery agents are described in Discovery Agents.

The options, ?Options, are specified in the form of a query list. The discovery options are described in Table 2.3. You can also inject transport options as described in Setting options on the discovered transports.

[Tip]Tip

If no options are required, you can drop the parentheses from the URI. The resulting URI would take the form discovery://DiscoveryAgentUri

A discovery agent is a mechanism that advertises available brokers to clients and other brokers. When a client, or broker, using a discovery URI starts up it will look for any brokers that are available using the specified discovery agent. The clients will update their lists periodically using the same mechanism.

How a discovery agent learns about the available brokers varies between agents. Some agents use a static list, some use a third party registry, and some rely on the brokers to provide the information. For discovery agents that rely on the brokers for information, it is necessary to enable the discovery agent in the message broker configuration. For example, to enable the multicast discovery agent on an Openwire endpoint, you edit the relevant transportConnector element as shown in Example 2.7.


Where the discoveryUri attribute on the transportConnector element is initialized to multicast://default.

[Tip]Tip

If a broker uses multiple transport connectors, you need to configure each transport connector to use a discovery agent individually. This means that different connectors can use different discovery mechanisms or that one or more of the connectors can be indiscoverable.

Fuse MQ Enterprise currently supports the following discovery agents:

The zeroconf discovery agent is derived from Apple’s Bonjour Networking technology, which defines the zeroconf protocol as a mechanism for discovering services on a network. Fuse MQ Enterprise bases its implementation of the zeroconf discovery agent on JmDSN, which is a service discovery protocol that is layered over IP/multicast and is compatible with Apple Bonjour.

The agent requires that each broker you want to advertise is configured to use a multicast discovery agent to publish its details to a multicast group. Clients using the zeroconf agent as part of the discovery URI they use for connecting to a broker will use the agent to receive the list of available brokers advertising in the specified multicast group.

[Important]Important

Your local network (LAN) must be configured to use IP/multicast for the zeroconf agent to work.

A master/slave group consists of two or more brokers where one master broker is active and one or more slave brokers are on hot standby, ready to take over whenever the master fails or shuts down. All of the brokers store the message and event data processed by the master broker. So, when one of the slaves takes over as the new master broker the integrity of the messaging system is guaranteed.

Fuse MQ Enterprise supports two master/slave broker configurations:

  • Shared file system—the master and the slaves use a common persistence store that is located on a shared file system

  • Shared JDBC database—the masters and the slaves use a common JDBC persistence store

Figure 3.1 shows the initial state of a shared file system master/slave group. When all of the brokers are started, one of them grabs the exclusive lock on the broker data store and becomes the master. All of the other brokers remain slaves and pause while waiting for the exclusive lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 3.2 shows the state of the master/slave group after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the data store frees up and another broker grabs the lock and gets promoted to master.


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.

Figure 3.4 shows the initial state of a JDBC master/slave group. When all of the brokers are started, one of them grabs the mutex lock on the database table and becomes the master. All of the other brokers become slaves and pause while waiting for the lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 3.5 shows the state of the group after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the database table frees up and another broker grabs the lock and gets promoted to master.


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.

In a JDBC master/slave configuration, there is nothing special to distinguish a master broker from the slave brokers. The membership of a particular master/slave group is defined by the fact that all of the brokers in the cluster use the same JDBC persistence layer and store their data in the same database tables.

There is one important requirement when configuring the JDBC persistence adapter for use in a shared database master/slave cluster. You must use the direct JDBC persistence adapter. This is because the journal used by the journaled JDBC persistence adapter is not replicated and batch updates are used to sync with the JDBC store. Therefore it is not possible to guarantee that the latest updates are on the shared JDBC store.

Example 3.3 shows the configuration used be a master/slave group that stores the shared broker data in an Oracle database.


D

discovery agent
Fuse Fabric, Fuse Fabric Discovery Agent
multicast, Multicast Discovery Agent
static, Static Discovery Agent
zeroconf, Zeroconf Discovery Agent
discovery protocol
backOffMultiplier, Transport options
initialReconnectDelay, Transport options
maxReconnectAttempts, Transport options
maxReconnectDelay, Transport options
URI, URI syntax
useExponentialBackOff, Transport options
discovery URI, URI syntax
discovery://, URI syntax
discoveryUri, Configuring a broker, Configuring a broker
dynamic failover, Dynamic Failover
broker configuration, Broker-side configuration
client configuration, Client-side configuration

F

fabric://, URI
failover, Failover Protocol
backOffMultiplier, Transport options
backup, Transport options
broker properties, Broker-side configuration
dynamic, Dynamic Failover
initialReconnectDelay, Transport options
maxCacheSize, Transport options
maxReconnectAttempts, Transport options
maxReconnectDelay, Transport options
randomize, Transport options
startupMaxReconnectAttempts, Transport options
static, Static Failover
timeout, Transport options
trackMessages, Transport options
updateURIsSupported, Transport options
updateURIsURL, Transport options
useExponentialBackOff, Transport options
failover URI, Failover URI
transport options, Transport options
failover://, Failover URI
Fuse Fabric discovery agent
URI, URI

J

jdbcPersistenceAdapter, Configuring the brokers

M

master broker
reintroduction
shared file system, Reintroducing a failed node
shared JDBC, Reintroducing a failed node
master/slave
broker networks, Configuring the connection to a master/slave group
network of brokers, Configuring the connection to a master/slave group
masterslave, Configuring the connection to a master/slave group
multicast discovery agent
broker configuration, Configuring a broker
URI, URI
multicast://, URI

S

shared file system master/slave
advantages, Overview
broker configuration, Configuring the brokers, Configuring the brokers
client configuration, Configuring the clients
disadvantages, Overview
incompatible SANs, File locking requirements
initial state, Initial state
master failure, State after failure of the master
NFSv3, File locking requirements
NFSv4, File locking requirements
OCFS2, File locking requirements
recovery strategies, State after failure of the master
reintroducing a node, Reintroducing a failed node
shared JDBC master/slave
advantages, Overview
client configuration, Configuring the clients
disadvantages, Overview
initial state, Initial state
master failure, After failure of the master
recovery strategies, After failure of the master
reintroducing a node, Reintroducing a failed node
static discovery agent
URI, Using the agent
static failover, Static Failover
static://, Using the agent

T

transportConnector
discoveryUri, Configuring a broker, Configuring a broker

Z

zeroconf discovery agent
broker configuration, Configuring a broker
URI, URI
zeroconf://, URI