Fuse Message Broker

Fault Tolerant Messaging

Version 5.5

Febuary 2012
Trademark Disclaimer
Third Party Acknowledgements

Updated: 27 Mar 2014

Table of Contents

1. Introduction
2. Client Failover
Failover Protocol
Static Failover
Dynamic Failover
Discovery Protocol
Discovery Agents
Static Discovery Agent
Multicast Discovery Agent
Zeroconf Discovery Agent
Dynamic Discovery Protocol
3. Master/Slave
Shared Nothing Master/Slave
Shared File System Master/Slave
Shared JDBC Master/Slave
4. Master/Slave and Broker Networks
Index

List of Figures

3.1. Shared Nothing Master/Slave Group Initial State
3.2. Shared Nothing Master/Slave Group after Master Failure
3.3. Shared File System Initial State
3.4. Shared File System after Master Failure
3.5. Shared File System after Master Restart
3.6. JDBC Master/Slave Initial State
3.7. JDBC Master/Slave after Master Failure
3.8. JDBC Master/Slave after Master Restart
4.1. Master/Slave Groups on Two Host Machines
4.2. Broker Network Consisting of Host Pairs

List of Tables

2.1. Failover Transport Options
2.2. Broker-side Failover Properties
2.3. Dynamic Discovery Protocol Options
3.1. Configuration Options for a Master in a Shared Nothing Master/Slave Group
3.2. Attributes for Configuring the Master Connector Service
3.3. Attributes for Configuring a Master Connector on the Broker

List of Examples

2.1. Simple Failover URI
2.2. Broker for Dynamic Failover
2.3. Failover URI for Connecting to a Failover Cluster
2.4. Enabling a Discovery Agent on a Broker
2.5. Static Discovery Agent URI Format
2.6. Discovery URL using the Static Discovery Agent
2.7. Multicast Discovery Agent URI Format
2.8. Enabling a Multicast Discovery Agent on a Broker
2.9. Client Connection URL using Multicast Discovery
2.10. Zeroconf Discovery Agent URI Format
2.11. Enabling a Multicast Discovery Agent on a Broker
2.12. Client Connection URL using Zeroconf Discovery
2.13. Dynamic Discovery URI
2.14. Discovery Protocol URI
2.15. Injecting Transport Options into a Discovered Transport
3.1. Master Configuration for Shared Nothing Master/Slave Group
3.2. Configuring the Master Connector as a Service
3.3. Configuring the Master Connector Directly
3.4. URI for Connecting to a Master/Slave Cluster
3.5. Shared File System Broker Configuration
3.6. Client URL for a Shared File System Master/Slave Group
3.7. JDBC Master/Slave Broker Configuration
3.8. Client URL for a Shared JDBC Master/Slave Group
4.1. Network Connector to a Master/Slave Group

The failover protocol facilitates quick recovery from network failures. When a recoverable network error occurs the protocol catches the error and automatically attempts to reestablish the JMS connection to an alternate broker endpoint without the need to recreate all of the JMS objects associated with the JMS connection. The failover URI is composed of one or more URIs that represent different broker endpoints. By default, the protocol randomly chooses a URI from the list and attempts to establish a network connection to it. If it does not succeed, or if it subsequently fails, a new network connection is established to one of the other URIs in the list.

For true high-availability and fail over capabilities, you will need to set up your brokers in a network of brokers. See Using Networks of Brokers.

You can set up failover in one of the following ways:

  • Static—the client is configured with a static list of available URIs

  • Dynamic—the brokers push information about the available broker connections

The failover protocol supports the transport options described in Table 2.1.

Table 2.1. Failover Transport Options

OptionDefaultDescription
initialReconnectDelay 10Specifies the number of milliseconds to wait before the first reconnect attempt.
maxReconnectDelay 30000Specifies the maximum amount of time, in milliseconds, to wait between reconnect attempts.
useExponentialBackOff trueSpecifies whether to use an exponential back-off between reconnect attempts.
backOffMultiplier 2Specifies the exponent used in the exponential back-off algorithm.
maxReconnectAttempts -1Specifies the maximum number of reconnect attempts before an error is returned to the client. -1 specifies unlimited attempts. 0 specifies that an initial connection attempt is made at start-up, but no attempts to failover over to a secondary broker will be made.
startupMaxReconnectAttempts 0Specifies the maximum number of reconnect attempts before an error is returned to the client on the first attempt by the client to start a connection. 0 specifies unlimited attempts.
randomize trueSpecifies if a URI is chosen at random from the list. Otherwise, the list is traversed from left to right.
backup falseSpecifies if the protocol initializes and holds a second transport connection to enable fast failover.
timeout -1Specifies the amount of time, in milliseconds, to wait before sending an error if a new connection is not established. -1 specifies an infinite timeout value.
trackMessages falseSpecifies if the protocol keeps a cache of in-flight messages that are flushed to a broker on reconnect.
maxCacheSize 131072Specifies the size, in bytes, used for the cache used to track messages.
updateURIsSupported trueSpecifies whether the client accepts updates to its list of known URIs from the connected broker. Setting this to false inhibits the client's ability to use dynamic failover. See Dynamic Failover.

Important

Brokers should never use a failover uri to configure a transport connector. The failover protocol does not support listening for incoming messages.

Configuring a broker to participate in dynamic failover requires two things:

Table 2.2 describes the broker-side properties that can be used to configure a failover cluster. These properties are attributes on the broker's transportConnector element.


Example 2.2 shows the configuration for a broker that participates in dynamic failover.

Example 2.2. Broker for Dynamic Failover

<beans ... >
  <broker>
    ...
    <networkConnectors>
1      <networkConnector uri="multicast://default" />
    </networkConnectors>
    ...
    <transportConnectors>
      <transportConnector name="openwire"
          uri="tcp://0.0.0.0:61616"
2         discoveryUri="multicast://default"
3         updateClusterClients="true"
4         updateClusterFilter="*A*,*B*" />  
    </transportConnectors>
    ...
  </broker>
</beans>

The configuration in Example 2.2 does the following:

1

Creates a network connector that connects to any discoverable broker that uses the multicast transport.

2

Makes the broker discoverable by other brokers over the multicast protocol.

3

Makes the broker update the list of available brokers for clients that connect using the failover protocol.

Note

Clients will only be updated when new brokers join the cluster, not when a broker leaves the cluster.

4

Creates a filter so that only those brokers whose names start with the letter A or the letter B are considered to belong to the failover cluster.

Example 2.3 shows the URI for a client that uses the failover protocol to connect to the broker and its cluster.


Dynamic failover provides a lot of control over how a client generates its list of available brokers, but it has weaknesses. It requires that you know the address of the initial broker and that the initial broker is active when the client starts up. It also requires that all of the brokers being used for failover are configured in a network of brokers.

Fuse Message Broker's discovery protocol offers an alternative method for dynamically generating a list of brokers that are available for client failover. The protocol feature allows brokers to advertise their availability and for clients to dynamically discover them. This is accomplished using two pieces:

A discovery agent is a mechanism that advertises available brokers to clients and other brokers. When a client, or broker, using a discovery URI starts up it will look for any brokers that are available using the specified discovery agent. The clients will update their lists periodically using the same mechanism.

How a discover agent learns about the available brokers varies between agents. Some agents use a static list, some use a third party registry, and some rely on the brokers to provide the information. For discovery agents that rely on the brokers for information, it is necessary to enable the discovery agent in the message broker configuration. For example, to enable the multicast discovery agent on an Openwire endpoint, you edit the relevant transportConnector element as shown in Example 2.4.


Where the discoveryUri attribute on the transportConnector element is initialized to multicast://default.

Tip

If a broker uses multiple transport connectors, you need to configure each transport connector to use a discovery agent individually. This means that different connectors can use different discovery mechanisms or that one or more of the connectors can be indiscoverable.

Fuse Message Broker currently supports the following discovery agents:

The zeroconf discovery agent is derived from Apple’s Bonjour Networking technology, which defines the zeroconf protocol as a mechanism for discovering services on a network. Fuse Message Broker bases its implementation of the zeroconf discovery agent on JmDSN, which is a service discovery protocol that is layered over IP/multicast and is compatible with Apple Bonjour.

The agent requires that each broker you want to advertise is configured to use a multicast discovery agent to publish its details to a multicast group. Clients using the zeroconf agent as part of the discovery URI they use for connecting to a broker will use the agent to receive the list of available brokers advertising in the specified multicast group.

Important

Your local network (LAN) must be configured to use IP/multicast for the zeroconf agent to work.

Example 2.13 shows the syntax for a discovery URI.


DiscoveryAgentUri is URI for the discovery agent used to build up the list of available brokers. Discovery agents are described in Discovery Agents.

The options, ?Options, are specified in the form of a query list. The discovery options are described in Table 2.3. You can also inject transport options as described in Setting options on the discovered transports.

Tip

If no options are required, you can drop the parentheses from the URI. The resulting URI would take the form discovery://DiscoveryAgentUri

A master/slave group consists of two or more brokers where one master broker is active and one or more slave brokers are on hot standby, ready to take over whenever the master fails or shuts down. All of the brokers store the message and event data processed by the master broker. So, when one of the slaves takes over as the new master broker the integrity of the messaging system is guaranteed.

Fuse Message Broker supports three master/slave broker configurations:

  • Shared nothing—the master forwards a copy of every persistent message it receives to a single slave broker

  • Shared file system—the master and the slaves use a common persistence store that is located on a shared file system

  • Shared JDBC database—the masters and the slaves use a common JDBC persistence store

When using shared nothing master/slave there are two approaches to configuring the slave:

Assuming that you choose the mode of operation where the slave takes over from the master, your clients will need to include logic for failing over to the new master. Adding the fail over logic requires that the clients use the masterslave protocol. This protocol is an instance of the failover protocol described in Failover Protocol that is specifically tailored for shared noting master/slave pairs.

If you had a two broker cluster where the master is configured to accept client connections on tcp://masterhost:61616 and the slave is configured accept client connections on tcp://slavehost:61616, you would use the masterslave URI shown in Example 3.4 for your clients.


Figure 3.3 shows the initial state of a shared file system master/slave group. When all of the brokers are started, one of them grabs the exclusive lock on the broker data store and becomes the master. All of the other brokers remain slaves and pause while waiting for the exclusive lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 3.4 shows the state of the master/slave group after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the data store frees up and another broker grabs the lock and gets promoted to master.


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.

Figure 3.6 shows the initial state of a JDBC master/slave group. When all of the brokers are started, one of them grabs the mutex lock on the database table and becomes the master. All of the other brokers become slaves and pause while waiting for the lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 3.7 shows the state of the group after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the database table frees up and another broker grabs the lock and gets promoted to master.


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.

In a JDBC master/slave configuration, there is nothing special to distinguish a master broker from the slave brokers. The membership of a particular master/slave group is defined by the fact that all of the brokers in the cluster use the same JDBC persistence layer and store their data in the same database tables.

Example 3.7 shows the configuration used be a master/slave group that stores the shared broker data in an Oracle database.


Important

The persistence adapter is configured as a direct JDBC persistence layer, using the jdbcPersistenceAdapter element. You must not use the journaled persistence adapter, which is configured using the journalPersistenceAdapter element, in this scenario.

B

broker
masterConnectorURI, Configuring the slave
shutdownOnMasterFailure, Configuring the slave
shutdownOnSlaveFailure, Configuring the master
waitForSlave, Configuring the master
broker networks
master/slave, Configuring the connection to a master/slave group
broker properties
rebalanceClusterClients, Broker-side configuration
updateClusterClients, Broker-side configuration
updateClusterClientsOnRemove, Broker-side configuration
updateClusterFilter, Broker-side configuration
updateURIsURL, Broker-side configuration

D

discovery agent
multicast, Multicast Discovery Agent
static, Static Discovery Agent
zeroconf, Zeroconf Discovery Agent
discovery protocol
backOffMultiplier, Transport options
initialReconnectDelay, Transport options
maxReconnectAttempts, Transport options
maxReconnectDelay, Transport options
URI, URI syntax
useExponentialBackOff, Transport options
discovery URI, URI syntax
discovery://, URI syntax
discoveryUri, Configuring a broker, Configuring a broker
dynamic failover, Dynamic Failover
broker configuration, Broker-side configuration
client configuration, Client-side configuration

F

failover, Failover Protocol
backOffMultiplier, Transport options
backup, Transport options
broker properties, Broker-side configuration
dynamic, Dynamic Failover
initialReconnectDelay, Transport options
maxCacheSize, Transport options
maxReconnectAttempts, Transport options
maxReconnectDelay, Transport options
randomize, Transport options
startupMaxReconnectAttempts, Transport options
static, Static Failover
timeout, Transport options
trackMessages, Transport options
updateURIsSupported, Transport options
useExponentialBackOff, Transport options
failover URI, Failover URI
transport options, Transport options
failover://, Failover URI

J

jdbcPersistenceAdapter, Configuring the brokers

M

master broker
reintroduction
shared file system, Reintroducing a failed node
shared JDBC, Reintroducing a failed node
shared nothing, Reintroducing the master
shared nothing master/slave, Configuring the master
master connector, Initial state
master/slave
broker networks, Configuring the connection to a master/slave group
network of brokers, Configuring the connection to a master/slave group
masterConnector, Configuring the slave
password, Configuring the slave
remoteURI, Configuring the slave
userName, Configuring the slave
masterslave, Configuring the connection to a master/slave group
masterslave URI, Configuring the clients
masterslave://, Configuring the clients
multicast discovery agent
broker configuration, Configuring a broker
URI, URI
multicast://, URI

S

shared file system master/slave
advantages, Overview
broker configuration, Configuring the brokers, Configuring the brokers
client configuration, Configuring the clients
disadvantages, Overview
incompatible SANs, File locking requirements
initial state, Initial state
master failure, State after failure of the master
NFSv3, File locking requirements
NFSv4, File locking requirements
OCFS2, File locking requirements
recovery strategies, State after failure of the master
reintroducing a node, Reintroducing a failed node
shared JDBC master/slave
advantages, Overview
client configuration, Configuring the clients
disadvantages, Overview
initial state, Initial state
master failure, After failure of the master
recovery strategies, After failure of the master
reintroducing a node, Reintroducing a failed node
shared nothing master/slave
client configuration, Configuring the clients
initial state, Initial state
master configuration, Configuring the master
master failure, State after failure of the master
recovery strategies, State after failure of the master
reintroducing the master, Reintroducing the master
slave configuration, Configuring the slave
shutdownOnSlaveFailure, Configuring the master
slave broker
shared nothing master/slave, Configuring the slave
static discovery agent
URI, Using the agent
static failover, Static Failover
static://, Using the agent

T

transportConnector
discoveryUri, Configuring a broker, Configuring a broker

W

waitForSlave, Configuring the master

Z

zeroconf discovery agent
broker configuration, Configuring a broker
URI, URI
zeroconf://, URI