Fuse ESB

Clustering Guide

Version 4.4.1

Sept. 2011
Trademark Disclaimer
Third Party Acknowledgements

Updated: 06 Jun 2013

Table of Contents

1. Failover and Discovery Protocols
Failover Protocol
Dynamic Discovery Protocol
Discovery Agents
2. Broker Networks
Network Connectors
Basic Network Concepts
Dynamic and Static Propagation
Filtering Messages and Destinations
Optimizing Routes
Balancing Consumer Load
Managing Producer Load
3. Load Balancing
Load Balancing over Consumers
4. Fault Tolerance
Master/Slave Patterns
Introduction to Master/Slave Clusters
Pure Master/Slave
Shared File System Master/Slave
JDBC Master/Slave
Fault Tolerant Broker Network
Index

List of Figures

2.1. Single Connector
2.2. Connectors in Each Direction
2.3. Duplex Connector
2.4. Multiple Connectors
2.5. Conduit Subscriptions
2.6. The Utility Graph
2.7. A Digraph
2.8. Concentrator Topology
2.9. Hub and Spoke Topology
2.10. Tree Topology
2.11. Mesh Topology
2.12. The Complete Graph, K5
2.13. Dynamic Propagation of Queue Messages
2.14. Static Propagation of Queue Messages
2.15. Duplex Mode and Static Propagation
2.16. Self-Avoiding Paths
2.17. Interaction between JMS Selector and Conduit Subscription
2.18. Shortest Route in a Mesh Network
2.19. Duplicate Subscriptions in a Network
2.20. Message Flow when Conduit Subscriptions Enabled
2.21. Message Flow when Conduit Subscriptions Disabled
2.22. Load Balancing with the Concentrator Topology
4.1. Pure Master/Slave Initial State
4.2. Pure Master/Slave after Master Failure
4.3. Shared File System Initial State
4.4. Shared File System after Master Failure
4.5. Shared File System after Master Restart
4.6. JDBC Master/Slave Initial State
4.7. JDBC Master/Slave after Master Failure
4.8. JDBC Master/Slave after Master Restart
4.9. Master/Slave Pairs on Two Host Machines
4.10. Broker Network Consisting of Host Pairs

List of Tables

1.1. Transport Connector Failover Properties
1.2. Discovery Transport Options
1.3. Multicast Discovery Agent Options
4.1. Master/Slave Patterns

List of Examples

1.1. Broker for Dynamic Failover
1.2. Failover URI for Connecting to a Failover Cluster
1.3. Making a TCP connector discoverable
1.4. Simple discovery agent URL
2.1. Single connector configuration
2.2. Two way connector
2.3. Duplex connector configuration
2.4. Connector with Filtered Destinations
2.5. Separate Configuration of Topics and Queues
4.1. Failover URL for Connecting to a Master/Slave Cluster
4.2. Master Configuration for Pure Master/Slave
4.3. Slave Configuration for Pure Master/Slave
4.4. Alternative Slave Configuration
4.5. Shared file cluster configuration
4.6. Alternate shared file cluster configuration
4.7. JDBC master/slave broker configuration
4.8. Network Connector to a Master/Slave Cluster

Configuring a broker to participate in dynamic failover requires two things:

Table 1.1 shows the broker properties that configure a failover cluster. These properties are exposed as attributes on the transportConnector element.


Note

The update and rebalance features should only be enabled, only if the clients that connect to the broker cluster use Fuse Message Broker 5.4.0 or later. These features are incompatible with clients older than version 5.4.0.

Example 1.1 shows the configuration for a broker that participates in dynamic failover.

Example 1.1. Broker for Dynamic Failover

<beans ... >
  <broker>
    ...
    <networkConnectors>
1      <networkConnector uri="multicast://default" />
    </networkConnectors>
    ...
    <transportConnectors>
      <transportConnector name="openwire"
          uri="tcp://0.0.0.0:61616"
2         discoveryUri="multicast://default"
3         updateClusterClients="true"
4         updateClusterFilter="*A*,*B*" />  
    </transportConnectors>
    ...
  </broker>
</beans>

The configuration in Example 1.1 does the following:

1

Creates a network connector that connects to any discoverable broker that uses the multicast transport.

2

Makes the broker discoverable by other brokers over the multicast protocol.

3

Makes the broker update the list of available brokers for clients that connect using the failover protocol.

Note

Clients will only be updated when new brokers join the cluster, not when a broker leaves the cluster.

4

Creates a filter so that only those brokers whose names start with the letter A or the letter B are considered to belong to the failover cluster.

Example 1.2 shows the URI for a client that uses the failover protocol to connect to the broker and its cluster.


A discovery URI must conform to the following syntax:

discovery://(DiscoveryAgentUri)?TransportOptions

Where the discovery agent URI, DiscoveryAgentUri, identifies a discovery agent, as described in Discovery agents above. The transport options, ?TransportOptions, are specified in the form of a query list (where the supported options are described in Table 1.2 ). If no transport options are required, you can use the following alternative syntax:

discovery://DiscoveryAgentUri

It is also possible to apply options to the discovered transport by adding those options to the discovery URI. When parsing the URI, the discovery protocol consumes those options that it recognizes (the ones listed in Table 1.2). Any remaining options are held in reserve until a transport is discovered. When the discovered transport is resolved, the remaining options are applied to that transport.

For example, suppose that you expect to discover a TCP endpoint, you can then add some TCP transport options to your discovery URI. The following discovery URI shows how to set the connectionTimeout TCP transport option:

discovery://(multicast://default)?connectionTimeout=2000

The multicast discovery agent uses the IP multicast protocol to find any message brokers currently active on the local network. In order for the protocol to work, a multicast discovery agent must be enabled on each broker you want to advertise and messaging clients must be configured to use a discovery URI.

The URI for a multicast discovery agent must conform to the following syntax:

multicast://Address:Port?TransportOptions

The following transport options are supported:


All participants in the same discovery network must use the same group ID. To specify the IP adress, port number, and group ID explicitly, you can specify a URI like the following:

multicast://224.1.2.3:6255?group=mygroupname

For easy configuration, you could use the special default multicast URI, as follows:

multicast://default

This default URI is equivalent to the URI, multicast://239.255.2.3:6155?group=default.

Note

Your local network (LAN) must be configured appropriately for the IP/multicast protocol to work. If your clients fail to discover a broker using the multicast protocol, this could be because IP/multicast is not set up on your network.

The rendezvous discovery agent is derived from Apple’s Bonjour Networking technology, which defines the rendezvous protocol as a mechanism for discovering services on a network. To enable the protocol, a multicast discovery agent must be configured on each broker you want to advertise and messaging clients must be configured to use a discovery URI.

The URI for a rendezvous discovery agent must conform to the following syntax:

rendezvous://GroupID

Where the GroupID is an alphanumeric identifier. All participants in the same discovery network must use the same GroupID.

For example, to use a rendezvous discovery agent on the client side, where the client needs to connect to the groupA group, you would construct a discovery URL like the following:

discovery://(rendezvous://groupA)

Note

Your local network (LAN) must be configured appropriately for the IP/multicast protocol to work. If your clients fail to discover a broker using the rendezvous protocol, this could be because IP/multicast is not set up on your network.

Figure 2.1 shows a single network connector from broker A to broker B. The arrow on the connector indicates the direction of message propagation (from A to B). Subscriptions propagate in the opposite direction (from B to A). Because of the restriction on the direction of message flow in this network, it is advisable to connect producers only to broker A and consumers only to broker B. Otherwise, some messages might not be able to reach the intended consumers.


When the connector arrow points from A to B, this implies that the network connector is actually defined on broker A. For example, the following fragment from broker A's configuration file shows the network connector that connects to broker B:


The networkConnector element in the preceding example sets the following basic attributes:

By default, after passing through a network connector, subscriptions to the same queue or subscriptions to the same topic are automatically consolidated into a single subscription known as a conduit subscription. Figure 2.5 shows an overview of how the topic subscriptions from two consumers, C1 and C2, are consolidated into a single conduit subscription after propagating from broker B to broker A.


In this example, each consumer subscribes to the identical topic, t, which gives rise to the subscriptions, C1:t and C2:t in broker B. Both of these subscriptions propagate automatically from broker B to broker A. Because broker A has conduit subscriptions enabled, its network connector consolidates the duplicate subscriptions, C1:t and C2:t, into a single subscription, B:t. Now, if a message on topic t is sent to broker A, broker A sends a single copy of the message to broker B, to honor the conduit subscription, B:t. Broker B then sends a copy of the message to each consumer, to honor the topic subscriptions, C1:t and C2:t.

It is essential to enable conduit subscription in order to avoid duplication of topic messages. Consider what would happen in Figure 2.5 if conduit subscription was disabled. In this scenario, two subscriptions, B:C1:t and B:C2:t, would be registered in broker A. Now, if a message on topic t is sent to broker A, broker A would send two copies of the message to broker B, to honor the topic subscriptions, B:C1:t and B:C2:t. Broker B would then send two copies of the message to each consumer, to honor the topic subscriptions, C1:t and C2:t. In other words, each consumer would receive the topic message twice.

Conduit subscriptions can optionally be disabled by setting the conduitSubscriptions attribute to false on the networkConnector element. See Balancing Consumer Load for more details.

A graph is a mathematical entity consisting of a vertex set (analogous to the set of nodes in a network) and an edge set (analogous to the set of connections between network nodes). A graph can therefore be used to describe the underlying topology of a network. For example, consider the graph shown in Figure 2.6.


Formally, the utility graph[1] consists of the following:

  • The Vertex set, {A,B,C,X,Y,Z}.

  • The Edge set, {{A,X},{A,Y},{A,Z},{B,X},{B,Y},{B,Z},{C,X},{C,Y},{C,Z}}, where each edge is represented by the pair of vertices it joins. For example, the join from vertex A to vertex X is represented as the edge, {A,X}.

In graph theory, the distance, d(X,Y), between two vertices, X and Y, is the minimum number of edges that must be traversed in order to get from vertex X to vertex Y.

For example, if you consider the utility graph shown in Figure 2.6, you can see that d(A,X)=1 and d(X,Y)=2. The shortest distance between X and Y is realised by any of the paths XAY, XBY, or XCY.

The concept of distance is useful in network theory, because it corresponds to the length of the shortest (optimal) route between any two nodes in a network.



[1] The name of this graph derives from a puzzle, where you are asked to find a way to connect each of the three houses, A, B, and C, to the three utilities, X, Y, and Z, without having any of the wires or pipes cross over (in graph theory, a graph that can be drawn without edge crossings is called planar).

Static propagation refers to message propagation that occurs in the absence of subscription information. Sometimes, because of the way a broker network is set up, it can make sense to move messages between brokers, even when there is no relevant subscription information.

Static propagation is configured by specifying the queue (or queues) that you want to statically propagate. Into the relevant networkConnector element, insert staticallyIncludedDestinations as a child element and then list the queues and topics you want to propagate using the queue and topic child elements. For example, to specify that messages in the queue, TEST.FOO, are statically propagated from A to B, you would define the network connector in broker A's configuration as follows:

Consider the network shown in Figure 2.14. This network is set up so that consumers only attach to broker D or to broker E Messages sent to the queue, TEST.FOO, are configured to propagate statically on all on all of the network connectors, (A,B), (B,C), (C,D), and (C,E).


The static message propagation in this example proceeds as follows:

  1. Initially, there are no consumers attached to the network. A producer, P, connects to broker A and sends 10 messages to the queue, TEST.FOO.

  2. Because the network connector, (A,B), has enabled static propagation for the queue, TEST.FOO, the 10 messages on broker A are forwarded to broker B.

  3. Likewise, because the network connector, (B,C), has enabled static propagation for the queue, TEST.FOO, the 10 messages on broker B are forwarded to broker C.

  4. Finally, because the network connectors, (C,D) and (C,E), have enabled static propagation for the queue, TEST.FOO, the 10 messages on broker C are alternately sent to broker D and broker E. In other words, the brokers, D and E, receive every second message. Hence, at the end of the static propagation, there are 5 messages on broker D and 5 messages on broker E.

Note

Using the preceding static configuration, it is possible for messages to get stuck in a particular broker. For example, if a consumer now connects to broker E, it will receive the 5 messages stored on broker E, but it will not receive the 5 messages stored on broker D. The messages remain stuck on broker D until a consumer connects directly to it.

The following characters can be used to define wildcard matches for topic names and queue names:

.Separates segments in a path name.
*Matches any single segment in a path name.
>Matches any number of segments in a path name.

Wildcards are meant to be used with destination names that have a segmented structure, like a path name—for example, PRICE.STOCK.NASDAQ.IBM or PRICE.STOCK.NYSE.SUNW—where the segments are delimited by the . character. The following table shows some examples of destination wildcards and describes what names they would match.

Destination wildcardWhat it matches
PRICE.>Any price for any product on any exchange.
PRICE.STOCK.>Any price for a stock on any exchange.
PRICE.STOCK.NASDAQ.*Any stock price on NASDAQ.
PRICE.STOCK.*.IBMAny IBM stock price on any exchange.

It is possible to filter the messages that pass through a network connector by specifying destinations to include and destinations to exclude. The following child elements of networkConnector are used to filter dynamic destinations:

Example 2.4 shows a network connector that is configured to filter dynamic destinations. In this example, the connector transmits stock prices from any exchange except the NYSE and transmits orders to trade stocks for any exchange except the NYSE.


Fuse Message Broker supports standard JMS selectors, which enable consumers to filter messages from a particular destination by testing the contents of a message's JMS headers. When a consumer subscribes to a particular destination, the standard JMS API can be used to specify a selector (see javax.jms.Message for more details).

If your consumers use JMS selectors, you should be aware that some interaction can occur between JMS selectors and conduit subscriptions. In general, JMS selectors are always applied by the consumer's local broker (that is, the broker to which the consumer is directly connected). But if the conduit subscriptions option is enabled (which it is by default), JMS selector information is omitted from any conduit subscriptions transmitted to a remote broker. This implies that the remote broker (or brokers ) do not perform any filtering based on JMS selectors. There are certain scenarios where this behavior can cause problems.

For example, consider the broker network shown in Figure 2.17, where conduit subscription is enabled throughout the network. Both of the consumers, C1 and C2, subscribe to the same queue and they also define JMS selectors as follows: C1 selects messages for which the region header is equal to us; and C1 selects messages for which the region header is equal to emea.


Now consider what happens when a producer, P, connects to broker A and starts sending messages to the queue. The consumer subscriptions, s1 and s2, automatically propagate to broker A. But because these subscriptions are both on the same queue and the conduit subscriptions option is enabled, broker A combines the subscriptions into a single conduit subscription, cs, which does not include any selector details. When P starts sending messages to the queue, broker A forwards the messages alternately to broker B and broker C, without checking whether the messages satisfy the relevant selectors. If you are particularly unlucky, you might find that all of the messages for region emea end up on broker B and all of the messages for region us end up on broker C. In this case, none of the messages could be consumed.

If you are using a network topology such as a hub-and-spoke (see Figure 2.9) or a tree (see Figure 2.10), the network route is inherently deterministic and you do not need to concern yourself with choosing an optimum route. For other topologies, however, it is possible to have multiple alternative routes joining a producer to a consumer across the network. In such cases, it is usually preferable to simplify the routing behavior, so that an optimum route is preferred by the network.

Certains kinds of network—for example, Hub and spokes topology and Tree topology—have determinate routes. That is, there exists a unique route between any two brokers in the network.

If your network is a mesh, on the other hand, you might find that there are multiple routes joining some pairs of brokers. For such indeterminate networks, it is normally preferable for messages to propagate along the shortest route, in order to maximize the efficiency of the network. To ensure that the shortest route is preferred, enable the decreaseNetworkConsumerPriority option on all of the connectors in the network (the default is false). For example, you can enable this option on a network connector as follows:

<networkConnectors>
    <networkConnector name="linkToBrokerB"
       uri="static:(tcp://localhost:61002)"
       networkTTL="3"
       decreaseNetworkConsumerPriority="true" />
</networkConnectors>

When decreaseNetworkConsumerPriority is set to true, the priority is set as follows:

  • Local consumers (attached directly to the broker) have a priority of 0.

  • Network subscriptions have an initial priority of -5.

  • The priority of a network subscription is reduced by 1 for every network hop that it traverses

A broker sends messages preferentially to the subscription with the highest priority, but if the prefetch buffer is full, the broker will divert messages to the subscription with the next highest priority. If multiple subscriptions have the same priority, the broker distributes messages equally between those subscriptions.

Figure 2.18 illustrates the effect of setting this option in a broker network.


In this network, there are three alternative routes connecting producer P to consumer C1—these are PBAFEC1 (three broker hops), PBEC1 (one broker hop), and PBCDEC1 (three broker hops). When decreaseNetworkConsumerPriority is enabled, the route, PBEC1, has highest priority, so messages from P to C1 are sent preferentially along this route.

In some cases, setting decreaseNetworkConsumerPriority to true is not enough to ensure deterministic routing in a network. Consider the network of brokers, A, B, and C, shown in Figure 2.19. In this scenario, a producer, P, (which writes messages to queue, foo) connects to broker A and a consumer, C1, (which reads messages from queue, foo) connects to broker B. The network TTL is equal to 2, so two alternative routes are possible: the short route, PABC1, and the long route, PACBC1.


Now, if you set decreaseNetworkConsumerPriority to true, the short route is preferred. So, messages are propagated along the route PABC1. However, under heavy load conditions, the short route, PABC1, can become overloaded and in this case the broker, A, will fall back to the long route, PACBC1. The problem with this scenario is that when the consumer, C1, shuts down, it can lead to messages getting stuck on broker C. In order to avoid this problem, it is recommended that you set the suppressDuplicateQueueSubscriptions option to true on all of the network connectors in you network. For example, you can set this option as follows:

<networkConnectors>
    <networkConnector name="linkToBrokerB"
       uri="static:(tcp://localhost:61002)"
       networkTTL="3"
       decreaseNetworkConsumerPriority="true"
       suppressDuplicateQueueSubscriptions="true"/>
</networkConnectors>

The effect of enabling this option is that the broker allows only a single queue subscription to be created for a given consumer. This means that only a single route can be created between a producer and a consumer, so that the routing becomes fully deterministic. In general, it is recommended that you enable both the decreaseNetworkConsumerPriority option and the suppressDuplicateQueueSubscriptions option together.

Note

In the example shown in Figure 2.19, you could have suppressed the long route by reducing the network TTL to 1. Normally, however, in a large network you do not have the option of reducing the network TTL arbitrarily. The network TTL has to be large enough for messages to reach the most distant brokers in the network.

When connecting to a broker network, producers and consumers must be distributed across the available brokers. For example, in the case of a producer connecting to the concentrator topology shown in Figure 2.22, the producer should connect using a failover URL that shares the load across the brokers in the first layer, A, B, and C. Assuming that the brokers are running on separate hosts—brokerA, brokerB, and brokerC—and assuming that they all listen on IP port 61616, the producers should use the following failover URL to connect to the broker network:

failover://(tcp://brokerA:61616,tcp://brokerB:61616,tcp://brokerC:61616)

By default, a producer will randomly select one of the URLs in the failover list and attempt to connect to it. If the first connection attempt fails, the producer will try the other URLs in turn. For more details of the failover protocol, see Failover Protocol.

There are two alternative approaches to configuring the slave. Example 4.3 shows how to configure the slave using the masterConnector element, which enables you to specify username and password credentials for the connection.


The masterConnector element supports the following attributes:

remoteURI

Specifies the address of the master's transport connector port.

userName

Username for connecting to the master (if authentication is enabled).

password

Password for connecting to the master (if authentication is enabled).

Example 4.4 shows the alternative approach to configuration, by setting attributes on the broker element. This approach does not support setting credentials, however.


The masterConnector element supports the following attributes:

masterConnectorURI

Specifies the address of the master's transport connector port.

shutdownOnMasterFailure

If true, the slave shuts down when the master fails; otherwise, the slave takes over as the new master. Default is false.

The slave ensures that there is a separate copy of each message and acknowledgment on another machine, which can protect against catastrophic hardware failure. If the master fails, you might want the slave to shut down as well, if protection against data loss is your main priority. You can then manually duplicate the slave's copy of the data before restoring the system.

Figure 4.3 shows the initial state of a shared file system master/slave cluster. When all of the brokers in the cluster are started, one of them grabs the exclusive lock on the broker data file, thus becoming the master. All of the other brokers in the clusters remain slaves and pause while waiting for the exclusive lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 4.4 shows the state of the cluster after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the broker data file frees up and another broker in the cluster grabs the lock and gets promoted to master (broker2 in the figure).


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.

The only condition that brokers in a cluster must satisfy is that they all use the same non-journaling JDBC persistence layer with the broker data stored in the same underlying database tables.

For example, to store the shared broker data in an Oracle database, you could configure the non-journaled JDBC persistence layer, for all brokers in the cluster, as follows:


The persistence adapter is configured as a direct JDBC persistence layer, using the jdbcPersistenceAdapter element. You must not use the journaled persistence adapter (configurable using the journalPersistenceAdapter element) in this scenario.

Figure 4.6 shows the initial state of a JDBC master/slave cluster. When all of the brokers in the cluster are started, one of them grabs the mutex lock on the database table, thus becoming the master. All of the other brokers in the clusters remain slaves and pause while waiting for the lock to be freed up. Only the master starts its transport connectors, so all of the clients connect to it.


Figure 4.7 shows the state of the cluster after the original master has shut down or failed. As soon as the master gives up the lock (or after a suitable timeout, if the master crashes), the lock on the database table frees up and another broker in the cluster grabs the lock and gets promoted to master (broker2 in the figure).


After the clients lose their connection to the original master, they automatically try all of the other brokers listed in the failover URL. This enables them to find and connect to the new master.