Chapter 20. Clustering

With AMQ Broker, clustering allows brokers to be grouped together and share the message processing load. Each active broker in the cluster manages its own messages and handles its own connections. A cluster provides a number of benefits:

  • Brokers can be connected together in many different topologies.
  • Client connections can be balanced across the cluster.
  • Messages can be redistributed to avoid broker starvation.
  • Clients and brokers can connect to the cluster with minimal information about it.

High Availability (HA) is another benefit of clustering that groups brokers into master-slave pairs. One feature of HA is that it enables client requests to fail over to the slave broker in case the master loses connectivity to the network. See the chapter on HA and failover for more information.

You configure a cluster inside the broker’s BROKER_INSTANCE_DIR/etc/broker.xml configuration file. There are three major configuration elements related to clustering:

discovery-group
Determines how the broker discovers other members in the cluster. Discovery can be dynamic and use either UDP or JGroups to find other brokers on the network that are members of the same cluster. (JGroups is based on IP multicast, although TCP can also be used as transport.) Alternatively, discovery can be static and use a configured list of connections to specific brokers.
broadcast-group
Determines how the broker transmits cluster-related information, such as enrollment, to other brokers in the cluster. A broadcast-group can use either UDP or JGroups, but the choice must match its discovery-group counterpart. For example, if the broadcast-group is configured to use UDP, the discovery-group must also use UDP, including the same multicast address.
cluster-connection
Details about the broker’s connection to other brokers in the cluster. Uses a discovery-group to make the initial connection to each broker in the cluster.
Important

After a cluster broker has been configured, it is common to copy the configuration to other brokers to produce a symmetric cluster. However, when copying the broker files, do not copy any of the following directories from one broker to another:

  • bindings
  • journal
  • large-messages

When a broker is started for the first time and initializes its journal files, it also persists a special identifier to the journal directory. This id must be unique among brokers in the cluster, or the cluster will not form properly.

Important

The sequence of the elements in the cluster connection configuration has to be in a specific order. You must adhere to the order as shown in the schema file INSTALL_DIR/schema/artemis-configuration.xsd.

20.1. About Broadcast Groups

A broker uses a broadcast group to push information about its cluster-related connection to other potential cluster members on the network. The connection information is captured in the configuration as a connector. See Networking Connections: Acceptors and Connectors for details on connectors. There can be many broadcast groups per broker. If the broker has a backup, the backup’s connection information will be broadcast too. A broadcast-group can use either UDP or JGroups, but the choice must match its discovery-group counterpart.

Prerequisites

  • Configuring a broadcast-group requires a connector, which defines connection information for each broker. The configuration for the connector will be sent to other brokers in the cluster during discovery. The default configuration includes several connectors already, or you can create a new one. See About Connectors for more information.

Procedure

  • Add broadcast-group configuration to BROKER_INSTANCE_DIR/etc/broker.xml. Below is an example broadcast-group that uses UDP, followed by a description of each configuration element. You typically use the default broadcast group-address and group-port values, but you can specify any of the following elements to suit your environment.
<configuration>
  <core>
    ...
    <broadcast-groups>
       <broadcast-group name="my-broadcast-group"> 1
          <local-bind-address>172.16.9.3</local-bind-address> 2
          <local-bind-port>5432</local-bind-port> 3
          <group-address>231.7.7.7</group-address> 4
          <group-port>9876</group-port> 5
          <broadcast-period>2000</broadcast-period> 6
          <connector-ref>netty-connector</connector-ref> 7
       </broadcast-group>
    </broadcast-groups>
  </core>
</configuration>
1
You must use the name attribute to give the broadcast group a unique name.
2
The local-bind-address is the address to which the UDP socket is bound. If you have multiple network interfaces on your broker, you should specify which one you wish to use for broadcasts. If this property is not specified, the socket will be bound to an IP address chosen by the OS. This is a UDP specific attribute.
3
If you want to specify a local port to which the datagram socket is bound, you can specify it here. In most cases, you would just use the default value of -1, which signifies that an anonymous port should be used. This parameter is always specified in conjunction with local-bind-address. This is a UDP specific attribute.
4
The group-address is the multicast address to which the data will be broadcast. It is a class D IP address in the range 224.0.0.0 to 239.255.255.255, inclusive. The address 224.0.0.0 is reserved and is not available for use. This parameter is mandatory. This is a UDP specific attribute.
5
The group-port is the UDP port number used for broadcasting. This parameter is mandatory. This is a UDP specific attribute.
6
The broadcast-period is the interval in milliseconds between consecutive broadcasts. This parameter is optional; the default value is 2000 milliseconds.
7
The connector-ref declares a reference to a previously configured connector that is transmitted by the broadcast (see Network Connections: Acceptors and Connectors for more information).

Related Information

See the clustered-queue example under INSTALL_DIR/examples/features/clustered for a working example of a broadcast-group that uses UDP.

20.1.1. Configuring a Broadcast Group to Use JGroups

Sometimes using UDP is not an option due to constraints put on the network. In these situations, you can configure the broadcast group to communicate using JGroups as an alternative to UDP.

Prerequisites

JGroups communication requires a separate configuration file. See clustered-jgroups under INSTALL_DIR/examples/features/clustered for an example JGroups configuration file.

Procedure

To create a broadcast group that uses JGroups to communicate, add configuration to BROKER_INSTANCE_DIR/etc/broker.xml configuration file. Below is an example broadcast-group that uses JGroups, followed by a description of each configuration element.

<configuration>
  <core>
    ...
    <broadcast-groups>
       <broadcast-group name="my-broadcast-group"> 1
          <jgroups-file>test-jgroups-file_ping.xml</jgroups-file> 2
          <jgroups-channel>activemq_broadcast_channel</jgroups-channel> 3
          <broadcast-period>2000</broadcast-period> 4
        <connector-ref>netty-connector</connector-ref> 5
       </broadcast-group>
    </broadcast-groups>
  </core>
</configuration>

To use JGroups to broadcast, you must specify the following:

1
You must use the name attribute to give the broadcast group a unique name.
2
The name of JGroups configuration file to initialize JGroups channels. The file must be in the Java resource path so that the broker can load it.
3
The name of the JGroups channel to connect to for broadcasting.
4
The period in milliseconds between consecutive broadcasts. This parameter is optional; the default value is 2000 milliseconds.
5
The connector-ref declares a reference to a previously configured connector that is transmitted by the broadcast (see Network Connections: Acceptors and Connectors for more information.

Related Information

See clustered-jgroups under INSTALL_DIR/examples/features/clustered for a working example of a broadcast-group that uses JGroups.

20.2. About Discovery Groups

While the broadcast group defines how cluster-related information is transmitted, a discovery group defines how connector information is received. Discovery groups maintain a list of connectors—​one entry for each broker. As it receives broadcasts from a broker, it updates its entry. If it has not received a broadcast from a broker for a length of time, it will remove the entry.

Discovery groups are typically used in two places:

  • By cluster connections so they know how to obtain an initial connection to download the topology.
  • By messaging clients so they know how to obtain an initial connection to download the topology.

Although a discovery group always accepts broadcasts, its current list of available live and backup brokers is used only when an initial connection is made. After that point, broker discovery is performed over the normal broker connections.

20.2.1. Configuring a Discovery Group to Use UDP

Define discovery groups in the BROKER_INSTANCE_DIR/etc/broker.xml configuration file. You can define multiple discovery groups.

Procedure

To create a discovery group that uses UDP to communicate, add configuration to BROKER_INSTANCE_DIR/etc/broker.xml configuration file. Below is an example, followed by a description of each configuration element.

<configuration>
  <core>
    ...
    <discovery-groups>
       <discovery-group name="my-discovery-group"> 1
          <local-bind-address>172.16.9.7</local-bind-address> 2
          <group-address>231.7.7.7</group-address> 3
          <group-port>9876</group-port> 4
          <refresh-timeout>10000</refresh-timeout> 5
       </discovery-group>
    </discovery-groups>
    ...
  </core>
</configuration>
1
You must use the name attribute to give the discovery group a unique name.
2
Use local-bind-address to specify that this discovery group only listens to a specific interface. This is a UDP specific attribute.
3
The group-address is the multicast IP address of the group on which to listen. It should match the group-address in the broadcast group from which you want to listen. This parameter is mandatory. This is a UDP specific attribute.
4
The group-port is UDP port of the multicast group. It should match the group-port in the broadcast group from which you wish to listen. This parameter is mandatory. This is a UDP specific attribute.
5
The refresh-timeout is the period the discovery group waits after receiving the last broadcast from a particular broker before removing that broker’s connector pair entry from its list. You would normally set this to a value significantly higher than the broadcast-period on the broadcast group, or - due to a slight difference in timing - brokers might intermittently disappear from the list even though they are still broadcasting. This parameter is optional. The default value is 10000 milliseconds (10 seconds).

Related Information

See the clustered-queue example under INSTALL_DIR/examples/features/clustered for a working example of a discovery-group that uses UDP.

20.2.2. Configuring a Discovery Group to Use JGroups

Sometimes using UDP is not an option due to constraints put on the network. In these situations, you can configure the broadcast group to communicate using JGroups as an alternative to UDP. Define discovery groups in the BROKER_INSTANCE_DIR/etc/broker.xml configuration file. You can define multiple discovery groups.

Prerequisites

JGroups communication requires a separate configuration file. See clustered-jgroups under INSTALL_DIR/examples/features/clustered for an example JGroups configuration file.

Procedure

To create a discovery group that uses JGroups, add configuration to BROKER_INSTANCE_DIR/etc/broker.xml. Below is an example, followed by a description of each configuration element. To receive broadcasts from JGroups channels, you must specify each of the elements listed.

<configuration>
  <core>
    ...
    <discovery-groups>
       <discovery-group name="my-discovery-group">1
          <jgroups-file>test-jgroups-file_ping.xml</jgroups-file>2
          <jgroups-channel>activemq_broadcast_channel</jgroups-channel>3
          <refresh-timeout>10000</refresh-timeout>4
       </discovery-group>
    </discovery-groups>
    ...
  </core>
</configuration>
1
You must use the name attribute to give the discovery group a unique name.
2
The name of JGroups configuration file to initialize JGroups channels. The file must be in the Java resource path so that the broker can load it.
3
The name of the JGroups channel to connect to for broadcasting.
4
The refresh-timeout is the period the discovery group waits after receiving the last broadcast from a particular broker before removing that broker’s connector pair entry from its list. You would normally set this to a value significantly higher than the broadcast-period on the broadcast group, or - due to a slight difference in timing - brokers might intermittently disappear from the list even though they are still broadcasting. This parameter is optional. The default value is 10000 milliseconds (10 seconds).

Related Information

See clustered-jgroups under INSTALL_DIR/examples/features/clustered for a working example of a discovery-group that uses JGroups.

20.3. About Cluster Connections

Cluster connections group brokers into clusters so that messages can be load balanced between the brokers in the cluster. You can configure multiple cluster connections for each broker.

Each cluster connection only applies to addresses that match its assigned address. The assigned address can be any value, and you can have many cluster connections with different addresses, simultaneously balancing messages for those addresses, potentially to different clusters of brokers. By having multiple cluster connections on different addresses, a single broker can effectively take part in multiple clusters simultaneously.

The address field also supports comma separated lists of addresses. Use exclude syntax, ! to prevent an address from being matched. Below are some example addresses:

jms.eu
Matches all addresses starting with jms.eu.
!jms.eu
Matches all addresses except for those starting with jms.eu
jms.eu.uk,jms.eu.de
Matches all addresses starting with either jms.eu.uk or jms.eu.de
jms.eu,!jms.eu.uk
Matches all addresses starting with jms.eu, but not those starting with jms.eu.uk

However, you should not to have multiple cluster connections with overlapping addresses, for example, "europe" and "europe.news", because the same messages could then be distributed between more than one cluster connection, possibly resulting in duplicate deliveries.

20.3.1. Configuring a Cluster Connection

Configure a cluster connections by adding a cluster-connection element to BROKER_INSTANCE_DIR/etc/broker.xml.

Prerequisites

  1. A connector for other brokers in the cluster to use when communicating to this broker. The configuration for the connector will be sent to other brokers in the cluster during discovery. The default configuration includes several connectors already, or you can create a new one. See About Connectors for more information.
  2. A discovery-group for this broker to use while making initial connections to the cluster. See About Discovery Groups in this chapter for more information.

Procedure

To create a cluster connection, add configuration to BROKER_INSTANCE_DIR/etc/broker.xml. Below is an example, followed by a description of each configuration element. The example below is the minimal configuration required for a cluster connection.

<configuration>
  <core>
    ...
    <cluster-connections> 1
       <cluster-connection name="my-cluster"> 2
         <connector-ref>netty-connector</connector-ref> 3
         <discovery-group-ref discovery-group-name="my-discovery-group"/> 4
         ...
       </cluster-connection>
       ...
    </cluster-connections>
  </core>
</configuration>
1
You place each cluster-connection element under the parent element, cluster-connections.
2
A cluster-connection must be given a unique name.
3
The connector-ref points to a previously configured connector that will have its information broadcast to the cluster. This enables other brokers in the cluster to connect to this broker.
4
The discovery-group-ref points to a previously configured discovery-group that this broker will use to locate other members of the cluster. Alternatively, you can use a static list of brokers.

Related Information

See the table in the appendix for a full list of configuration elements and a description

20.3.2. Specifying a Static List of Cluster Members

Rather than use dynamic discovery to form a cluster, you can configure a static list of connectors that restricts the cluster to a limited set of brokers. Use this static discovery method to form non-symmetrical clusters such as chain clusters or ring clusters.

Prerequisites

A configured cluster connection. See Configuring a Cluster Connection for details on creating a cluster-connection within the configuration.

Procedure

To specify members of a cluster explicitly, add configuration to BROKER_INSTANCE_DIR/etc/broker.xml as in the following steps.

  1. Create a static-connectors element within the relevant cluster-connection.

    <configuration>
      <core>
        ...
        <cluster-connections>
           <cluster-connection name="my-cluster">
             ...
             <static-connectors></static-connectors>
             ...
           </cluster-connection>
        </cluster-connections>
      </core>
    </configuration>
  2. Add a connector-ref element that names the connector to use when creating connections to other brokers in the cluster.

    In the example below, there is a set of two brokers of which one will always be available. If there are other brokers in the cluster, they will be discovered by one of these connectors when an initial connection is made.

    <configuration>
      <core>
        ...
        <cluster-connections>
           <cluster-connection name="my-cluster">
               ...
               <static-connectors>
                 <connector-ref>server1-connector</connector-ref>
                 <connector-ref>server2-connector</connector-ref>
               </static-connectors>
               ...
           </cluster-connection>
        </cluster-connections>
      </core>
    </configuration>

Related Information

See clustered-static-discovery under INSTALL_DIR/examples/features/clustered for a working example that uses static discovery.

20.3.3. Configuring a Client to Use Dynamic Discovery

You can configure a Red Hat AMQ Core JMS client to discover a list of brokers when attempting to establish a connection.

Configuring Dynamic Discovery Using JMS

If you are using JNDI on the client to look up your JMS connection factory instances, you can specify these parameters in the JNDI context environment. Typically the parameters are defined in a file named jndi.properties. The host and part in the URL for the connection factory should match the group-address and group-port from the corresponding broadcast-group inside broker’s broker.xml configuration file. Below is an example of a jndi.properties file configured to connect to a broker’s discovery group.

java.naming.factory.initial = ActiveMQInitialContextFactory
connectionFactory.myConnectionFactory=udp://231.7.7.7:9876

When this connection factory is downloaded from JNDI by a client application and JMS connections are created from it, those connections will be load-balanced across the list of servers that the discovery group maintains by listening on the multicast address specified in the broker’s discovery group configuration.

As an alternative to using JNDI, you can use specify the discovery group parameters directly in your Java code when creating the JMS connection factory. The code below provides an example of how to do this.

final String groupAddress = "231.7.7.7";
final int groupPort = 9876;

DiscoveryGroupConfiguration discoveryGroupConfiguration = new DiscoveryGroupConfiguration();
UDPBroadcastEndpointFactory udpBroadcastEndpointFactory = new UDPBroadcastEndpointFactory();
udpBroadcastEndpointFactory.setGroupAddress(groupAddress).setGroupPort(groupPort);
discoveryGroupConfiguration.setBroadcastEndpointFactory(udpBroadcastEndpointFactory);

ConnectionFactory jmsConnectionFactory = ActiveMQJMSClient.createConnectionFactoryWithHA(discoveryGroupConfiguration, JMSFactoryType.CF);

Connection jmsConnection1 = jmsConnectionFactory.createConnection();
Connection jmsConnection2 = jmsConnectionFactory.createConnection();

The refresh timeout can be set directly on the DiscoveryGroupConfiguration by using the setter method setRefreshTimeout(). The default value is 10000 milliseconds.

On first usage, the connection factory will make sure it waits this long since creation before creating the first connection. The default wait time is 10000 milliseconds, but you can change it by passing a new value to DiscoveryGroupConfiguration.setDiscoveryInitialWaitTimeout().

20.3.4. Configuring a Client to Use Static Discovery

Sometimes it may be impossible to use UDP on the network you are using. In this case you can configure a connection with an initial list if possible servers. The list can be just one broker that you know will always be available, or a list of brokers where at least one will be available.

This does not mean that you have to know where all your servers are going to be hosted, you can configure these servers to use the reliable servers to connect to. After they are connected, their connection details will be propagated via the server the client.

Both Red Hat AMQ Core JMS and Java EE JMS clients can use a static list to discover brokers.

Configuring Static Discovery

If you are using JNDI on the client to look up your JMS connection factory instances, you can specify these parameters in the JNDI context environment. Typically the parameters are defined in a file named jndi.properties. Below is an example jndi.properties file that provides a static list of brokers instead of using dynamic discovery.

java.naming.factory.initial=org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory
connectionFactory.myConnectionFactory=(tcp://myhost:61616,tcp://myhost2:61616)

When the above connection factory is used by a client, its connections will be load-balanced across the list of brokers defined within the parentheses ().

If you are instantiating the JMS connection factory directly, you can specify the connector list explicitly when creating the JMS connection factory, as in the example below.

HashMap<String, Object> map = new HashMap<String, Object>();
map.put("host", "myhost");
map.put("port", "61616");
TransportConfiguration broker1 = new TransportConfiguration(NettyConnectorFactory.class.getName(), map);

HashMap<String, Object> map2 = new HashMap<String, Object>();
map2.put("host", "myhost2");
map2.put("port", "61617");
TransportConfiguration broker2 = new TransportConfiguration(NettyConnectorFactory.class.getName(), map2);

ActiveMQConnectionFactory cf = ActiveMQJMSClient.createConnectionFactoryWithHA(JMSFactoryType.CF, broker1, broker2);

20.4. Enabling Message Redistribution

Cluster connections allow brokers to load balance their messages. For example, consider a cluster of four brokers A, B, C, and D. Each broker contains configuration for a cluster connection for a queue named OrderQueue. When a client connects to broker A, the broker will forward messages sent to OrderQueue to brokers B, C, and D. The broker distributes the messages in a round-robin fashion. The exact order depends on the order in which the brokers started. Also, you can configure a cluster connection to load balance messages only to other brokers that have a matching consumer.

You can configure the broker to automatically redistribute messages from queues that do not have any consumers to queues that do have consumers.

If message load balancing is OFF or ON_DEMAND, messages are not moved to queues that do not have consumers to consume them. However, if a matching consumer on a queue closes after the messages have been sent to the queue, the messages will stay in the queue without being consumed. This scenario is called starvation, and message redistribution can be used to move these messages to queues with matching consumers.

By default, message redistribution is disabled, but you can enable it for an address, and can configure the redistribution delay to define how often the messages should be redistributed.

Prerequisites

A configured cluster connection. See Configuring a Cluster Connection for details on creating a cluster-connection within the configuration.

Procedure

To enable message redistribution, add the following configuration to the BROKER_INSTANCE_DIR/etc/broker.xml configuration file:

  1. Verify that the cluster-connection has its message-load-balancing element set to ON_DEMAND. Add this element if it does not exist.

    <configuration>
      <core>
        ...
        <cluster-connections>
           <cluster-connection name="my-cluster">
             ...
             <message-load-balancing>ON_DEMAND</message-load-balancing>
             ...
           </cluster-connection>
        </cluster-connections>
      </core>
    </configuration>
  2. Enable message redistribution for the relevant queue by configuring an address-setting for it.

    1. First, add the parent element, address-settings, if it does not exist. This element contains all address-setting elements created for the broker.

      <configuration>
        <core>
          ...
          <address-settings></address-settings>
          ...
        </core>
      </configuration>
    2. Next add an address-setting for the queue. The value for the match attribute must a match for the name of the queue. In the example below, an address setting is created for the queue named my.queue. You can use the broker wildcard syntax instead of a literal match.

      <configuration>
        <core>
          ...
          <address-settings>
             <address-setting match="my.queue"></address-setting>
          </address-settings>
          ...
        </core>
      </configuration>
    3. Finally add the redistribution-delay element beneath the address-setting. In most cases, you should set a delay before redistributing, because it is common for a consumer to close but another one to be quickly created on the same queue. A value of 0 means the messages will be immediately redistributed. A value of -1, which is the default, signifies that messages will never be redistributed.

      <configuration>
        <core>
          ...
          <address-settings>
             <address-setting match="my.queue">
                <redistribution-delay>0</redistribution-delay>
             </address-setting>
          </address-settings>
          ...
        </core>
      </configuration>

Related Information

See queue-message-redistribution under INSTALL_DIR/examples/features/clustered for a working example of message redistribution.

20.5. Changing the Default Cluster User and Password

When creating connections between brokers of a cluster to form a cluster connection, the broker uses a cluster user and cluster password. You should change these values from their default, or remote clients will be able to make connections to the broker using the default values. The broker will also detect the default credentials when it starts, and display a warning.

Procedure

To change the cluster user and password, modify the BROKER_INSTANCE_DIR/etc/broker.xml configuration file as in the steps below.

  1. Under core, add the cluster-user and cluster-password elements.

    <configuration>
      <core>
        ...
        <cluster-user></cluster-user>
        <cluster-password></cluster-password>
        ...
      </core>
    </configuration>
  2. Add the desired values for cluster-user and cluster-password.

    <configuration>
      <core>
        ...
        <cluster-user>cluster_user</cluster-user>
        <cluster-password>cluster_user_password</cluster-password>
        ...
      </core>
    </configuration>

20.6. Using Client-Side Load Balancing

Client-side load balancing distributes client sessions created from a single connection factory to more than one broker. By connecting to different brokers, load-balanced sessions share their workloads across the cluster.

Different strategies, or policies, determine the way load-balanced sessions are distributed to brokers. As a convenience, AMQ Broker includes four Java classes, found in the package org.apache.activemq.artemis.api.core.client.loadbalance, that implement four common load balancing policies.

Table 20.1. Load Balancing Policies Included with AMQ Broker

If you want this kind of load balancing policy…​Use this class…​

Round Robin. The default policy. The first connection goes to a randomly chosen broker. Subsequent connections go to the next broker in sequence.

For example, brokers can be chosen in any of the following sequences:

  • A, B, C, D, A, …​
  • B, C, D, A, B, …​
  • C, D, A, B, C, …​
  • D, A, B, C, D, …​

RoundRobinConnectionLoadBalancingPolicy

Random. Each connection goes to a randomly chosen broker.

RandomConnectionLoadBalancingPolicy

Random Sticky. The first connection goes to a randomly chosen broker. Subsequent connections go to the same broker.

RandomStickyConnectionLoadBalancingPolicy

First Element. Connections always go to the first broker in the cluster.

FirstElementConnectionLoadBalancingPolicy

To use client-side load balancing, configure a connection factory with the fully qualified name of the class that implements the policy you want to use. It is recommended that you use one of the four classes from the package org.apache.activemq.artemis.api.core.client.loadbalance that are included with AMQ Broker. However, you can use your own policy by developing a Java class that implements the interface org.apache.activemq.artemis.api.core.client.loadbalance.ConnectionLoadBalancingPolicy. You have two choices for how to configure a connection factory to use client-side load balancing:

20.6.1. Setting the Load Balancing Policy By Using JNDI

You can use a JNDI context environment to configure a connection factory with a load balancing policy. Using JNDI is the recommended method because the context environment can be maintained in a text file that is external to your source code. Consequently, you do not need to recompile your client application after making changes to your load balancing policy.

Procedure

  1. Open the file named jndi.properties that contains configuration for the connection factory used by your clients.
  2. Append a name-value pair to the URL used by the connection factory when connecting to the broker. For the name, specify connectionLoadBalancingPolicyClassName, and for the value specify the fully qualified name of the class implementing the load balancing policy that you want to use:

    java.naming.factory.initial=org.apache.activemq.artemis.jndi.ActiveMQInitialContextFactory
    
    connection.myConnectionFactory=tcp://localhost:61616?connectionLoadBalancingPolicyClassName=org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy

20.6.2. Setting the Load Balancing Policy Programmatically

You can use the AMQ Core Protocol JMS client API to programmatically set the client-side load balancing policy used by a connection factory. A disadvantage to using the API is that you must recompile your client code before any configuration changes can take effect. It is recommended that you use JNDI to configure client-side load balancing instead. For more information, see Section 20.6.1, “Setting the Load Balancing Policy By Using JNDI”.

Procedure

  1. Open the source file that contains the instance of ActiveMQConnectionFactory used by your AMQ Core Protocol JMS clients.
  2. Add a method call to setConnectionLoadBalancingPolicyClassName() before the client code attempts a connection to the cluster. The method parameter must be the fully qualified name of the class implementing the load balancing policy that you want to use:

    ConnectionFactory jmsConnectionFactory =  ActiveMQJMSClient.createConnectionFactoryWithHA(...)
    
    jmsConnectionFactory.setConnectionLoadBalancingPolicyClassName("org.apache.activemq.artemis.api.core.client.loadbalance.RandomConnectionLoadBalancingPolicy");

20.7. Configuring Cluster Connections for Use in Various Topologies

Broker clusters can be connected together in many different topologies. However, symmetric and chain clusters are the most common. You can also scale clusters up and down without message loss.

20.7.1. Symmetric Clusters

With a symmetric cluster, every broker in the cluster is connected to every other broker in the cluster. This means that every broker in the cluster is no more than one hop away from every other broker.

To form a symmetric cluster, every broker in the cluster defines a cluster connection with the attribute max-hops set to 1. Typically, the cluster connection will use broker discovery in order to know what other brokers in the cluster it should connect to, although it is possible to explicitly define each target broker too in the cluster connection if, for example, UDP is not available on your network.

With a symmetric cluster, each broker knows about all the queues that exist on all of the other brokers, and what consumers they have. With this knowledge, it can determine how to load balance and redistribute messages around the brokers.

20.7.2. Chain Clusters

With a chain cluster, each broker in the cluster is not connected to every broker in the cluster directly. Instead, the brokers form a chain with a broker on each end of the chain and all other brokers just connecting to the previous and next brokers in the chain.

An example of a chain cluster would be a three broker chain consisting of brokers A, B and C. broker A is hosted in one network and has many producer clients connected to it sending order messages. Due to corporate policy, the order consumer clients need to be hosted in a different network, and that network is only accessible via a third network. In this setup, broker B acts as a mediator with no producers or consumers on it. Any messages arriving on broker A will be forwarded to broker B, which in turn forwards them to broker C where they can be consumed. broker A does not need to directly connect to C, but all of the brokers can still act as a part of the cluster.

To set up a cluster in this way, broker A would define a cluster connection that connects to broker B, and broker B would define a cluster connection that connects to broker C. In this case, the cluster connections only need to be in one direction, because messages are moving from broker A→B→C and never from C→B→A.

For this topology, you would set max-hops to 2. With a value of 2, the knowledge of what queues and consumers that exist on broker C would be propagated from broker C to broker B to broker A. broker A would then know to distribute messages to broker B when they arrive, even though broker B has no consumers itself. It would know that a further hop away is broker C, which does have consumers.

20.7.3. Scaling Clusters

If the size of a cluster changes frequently in your environment, you can scale it up or down with no message loss (even for non-durable messages).

You can scale up clusters by adding brokers, in which case there is no risk of message loss. Similarly, you can scale clusters down by removing brokers. However, to prevent message loss, you must first configure the broker to send its messages to another broker in the cluster.

To scale down a cluster:

  1. On the broker you want to scale down, open the BROKER_INSTANCE_DIR/etc/broker.xml file and set enabled to true.
  2. If necessary, set group-name to the name of the group that contains the broker to which you want the messages to be sent.

    If cluster brokers are grouped together with different group-name values, be careful how you set this parameter. If all of the brokers in a single group are shut down, then the messages from that broker or group will be lost.

  3. If the broker is using multiple cluster connections, then set scale-down-clustername to identify the name of the cluster-connection which should be used for scaling down.
  4. Shut down the broker gracefully using the BROKER_INSTANCE_DIR/artemis stop command.

    The broker finds another broker in the cluster and sends all of its messages (both durable and non-durable) to that broker. The messages are processed in order and go to the back of the respective queues on the other broker (just as if the messages were sent from an external client for the first time).