Chapter 17. High Availability

When using replication as the HA policy for your cluster, all data synchronization occurs over the network. A slave broker first needs to synchronize all existing data from the master broker before becoming capable of replacing it. The time it takes for this to happen depends on the amount of data to be synchronized and the connection speed. In general, synchronization occurs in parallel with current network traffic, so this does not cause any blocking on clients. However, synchronization blocks journal-related operations. The maximum length of time that this exchange blocks journal-related operations is controlled by the initial-replication-sync-timeout configuration element. There is a critical moment at the end of this process where the master broker must complete the synchronization and ensure the slave broker acknowledges the completion. After the initial synchronization phase is complete and the slave broker has all the current data from the master, then the master synchronizes data across the network as it receives it, to ensure that no data is lost if the master broker fails.

Figure 17.1. Replicated Store for High Availability

The advantage of using replication is that it doesn’t require a shared file system that is accessible by both the master and slave broker. The disadvantage is that performance can be reduced while data is replicated between a master broker and a slave broker.

The replicating master and slave pair must be part of a cluster. The cluster connection also defines how a slave broker identifies a master broker to which it can connect. See Clusters for details on how to configure a cluster connection.

Within a cluster using data replication, there are two ways that a slave broker locates a master broker:

The slave searches for any master broker that it is configured to connect to. It then tries to replicate with each master broker in turn until it finds a master broker that has no current slave configured. If no master broker is available it waits until the cluster topology changes and repeats the process.

When the master broker stops or crashes, its replicating slave becomes active and take over its duties. Specifically, the slave becomes active when it loses connection to its master broker. This can be problematic because a connection loss might be due to a temporary network problem. In order to address this issue, the slave tries to determine whether it can connect to the other brokers in the cluster. If it can connect to more than half the brokers, it becomes active. If it can connect to fewer than half, the slave does not become active but tries to reconnect with the master. This avoids a split-brain situation.

When using a shared-store high availability policy, both master and slave brokers access a single directory on a shared file system that is accessible by both the master and slave nodes. Typically, this is some kind of Storage Area Network (SAN), which uses a dedicated high-performance network to provide block-level storage. In general, Network Attached Storage (NAS) is not recommended in scenarios where optimal throughput is required. Depending on the network’s configuration, NAS performance can be slower when compared to SAN.

If a master broker encounters a failure, a slave broker loads the data from the shared file system and takes over for the failed master. Clients can then reconnect to the slave broker and resume handling requests.

The advantage of using shared-store high availability is that there is no performance overhead incurred for replicating data between a master broker and a slave broker. Instead, a master and a slave broker read and write data to and from the shared store independently.

The disadvantage of shared-store replication is that it requires a shared file system. And when the slave broker takes over, it needs to load the journal from the shared-store, which can take some time, depending on the amount of data.

Figure 17.2. shared-store for High Availability

It is also possible to colocate slave brokers in the same JVM as a master broker. A master broker can be configured to request another master to start a slave broker that resides in its Java Virtual Machine. You can colocate slave brokers using either shared-store or replication as your HA policy. The new slave broker inherits its configuration from the master broker creating it. The name of the slave is set to colocated_backup_n where n is the number of backups the master broker has created.

The slave inherits configuration for its connectors and acceptors from the master broker creating it. However, AMQ Broker applies a default port offset of 100 for each. For example, if the master contains configuration for a connection that uses port 61616, the first slave created uses port 61716, the second uses 61816, and so on.

Directories for the journal, large messages, and paging are set according to the HA strategy you choose. If you choose shared-store, the requesting broker notifies the target broker which directories to use. If replication is chosen, directories are inherited from the creating broker and have the new backup’s name appended to them.

Figure 17.3. Co-located Master and Slave Brokers

You can configure brokers to scale down as an alternative to using a replication or shared-store HA policy. When configured for scale down, a master broker copies its messages and transaction state to another master broker before shutting down. The advantage of scale down is that you do not need full backups to provide some form of HA. However, scaling down handles only cases where a broker stops gracefully. It is not made to handle an unexpected failure gracefully.

Another disadvantage is that it is possible to lose message ordering when scaling down. This happens because the messages in the broker that is scaling down are appended to the end of the queues of the other broker. For example, two master brokers have ten messages distributed evenly between them. If one of the brokers scales down, the messages sent to the other broker are added to queue after the ones already there. Consequently, after Broker 2 scales down, the order of the messages in Broker 1 would be 1, 3, 5, 7, 9, 2, 4, 6, 8, 10.

When a broker is preparing to scale down, it sends a message to its clients before they are disconnected informing them which new broker is ready to process their messages. However, clients should reconnect to the new broker only after their initial broker has finished scaling down. This ensures that any state, such as queues or transactions, is available on the other broker when the client reconnects. The normal reconnect settings apply when the client is reconnecting so these should be high enough to deal with the time needed to scale down.

Figure 17.4. Scaling Down Master Brokers

<configuration>
  <core>
    ...
    <ha-policy>
       <replication>
          <colocated>
             <backup-request-retries>44</backup-request-retries>
             <backup-request-retry-interval>33</backup-request-retry-interval>
             <max-backups>3</max-backups>
             <request-backup>false</request-backup>
             <backup-port-offset>33</backup-port-offset>
             <master>
                <group-name>purple</group-name>
                <check-for-live-server>true</check-for-live-server>
                <cluster-name>abcdefg</cluster-name>
             </master>
             <slave>
                <group-name>tiddles</group-name>
                <max-saved-replicated-journals-size>22</max-saved-replicated-journals-size>
                <cluster-name>33rrrrr</cluster-name>
                <restart-backup>false</restart-backup>
                <scale-down>
                   <!--a grouping of servers that can be scaled down to-->
                   <group-name>boo!</group-name>
                   <!--either a discovery group-->
                   <discovery-group-ref discovery-group-name="wahey"/>
                </scale-down>
             </slave>
          </colocated>
       </replication>
    </ha-policy>
    ...
  </core>
</configuration>