8.4. The pacemaker_remote Service

The pacemaker_remote service allows nodes not running corosync to integrate into the cluster and have the cluster manage their resources just as if they were real cluster nodes.
Among the capabilities that the pacemaker_remote service provides are the following:
  • The pacemaker_remote service allows you to scale beyond the corosync 16-node limit.
  • The pacemaker_remote service allows you to manage a virtual environment as a cluster resource and also to manage individual services within the virtual environment as cluster resources.
The following terms are used to describe the pacemaker_remote service.
  • cluster node — A node running the High Availability services (pacemaker and corosync).
  • remote node — A node running pacemaker_remote to remotely integrate into the cluster without requiring corosync cluster membership. A remote node is configured as a cluster resource that uses the ocf:pacemaker:remote resource agent.
  • guest node — A virtual guest node running the pacemaker_remote service. A guest node is configured using the remote-node metadata option of a resource agent such as ocf:pacemaker:VirtualDomain. The virtual guest resource is managed by the cluster; it is both started by the cluster and integrated into the cluster as a remote node.
  • pacemaker_remote — A service daemon capable of performing remote application management within remote nodes and guest nodes (KVM and LXC) in a Pacemaker cluster environment. This service is an enhanced version of Pacemaker’s local resource management daemon (LRMD) that is capable of managing resources remotely on a node not running corosync.
  • LXC — A Linux Container defined by the libvirt-lxc Linux container driver.
A Pacemaker cluster running the pacemaker_remote service has the following characteristics.
  • The remote nodes and/or the guest nodes run the pacemaker_remote service (with very little configuration required on the virtual machine side).
  • The cluster stack (pacemaker and corosync), running on the cluster nodes, connects to the pacemaker_remote service on the remote nodes, allowing them to integrate into the cluster.
  • The cluster stack (pacemaker and corosync), running on the cluster nodes, launches the guest nodes and immediately connects to the pacemaker_remote service on the guest nodes, allowing them to integrate into the cluster.
The key difference between the cluster nodes and the remote and guest nodes that the cluster nodes manage is that the remote and guest nodes are not running the cluster stack. This means the remote and guest nodes have the following limitations:
  • they do not take place in quorum
  • they do not execute fencing device actions
  • they are not eligible to be be the cluster's Designated Controller (DC)
  • they do not themselves run the full range of pcs commands
On the other hand, remote nodes and guest nodes are not bound to the scalability limits associated with the cluster stack.
Other than these noted limitations, the remote nodes behave just like cluster nodes in respect to resource management, and the remote and guest nodes can themselves be fenced. The cluster is fully capable of managing and monitoring resources on each remote and guest node: You can build constraints against them, put them in standby, or perform any other action you perform on cluster nodes with the pcs commands. Remote and guest nodes appear in cluster status output just as cluster nodes do.

8.4.1. Host and Guest Authentication

The connection between cluster nodes and pacemaker_remote is secured using Transport Layer Security (TLS) with pre-shared key (PSK) encryption and authentication over TCP (using port 3121 by default). This means both the cluster node and the node running pacemaker_remote must share the same private key. By default this key must be placed at /etc/pacemaker/authkey on both cluster nodes and remote nodes.

8.4.2. Guest Node Resource Options

When configuring a virtual machine or LXC resource to act as a guest node, you create a VirtualDomain resource, which manages the virtual machine. For descriptions of the options you can set for a VirtualDomain resource, use the following command.
# pcs resource describe VirtualDomain
In addition to the VirtualDomain resource options, you can configure metadata options to both enable the resource as a guest node and define the connection parameters. Table 8.4, “Metadata Options for Configuring KVM/LXC Resources as Remote Nodes” describes these metadata options.

Table 8.4. Metadata Options for Configuring KVM/LXC Resources as Remote Nodes

FieldDefaultDescription
remote-node
<none>
The name of the guest node this resource defines. This both enables the resource as a guest node and defines the unique name used to identify the guest node. WARNING: This value cannot overlap with any resource or node IDs.
remote-port
3121
Configures a custom port to use for the guest connection to pacemaker_remote.
remote-addr
remote-node value used as host name
The IP address or host name to connect to if remote node’s name is not the host name of the guest
remote-connect-timeout
60s
Amount of time before a pending guest connection will time out

8.4.3. Remote Node Resource Options

You configure a remote node as a cluster resource with the pcs resource create command, specifying ocf:pacemaker:remote as the resource type. Table 8.5, “Resource Options for Remote Nodes” describes the resource options you can configure for a remote resource.

Table 8.5. Resource Options for Remote Nodes

FieldDefaultDescription
reconnect_interval
0
Time in seconds to wait before attempting to reconnect to a remote node after an active connection to the remote node has been severed. This wait is recurring. If reconnect fails after the wait period, a new reconnect attempt will be made after observing the wait time. When this option is in use, Pacemaker will keep attempting to reach out and connect to the remote node indefinitely after each wait interval.
server
Server location to connect to. This can be an IP address or host name.
port
TCP port to connect to.

8.4.4. Changing Default pacemaker_remote Options

If you need to change the default port or authkey location for either Pacemaker or pacemaker_remote, there are environment variables you can set that affect both of those daemons. These environment variables can be enabled by placing them in the /etc/sysconfig/pacemaker file as follows.
#==#==# Pacemaker Remote
# Use a custom directory for finding the authkey.
PCMK_authkey_location=/etc/pacemaker/authkey
#
# Specify a custom port for Pacemaker Remote connections
PCMK_remote_port=3121
Note that when you change the default key location on a particular node (cluster node, guest node or remote node), it is sufficient to set PCMK_authkey_location on that node (and put the key in that location). It is not necessary that the location be the same on every node, although doing so makes administration easier.
When changing the default port used by a particular guest node or remote node, the PCMK_remote_port variable must be set in that node's /etc/sysconfig/pacemaker file, and the cluster resource creating the guest node or remote node connection must also be configured with the same port number (using the remote-port metadata option for guest nodes, or the port option for remote nodes).

8.4.5. Configuration Overview: KVM Guest Node

This section provides a high-level summary overview of the steps to perform to have Pacemaker launch a virtual machine and to integrate that machine as a guest node, using libvirt and KVM virtual guests.
  1. After installing the virtualization software and enabling the libvirtd service on the cluster nodes, put the same encryption key with the path /etc/pacemaker/authkey on every cluster node and virtual machine. This secures remote communication and authentication.
    Run the following set of commands on every node to create the authkey directory with secure permissions.
    # mkdir -p --mode=0750 /etc/pacemaker
    # chgrp haclient /etc/pacemaker
    The following command shows one method to create an encryption key. You should create the key only once and then copy it to all of the nodes.
    # dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1
  2. On every virtual machine, install pacemaker_remote packages, start the pacemaker_remote service and enable it to run on startup, and allow TCP port 3121 through the firewall.
    # yum install pacemaker-remote resource-agents
    # systemctl start pacemaker_remote.service
    # systemctl enable pacemaker_remote.service
    # firewall-cmd --add-port 3121/tcp --permanent
    # firewall-cmd --reload
  3. Give each virtual machine a static network address and unique host name, which should be known to all nodes. For information on setting a static IP address for the guest virtual machine, see the Virtualization Deployment and Administration Guide.
  4. To create the VirtualDomain resource agent for the management of the virtual machine, Pacemaker requires the virtual machine's xml config file to be dumped to a file on disk. For example, if you created a virtual machine named guest1, dump the xml to a file somewhere on the host. You can use a file name of your choosing; this example uses /etc/pacemaker/guest1.xml.
    # virsh dumpxml guest1 > /etc/pacemaker/guest1.xml
  5. If it is running, shut down the guest node. Pacemaker will start the node when it is configured in the cluster.
  6. Create the VirtualDomain resource, configuring the remote-note resource meta option to indicate that the virtual machine is a guest node capable of running resources.
    In the example below, the meta-attribute remote-node=guest1 tells pacemaker that this resource is a guest node with the host name guest1 that is capable of being integrated into the cluster. The cluster will attempt to contact the virtual machine’s pacemaker_remote service at the host name guest1 after it launches.
    From a cluster node, enter the following command.
    # pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="/virtual_machines/vm-guest1.xml" meta remote-node=guest1
  7. After creating the VirtualDomain resource, you can treat the guest node just as you would treat any other node in the cluster. For example, you can create a resource and place a resource constraint on the resource to run on the guest node as in the following commands, which are run from a cluster node. As of Red Hat Enterprise Linux 6.8, you can include guest nodes in groups, which allows you to group a storage device, file system, and VM.
    # pcs resource create webserver apache params configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s
    # pcs constraint location webserver prefers guest1

8.4.6. Configuration Overview: Remote Node

This section provides a high-level summary overview of the steps to perform to configure a Pacemaker remote node and to integrate that node into an existing Pacemaker cluster environment.
  1. On the node that you will be configuring as a remote node, allow cluster-related services through the local firewall.
    # firewall-cmd --permanent --add-service=high-availability
    success
    # firewall-cmd --reload
    success
    

    Note

    If you are using iptables directly, or some other firewall solution besides firewalld, simply open the following ports, which can be used by various clustering components: TCP ports 2224, 3121, and 21064, and UDP port 5405.
  2. Install the pacemaker_remote daemon on the remote node.
    # yum install -y pacemaker-remote resource-agents pcs
  3. All nodes (both cluster nodes and remote nodes) must have the same authentication key installed for the communication to work correctly. If you already have a key on an existing node, use that key and copy it to the remote node. Otherwise, create a new key on the remote node.
    Run the following set of commands on the remote node to create a directory for the authentication key with secure permissions.
    # mkdir -p --mode=0750 /etc/pacemaker
    # chgrp haclient /etc/pacemaker
    The following command shows one method to create an encryption key on the remote node.
    # dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1
  4. Start and enable the pacemaker_remote daemon on the remote node.
    # systemctl enable pacemaker_remote.service
    # systemctl start pacemaker_remote.service
  5. On the cluster node, create a location for the shared authentication key with the same path as the authentication key on the remote node and copy the key into that directory. In this example, the key is copied from the remote node where the key was created.
    # mkdir -p --mode=0750 /etc/pacemaker
    # chgrp haclient /etc/pacemaker
    # scp remote1:/etc/pacemaker/authkey /etc/pacemaker/authkey
  6. Run the following command from a cluster node to create a remote resource. In this case the remote node is remote1.
    # pcs resource create remote1 ocf:pacemaker:remote
    
  7. After creating the remote resource, you can treat the remote node just as you would treat any other node in the cluster. For example, you can create a resource and place a resource constraint on the resource to run on the remote node as in the following commands, which are run from a cluster node.
    # pcs resource create webserver apache params configfile=/etc/httpd/conf/httpd.conf op monitor interval=30s
    # pcs constraint location webserver prefers remote1

    Warning

    Never involve a remote node connection resource in a resource group, colocation constraint, or order constraint.
  8. Configure fencing resources for the remote node. Remote nodes are fenced the same way as cluster nodes. Configure fencing resources for use with remote nodes the same as you would with cluster nodes. Note, however, that remote nodes can never initiate a fencing action. Only cluster nodes are capable of actually executing a fencing operation against another node.

8.4.7. System Upgrades and pacemaker_remote

As of Red Hat Enterprise Linux 6.8, if the pacemaker_remote service is stopped on an active Pacemaker Remote node, the cluster will gracefully migrate resources off the node before stopping the node. This allows you to perform software upgrades and other routine maintenance procedures without removing the node from the cluster. Once pacemaker_remote is shut down, however, the cluster will immediately try to reconnect. If pacemaker_remote is not restarted within the resource's monitor timeout, the cluster will consider the monitor operation as failed.
If you wish to avoid monitor failures when the pacemaker_remote service is stopped on an active Pacemaker Remote node, you can use the following procedure to take the node out of the cluster before performing any system administration that might stop pacemaker_remote

Warning

For Red Hat Enterprise Linux release 6.7 and earlier, if pacemaker_remote stops on a node that is currently integrated into a cluster, the cluster will fence that node. If the stop happens automatically as part of a yum update process, the system could be left in an unusable state (particularly if the kernel is also being upgraded at the same time as pacemaker_remote). For Red Hat Enterprise Linux release 6.7 and earlier you must use the following procedure to take the node out of the cluster before performing any system administration that might stop pacemaker_remote.
Use the following procedure to take a node out of a cluster when performing maintenance on a node running pacemaker_remote:
  1. Stop the node's connection resource with the pcs resource disable resourcename, which will move all services off the node. For guest nodes, this will also stop the VM, so the VM must be started outside the cluster (for example, using virsh) to perform any maintenance.
  2. Perform the desired maintenance.
  3. When ready to return the node to the cluster, re-enable the resource with the pcs resource enable.

8.4.8. Converting a VM Resource to a Guest Node

Use the following command to convert an existing VirtualDomain resource into a guest node. You do not need to run this command if the resource was originally created as a guest node.
pcs cluster remote-node add hostname resource_id [options]
Use the following command to disable a resource configured as a guest node on the specified host.
pcs cluster remote-node remove hostname