Red Hat Training

A Red Hat training course is available for Red Hat Enterprise Linux

9.5. Pacemaker Support for Docker Containers (Technology Preview)

Important

Pacemaker support for Docker containers is provided for technology preview only. For details on what "technology preview" means, see Technology Preview Features Support Scope.
There is one exception to this feature being Technology Preview: As of Red Hat Enterprise Linux 7.4, Red Hat fully supports the usage of Pacemaker bundles for Red Hat Openstack Platform (RHOSP) deployments.
Pacemaker supports a special syntax for launching a Docker container with any infrastructure it requires: the bundle. After you have created a Pacemaker bundle, you can create a Pacemaker resource that the bundle encapsulates.

9.5.1. Configuring a Pacemaker Bundle Resource

The syntax for the command to create a Pacemaker bundle for a Docker container is as follows. This command creates a bundle that encapsulates no other resources. For information on creating a cluster resource in a bundle see Section 9.5.2, “Configuring a Pacemaker Resource in a Bundle”.
pcs resource bundle create bundle_id container docker [container_options] [network network_options] [port-map port_options]...  [storage-map storage_options]... [meta meta_options] [--disabled] [--wait[=n]]
The required bundle_id parameter must be a unique name for the bundle. If the --disabled option is specified, the bundle is not started automatically. If the --wait option is specified, Pacemaker will wait up to n seconds for the bundle to start and then return 0 on success or 1 on error. If n is not specified it defaults to 60 minutes.
The following sections describe the parameters you can configure for each element of a Pacemaker bundle.

9.5.1.1. Docker Parameters

Table 9.6, “Docker Container Parameters” describes the docker container options you can set for a bundle.

Note

Before configuring a docker bundle in Pacemaker, you must install Docker and supply a fully configured Docker image on every node allowed to run the bundle.

Table 9.6. Docker Container Parameters

FieldDefaultDescription
image
Docker image tag (required)
replicas
Value of promoted-max if that is positive, otherwise 1.
A positive integer specifying the number of container instances to launch
replicas-per-host
1
A positive integer specifying the number of container instances allowed to run on a single node
promoted-max
0
A non-negative integer that, if positive, indicates that the containerized service should be treated as a multistate service, with this many replicas allowed to run the service in the master role
network
If specified, this will be passed to the docker run command as the network setting for the Docker container.
run-command
/usr/sbin/pacemaker_remoted if the bundle contains a resource, otherwise none
This command will be run inside the container when launching it ("PID 1"). If the bundle contains a resource, this command must start the pacemaker_remoted daemon (but it could, for example, be a script that performs others tasks as well).
options
Extra command-line options to pass to the docker run command

9.5.1.2. Bundle Network Parameters

Table 9.7, “Bundle Resource Network Parameters” describes the network options you can set for a bundle.

Table 9.7. Bundle Resource Network Parameters

FieldDefaultDescription
add-host
TRUE
If TRUE, and ip-range-start is used, Pacemaker will automatically ensure that the /etc/hosts file inside the containers has entries for each replica name and its assigned IP.
ip-range-start
If specified, Pacemaker will create an implicit ocf:heartbeat:IPaddr2 resource for each container instance, starting with this IP address, using as many sequential addresses as were specified as the replicas parameter for the Docker element. These addresses can be used from the host’s network to reach the service inside the container, although it is not visible within the container itself. Only IPv4 addresses are currently supported.
host-netmask
32
If ip-range-start is specified, the IP addresses are created with this CIDR netmask (as a number of bits).
host-interface
If ip-range-start is specified, the IP addresses are created on this host interface (by default, it will be determined from the IP address).
control-port
3121
If the bundle contains a Pacemaker resource, the cluster will use this integer TCP port for communication with Pacemaker Remote inside the container. Changing this is useful when the container is unable to listen on the default port, which could happen when the container uses the host’s network rather than ip-range-start (in which case replicas-per-host must be 1), or when the bundle may run on a Pacemaker Remote node that is already listening on the default port. Any PCMK_remote_port environment variable set on the host or in the container is ignored for bundle connections.
When a Pacemaker bundle configuration uses the control-port parameter, then if the bundle has its own IP address the port needs to be open on that IP address on and from all full cluster nodes running corosync. If, instead, the bundle has set the network="host" container parameter, the port needs to be open on each cluster node's IP address from all cluster nodes.

Note

Replicas are named by the bundle ID plus a dash and an integer counter starting with zero. For example, if a bundle named httpd-bundle has configured replicas=2, its containers will be named httpd-bundle-0 and httpd-bundle-1.
In addition to the network parameters, you can optionally specify port-map parameters for a bundle. Table 9.8, “Bundle Resource port-map Parameters” describes these port-map parameters.

Table 9.8. Bundle Resource port-map Parameters

FieldDefaultDescription
id
A unique name for the port mapping (required)
port
If this is specified, connections to this TCP port number on the host network (on the container’s assigned IP address, if ip-range-start is specified) will be forwarded to the container network. Exactly one of port or range must be specified in a port-mapping.
internal-port
Value of port
If port and internal-port are specified, connections to port on the host’s network will be forwarded to this port on the container network.
range
If range is specified, connections to these TCP port numbers (expressed as first_port-last_port) on the host network (on the container’s assigned IP address, if ip-range-start is specified) will be forwarded to the same ports in the container network. Exactly one of port or range must be specified in a port mapping.

Note

If the bundle contains a resource, Pacemaker will automatically map the control-port, so it is not necessary to specify that port in a port mapping.

9.5.1.3. Bundle Storage Parameters

You can optionally configure storage-map parameters for a bundle. Table 9.9, “Bundle Resource Storage Mapping Parameters” describes these parameters.

Table 9.9. Bundle Resource Storage Mapping Parameters

FieldDefaultDescription
id
A unique name for the storage mapping (required)
source-dir
The absolute path on the host’s filesystem that will be mapped into the container. Exactly one of source-dir and source-dir-root parameter must be specified when configuring a storage-map parameter.
source-dir-root
The start of a path on the host’s filesystem that will be mapped into the container, using a different subdirectory on the host for each container instance. The subdirectory will be named with the same name as the bundle name, plus a dash and an integer counter starting with 0. Exactly one source-dir and source-dir-root parameter must be specified when configuring a storage-map parameter.
target-dir
The path name within the container where the host storage will be mapped (required)
options
File system mount options to use when mapping the storage
As an example of how subdirectories on a host are named using the source-dir-root parameter, if source-dir-root=/path/to/my/directory, target-dir=/srv/appdata, and the bundle is named mybundle with replicas=2, then the cluster will create two container instances with host names mybundle-0 and mybundle-1 and create two directories on the host running the containers: /path/to/my/directory/mybundle-0 and /path/to/my/directory/mybundle-1. Each container will be given one of those directories, and any application running inside the container will see the directory as /srv/appdata.

Note

Pacemaker does not define the behavior if the source directory does not already exist on the host. However, it is expected that the container technology or its resource agent will create the source directory in that case.

Note

If the bundle contains a Pacemaker resource, Pacemaker will automatically map the equivalent of source-dir=/etc/pacemaker/authkeytarget-dir=/etc/pacemaker/authkey and source-dir-root=/var/log/pacemaker/bundlestarget-dir=/var/log into the container, so it is not necessary to specify those paths in when configuring storage-map parameters.

Important

The PCMK_authkey_location environment variable must not be set to anything other than the default of /etc/pacemaker/authkey on any node in the cluster.

9.5.2. Configuring a Pacemaker Resource in a Bundle

A bundle may optionally contain one Pacemaker cluster resource. As with a resource that is not contained in a bundle, the cluster resource may have operations, instance attributes, and metadata attributes defined. If a bundle contains a resource, the container image must include the Pacemaker Remote daemon, and ip-range-start or control-port must be configured in the bundle. Pacemaker will create an implicit ocf:pacemaker:remote resource for the connection, launch Pacemaker Remote within the container, and monitor and manage the resource by means of Pacemaker Remote. If the bundle has more than one container instance (replica), the Pacemaker resource will function as an implicit clone, which will be a multistate clone if the bundle has configured the promoted-max option as greater than zero.
You create a resource in a Pacemaker bundle with the pcs resource create command by specifying the bundle parameter for the command and the bundle ID in which to include the resource. For an example of creating a Pacemaker bundle that contains a resource, see Section 9.5.4, “Pacemaker Bundle Configuration Example”.

Important

Containers in bundles that contain a resource must have an accessible networking environment, so that Pacemaker on the cluster nodes can contact Pacemaker Remote inside the container. For example, the docker option --net=none should not be used with a resource. The default (using a distinct network space inside the container) works in combination with the ip-range-start parameter. If the docker option --net=host is used (making the container share the host’s network space), a unique control-port parameter should be specified for each bundle. Any firewall must allow access to the control-port.

9.5.2.1. Node Attributes and Bundle Resources

If the bundle contains a cluster resource, the resource agent may want to set node attributes such as master scores. However, with containers, it is not apparent which node should get the attribute.
If the container uses shared storage that is the same no matter which node the container is hosted on, then it is appropriate to use the master score on the bundle node itself. On the other hand, if the container uses storage exported from the underlying host, then it may be more appropriate to use the master score on the underlying host. Since this depends on the particular situation, the container-attribute-target resource metadata attribute allows the user to specify which approach to use. If it is set to host, then user-defined node attributes will be checked on the underlying host. If it is anything else, the local node (in this case the bundle node) is used. This behavior applies only to user-defined attributes; the cluster will always check the local node for cluster-defined attributes such as #uname.
If container-attribute-target is set to host, the cluster will pass additional environment variables to the resource agent that allow it to set node attributes appropriately.

9.5.2.2. Metadata Attributes and Bundle Resources

Any metadata attribute set on a bundle will be inherited by the resource contained in a bundle and any resources implicitly created by Pacemaker for the bundle. This includes options such as priority, target-role, and is-managed.

9.5.3. Limitations of Pacemaker Bundles

Pacemaker bundles operate with the following limitations:
  • Bundles may not be included in groups or explicitly cloned with a pcs command. This includes a resource that the bundle contains, and any resources implicitly created by Pacemaker for the bundle. Note, however, that if a bundle is configured with a value of replicas greater than one, the bundle behaves as if it were a clone.
  • Restarting Pacemaker while a bundle is unmanaged or the cluster is in maintenance mode may cause the bundle to fail.
  • Bundles do not have instance attributes, utilization attributes, or operations, although a resource contained in a bundle may have them.
  • A bundle that contains a resource can run on a Pacemaker Remote node only if the bundle uses a distinct control-port.

9.5.4. Pacemaker Bundle Configuration Example

The following example creates a Pacemaker bundle resource with a bundle ID of httpd-bundle that contains an ocf:heartbeat:apache resource with a resource ID of httpd.
This procedure requires the following prerequisite configuration:
  • Docker has been installed and enabled on every node in the cluster.
  • There is an existing Docker image, named pcmktest:http
  • The container image includes the Pacemaker Remote daemon.
  • The container image includes a configured Apache web server.
  • Every node in the cluster has directories /var/local/containers/httpd-bundle-0, /var/local/containers/httpd-bundle-1, and /var/local/containers/httpd-bundle-2, containing an index.html file for the web server root. In production, a single, shared document root would be more likely, but for the example this configuration allows you to make the index.html file on each host different so that you can connect to the web server and verify which index.html file is being served.
This procedure configures the following parameters for the Pacemaker bundle:
  • The bundle ID is httpd-bundle.
  • The previously-configured Docker container image is pcmktest:http.
  • This example will launch three container instances.
  • This example will pass the command-line option --log-driver=journald to the docker run command. This parameter is not required, but is included to show how to pass an extra option to the docker command. A value of --log-driver=journald means that the system logs inside the container will be logged in the underlying hosts's systemd journal.
  • Pacemaker will create three sequential implicit ocf:heartbeat:IPaddr2 resources, one for each container image, starting with the IP address 192.168.122.131.
  • The IP addresses are created on the host interface eth0.
  • The IP addresses are created with a CIDR netmask of 24.
  • This example creates a port map ID of http-port; connections to port 80 on the container's assigned IP address will be forwarded to the container network.
  • This example creates a storage map ID of httpd-root. For this storage mapping:
    • The value of source-dir-root is /var/local/containers, which specifies the start of the path on the host's file system that will be mapped into the container, using a different subdirectory on the host for each container instance.
    • The value of target-dir is /var/www/html, which specifies the path name within the container where the host storage will be mapped.
    • The file system rw mount option will be used when mapping the storage.
    • Since this example container includes a resource, Pacemaker will automatically map the equivalent of source-dir=/etc/pacemaker/authkey in the container, so you do not need to specify that path in the storage mapping.
In this example, the existing cluster configuration is put into a temporary file named temp-cib.xml, which is then copied to a file named temp-cib.xml.deltasrc. All modifications to the cluster configuration are made to the tmp-cib.xml file. When the udpates are complete, this procedure uses the diff-against option of the pcs cluster cib-push command so that only the updates to the configuration file are pushed to the active configuration file.
# pcs cluster cib tmp-cib.xml
# cp tmp-cib.xml tmp-cib.xml.deltasrc
# pcs -f tmp.cib.xml resource bundle create httpd-bundle \
container docker image=pcmktest:http replicas=3 \
options=--log-driver=journald \
network ip-range-start=192.168.122.131 host-interface=eth0 \
host-netmask=24 port-map id=httpd-port port=80 \
storage-map id=httpd-root source-dir-root=/var/local/containers \
target-dir=/var/www/html options=rw \
# pcs -f tmp-cib.xml resource create httpd ocf:heartbeat:apache \
statusurl=http://localhost/server-status bundle httpd-bundle
# pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.xml.deltasrc