Chapter 7. Message migration when scaling down pods

Important

Message migration, which is enabled by the use of the scaledown controller, is currently a Technology Preview feature. Technology Preview features are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them for production. For more information about technology preview at Red Hat, see Technology Preview Support Scope.

When a persistent template is used to deploy a broker pod that uses a StatefulSet, that broker pod has its own external file system, which remains intact, even if the pod stops or restarts. However, if pods are scaled down and not restarted, data and information such as messages, destinations, or transactions are no longer available to clients.

Message migration addresses the issue of unavailable data and can be obtained by applying the scaledown controller image, which monitors each broker pod. If a broker pod is scaled down or stopped, the scaledown controller recovers the messages by transferring its contents to another broker running in the cluster.

If broker pods are scaled down to 0 (zero), message migration does not occur, since there is no running broker pod to which the message data can be migrated.

7.1. Installing the scaledown controller

AMQ Broker on OCP message migration capabilites are packaged in the scaledown controller container image. This section describes how to enable the broker message migration capabilities on OpenShift Container Platform image streams and application templates.

Procedure

  1. Log in to OpenShift as a cluster administrator (or as a user that has project administrator access to the global OpenShift project), for example: 

    $ oc login -u system:admin

  2. Run the following command to update the AMQ Broker templates: 

    $ oc create -n amq-demo -f https://raw.githubusercontent.com/jboss-container-images/jboss-amq-7-broker-openshift-image/72-1.0.GA/templates/amq-broker-72-persistence-clustered-controller.yaml
    Note

    You could receive an "already exists" error messages after invoking the create command.

7.2. Using the scaledown controller

To deploy the scaledown controller to migrate messages and drain pods, run the the StatefulSet scaledown controller at the broker pod namespace level. The StatefulSet scaledown controller must be deployed in the same namespace as the stateful applications (in this case, broker pods). It operates only on StatefulSets in that namespace.

Note

You do not need cluster-level privileges to complete this procedure. You must run the StatefulSet scaledown controller at the namespace level.

Prerequisites

Procedure

  1. Configure the Broker on OCP StatefulSet controller template in your namespace.

  2. Configure the scaledown controller template in your StatefulSet definition. The following code example represents the drainer pod definition: 

    apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-statefulset
  annotations:
    statefulsets.kubernetes.io/drainer-pod-template: |
      {
        "metadata": {
          "labels": {
            "app": "datastore-drainer"
          }
        },
        "spec": {
          "containers": [
            {
              "name": "drainer",
              "image": "my-drain-container",
              "volumeMounts": [
                {
                  "name": "data",
                  "mountPath": "/var/data"
                }
              ]
            }
          ]
        }
      }
spec: