Chapter 9. AMQ Streams and Kafka upgrades

Procedure

1. Backup the existing Cluster Operator resources:

   oc get all -l app=strimzi -o yaml > strimzi-backup.yaml

2. Update the Cluster Operator.

   Modify the installation files according to the OpenShift project or Kubernetes namespace the Cluster Operator is running in.

   On Linux, use:

   sed -i 's/namespace: .*/namespace: my-namespace/' install/cluster-operator/*RoleBinding*.yaml

   On MacOS, use:

   sed -i '' 's/namespace: .*/namespace: my-namespace/' install/cluster-operator/*RoleBinding*.yaml

   If you modified one or more environment variables in your existing Cluster Operator Deployment, edit the install/cluster-operator/050-Deployment-cluster-operator.yaml file to reflect the changes that you made in the new version of the Cluster Operator.

3. When you have an updated configuration, deploy it along with the rest of the install resources:

   oc apply -f install/cluster-operator

   Wait for the rolling updates to complete.

4. Get the image for the Kafka pod to ensure the upgrade was successful:

   oc get po my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   The image tag shows the new AMQ Streams version followed by the Kafka version. For example, <New AMQ Streams version>-kafka-<Current Kafka version>.

5. Update existing resources to handle deprecated custom resource properties.

   • AMQ Streams resource upgrades

Procedure

1. Update the Kafka cluster configuration in an editor, as required:

   On OpenShift, use:

   oc edit kafka my-cluster

   1. If the log.message.format.version of the current Kafka version is the same as that of the new Kafka version, proceed to the next step.

      Otherwise, ensure that Kafka.spec.kafka.config has the log.message.format.version configured to the default for the current version.

      For example, if upgrading from Kafka 2.1.1:

      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.1.1
          config:
            log.message.format.version: "2.1"
            # ...

      If the log.message.format.version is unset, set it to the current version.

      Note

      The value of log.message.format.version must be a string to prevent it from being interpreted as a floating point number.

   2. Change the Kafka.spec.kafka.version to specify the new version (leaving the log.message.format.version as the current version).

      For example, if upgrading from Kafka 2.1.1 to 2.2.1:

      apiVersion: v1alpha1
      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.2.1 1
          config:
            log.message.format.version: "2.1" 2
            # ...

      1

      This is changed to the new version

      2

      This remains at the current version

   3. If the image for the Kafka version is different from the image defined in STRIMZI_KAFKA_IMAGES for the Cluster Operator, update Kafka.spec.kafka.image.

      See Section 9.5.1, “Kafka version and image mappings”

2. Save and exit the editor, then wait for rolling updates to complete.

   Check the update in the logs or by watching the pod state transitions:

   On OpenShift, use:

   oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version upgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

   oc get po -w

   If the current and new versions of Kafka have different interbroker protocol versions, check the Cluster Operator logs for an INFO level message:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 2 of 2 completed

   Alternatively, if the current and new versions of Kafka have the same interbroker protocol version, check for:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version upgrade from <from-version> to <to-version>, phase 1 of 1 completed

   The rolling updates:

   • Ensure each pod is using the broker binaries for the new version of Kafka

   • Configure the brokers to send messages using the interbroker protocol of the new version of Kafka

     Note

     Clients are still using the old version, so brokers will convert messages to the old version before sending them to the clients. To minimize this additional load, updates the clients as quickly as possible.

3. Depending on your chosen strategy for upgrading clients, upgrade all client applications to use the new version of the client binaries.

   See Section 9.5.2, “Strategies for upgrading clients”

   Warning

   You cannot downgrade after completing this step. If you need to revert the update at this point, follow the procedure Section 9.6.2, “Downgrading Kafka brokers and client applications”.

   If required, set the version property for Kafka Connect and Mirror Maker as the new version of Kafka:

   1. For Kafka Connect, update KafkaConnect.spec.version
   2. For MIrror Maker, update KafkaMirrorMaker.spec.version

4. If the log.message.format.version identified in step 1 is the same as the new version proceed to the next step.

   Otherwise change the log.message.format.version in Kafka.spec.kafka.config to the default version for the new version of Kafka now being used.

   For example, if upgrading to 2.2.1:

   apiVersion: v1alpha1
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.2.1
       config:
         log.message.format.version: "2.2"
         # ...

5. Wait for the Cluster Operator to update the cluster.

   The Kafka cluster and clients are now using the new Kafka version.

Procedure

1. Update the Kafka cluster configuration in an editor, as required:

   On OpenShift, use:

   oc edit kafka my-cluster

   1. Change the Kafka.spec.kafka.version to specify the previous version.

      For example, if downgrading from Kafka 2.2.1 to 2.1.1:

      apiVersion: v1alpha1
      kind: Kafka
      spec:
        # ...
        kafka:
          version: 2.1.1 1
          config:
            log.message.format.version: "2.1" 2
            # ...

      1

      This is changed to the previous version

      2

      This is unchanged

      Note

      You must format the value of log.message.format.version as a string to prevent it from being interpreted as a floating point number.

   2. If the image for the Kafka version is different from the image defined in STRIMZI_KAFKA_IMAGES for the Cluster Operator, update Kafka.spec.kafka.image.

      See Section 9.5.1, “Kafka version and image mappings”

2. Save and exit the editor, then wait for rolling updates to complete.

   Check the update in the logs or by watching the pod state transitions:

   On OpenShift use:

   oc logs -f <cluster-operator-pod-name> | grep -E "Kafka version downgrade from [0-9.]+ to [0-9.]+, phase ([0-9]+) of \1 completed"

   oc get po -w

   If the previous and current versions of Kafka have different interbroker protocol versions, check the Cluster Operator logs for an INFO level message:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 2 of 2 completed

   Alternatively, if the previous and current versions of Kafka have the same interbroker protocol version, check for:

   Reconciliation #<num>(watch) Kafka(<namespace>/<name>): Kafka version downgrade from <from-version> to <to-version>, phase 1 of 1 completed

3. Downgrade all client applications (consumers) to use the previous version of the client binaries.

   The Kafka cluster and clients are now using the previous Kafka version.