Chapter 7. Upgrading AMQ Streams

Procedure

1. Take note of any configuration changes made to the existing Cluster Operator resources (in the /install/cluster-operator directory). Any changes will be overwritten by the new version of the Cluster Operator.

2. Update the Cluster Operator.

   1. Modify the installation files for the new version according to the namespace the Cluster Operator is running in.

      On Linux, use:

      sed -i 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

      On MacOS, use:

      sed -i '' 's/namespace: .*/namespace: my-cluster-operator-namespace/' install/cluster-operator/*RoleBinding*.yaml

   2. If you modified one or more environment variables in your existing Cluster Operator Deployment, edit the install/cluster-operator/060-Deployment-cluster-operator.yaml file to use those environment variables.

3. When you have an updated configuration, deploy it along with the rest of the installation resources:

   oc apply -f install/cluster-operator

   Wait for the rolling updates to complete.

4. Get the image for the Kafka pod to ensure the upgrade was successful:

   oc get po my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   The image tag shows the new AMQ Streams version followed by the Kafka version. For example, <New AMQ Streams version>-kafka-<Current Kafka version>.

5. Update existing resources to handle deprecated custom resource properties.

   • AMQ Streams resource upgrades

Procedure

1. Update the Kafka cluster configuration:

   oc edit kafka my-cluster

2. If configured, ensure that Kafka.spec.kafka.config has the log.message.format.version and inter.broker.protocol.version set to the defaults for the current Kafka version.

   For example, if upgrading from Kafka version 2.5.0 to 2.6.0:

   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.5.0
       config:
         log.message.format.version: "2.5"
         inter.broker.protocol.version: "2.5"
         # ...

   If log.message.format.version and inter.broker.protocol.version are not configured, AMQ Streams automatically updates these versions to the current defaults after the update to the Kafka version in the next step.

   Note

   The value of log.message.format.version and inter.broker.protocol.version must be strings to prevent them from being interpreted as floating point numbers.

3. Change the Kafka.spec.kafka.version to specify the new Kafka version; leave the log.message.format.version and inter.broker.protocol.version at the defaults for the current Kafka version.

   Note

   Changing the kafka.version ensures that all brokers in the cluster will be upgraded to start using the new broker binaries. During this process, some brokers are using the old binaries while others have already upgraded to the new ones. Leaving the inter.broker.protocol.version unchanged ensures that the brokers can continue to communicate with each other throughout the upgrade.

   For example, if upgrading from Kafka 2.5.0 to 2.6.0:

   apiVersion: kafka.strimzi.io/v1beta1
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.6.0 1
       config:
         log.message.format.version: "2.5" 2
         inter.broker.protocol.version: "2.5" 3
         # ...

   1

   Kafka version is changed to the new version.

   2

   Message format version is unchanged.

   3

   Inter-broker protocol version is unchanged.

   Warning

   You cannot downgrade Kafka if the inter.broker.protocol.version for the new Kafka version changes. The inter-broker protocol version determines the schemas used for persistent metadata stored by the broker, including messages written to __consumer_offsets. The downgraded cluster will not understand the messages.

4. If the image for the Kafka cluster is defined in the Kafka custom resource, in Kafka.spec.kafka.image, update the image to point to a container image with the new Kafka version.

   See Kafka version and image mappings

5. Save and exit the editor, then wait for rolling updates to complete.

   Check the progress of the rolling updates by watching the pod state transitions:

   oc get pods my-cluster-kafka-0 -o jsonpath='{.spec.containers[0].image}'

   The rolling updates ensure that each pod is using the broker binaries for the new version of Kafka.

6. Depending on your chosen strategy for upgrading clients, upgrade all client applications to use the new version of the client binaries.

   If required, set the version property for Kafka Connect and MirrorMaker as the new version of Kafka:

   1. For Kafka Connect, update KafkaConnect.spec.version.
   2. For MirrorMaker, update KafkaMirrorMaker.spec.version.
   3. For MirrorMaker 2.0, update KafkaMirrorMaker2.spec.version.

7. If configured, update the Kafka resource to use the new inter.broker.protocol.version version. Otherwise, go to step 9.

   For example, if upgrading to Kafka 2.6.0:

   apiVersion: kafka.strimzi.io/v1beta1
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.6.0
       config:
         log.message.format.version: "2.5"
         inter.broker.protocol.version: "2.6"
         # ...

8. Wait for the Cluster Operator to update the cluster.

9. If configured, update the Kafka resource to use the new log.message.format.version version. Otherwise, go to step 10.

   For example, if upgrading to Kafka 2.6.0:

   apiVersion: kafka.strimzi.io/v1beta1
   kind: Kafka
   spec:
     # ...
     kafka:
       version: 2.6.0
       config:
         log.message.format.version: "2.6"
         inter.broker.protocol.version: "2.6"
         # ...

10. Wait for the Cluster Operator to update the cluster.

    • The Kafka cluster and clients are now using the new Kafka version.
    • The brokers are configured to send messages using the inter-broker protocol version and message format version of the new version of Kafka.

1. Replace the Kafka clients .jar file with the new version.
2. In the consumer configuration, append cooperative-sticky to the partition.assignment.strategy. For example, if the range strategy is set, change the configuration to range, cooperative-sticky.
3. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.
4. Reconfigure each consumer in the group by removing the earlier partition.assignment.strategy from the consumer configuration, leaving only the cooperative-sticky strategy.
5. Restart each consumer in the group in turn, waiting for the consumer to rejoin the group after each restart.

1. Replace the Kafka Streams .jar file with the new version.
2. In the Kafka Streams configuration, set the upgrade.from configuration parameter to the Kafka version you are upgrading from (for example, 2.3).
3. Restart each of the stream processors (nodes) in turn.
4. Remove the upgrade.from configuration parameter from the Kafka Streams configuration.
5. Restart each consumer in the group in turn.