Streams for Apache Kafka 3.0: Kafka 4 impact and adoption

Streams for Apache Kafka 3.0 is scheduled for release in 2025. The adoption of Apache Kafka 4 introduces significant changes that will affect how you deploy, configure, and operate your Kafka clusters. This article highlights these changes, including the removal of ZooKeeper support, the switch to Log4j2 for logging, and updates to supported versions and dependencies.

Additionally, we cover the discontinuation of legacy features like MirrorMaker 1 and Java 11, and the forthcoming support for RHEL 10. While some of these changes are not directly tied to Kafka 4, they are part of the broader Streams for Apache Kafka 3.0 release. Upcoming documentation updates will reflect these changes. Understanding these adjustments will help you prepare for a smooth migration and take advantage of the latest enhancements.

NOTE: Kafka 4.0 development is ongoing, and additional changes may impact adoption. Stay informed by reviewing Streams for Apache Kafka and Kafka release notes as updates become available.

Removal of ZooKeeper

ZooKeeper support ended with Kafka 3.9 and will not be available in Kafka 4, as KRaft (Kafka Raft metadata) mode replaces ZooKeeper for cluster management. In KRaft mode, metadata management and cluster coordination are built directly into Kafka.

To align with this change, Streams for Apache Kafka 3.0 also removes support for ZooKeeper-based clusters:

• Streams for Apache Kafka 2.9 (LTS) is the last release to support ZooKeeper, as Kafka 3.9 is the final version that supports ZooKeeper clusters.

• Streams for Apache Kafka 3.0 requires KRaft-based clusters. Ensure your Kafka cluster is using KRaft before upgrading.

Streams releases showing support

NOTE: Supported Kafka versions include the previous version for upgrade only.

Migration from ZooKeeper-based Kafka

To migrate to using Kafka in KRaft mode before upgrading to Streams for Apache Kafka 3.0, see the following detailed instructions:

• OpenShift: Migrating to KRaft mode

• RHEL: Migrating to KRaft mode

NOTE: Migration is only supported on dedicated controller nodes, not on nodes with dual roles as brokers and controllers.

KRaft mode limitations

For Kafka 3.8 and earlier, the controller quorums (which replace Zookeeper) were of fixed size (static). Dynamic controller quorums were introduced in Kafka 3.9.

Migration between Kafka’s static and dynamic controller quorums is not currently supported, though this feature is expected in a future Kafka release.

This limitation means the following for Streams for Apache Kafka:

• Streams for Apache Kafka on OpenShift
  Existing KRaft-based clusters using static controller quorums must continue using them. To ensure compatibility with existing KRaft-based clusters, Streams for Apache Kafka on OpenShift continues to use static controller quorums as well.

• Streams for Apache Kafka on RHEL
  Streams for Apache Kafka 2.9 on RHEL supports static and dynamic controller quorums, with dynamic quorums recommended for new deployments.

Removal of MirrorMaker 1

Streams 2.9 is the last release to support MirrorMaker 1. From Streams for Apache Kafka 3.0, only MirrorMaker 2 is supported.

MirrorMaker 2 offers enhanced replication options, supporting both active/passive and active/active configurations. It uses the Kafka Connect framework, with connectors managing data transfer between clusters.

To replicate topics without renaming (similar to MirrorMaker 1), configure MirrorMaker 2 to use the IdentityReplicationPolicy.

Migration from MirrorMaker 1 to MirrorMaker 2 is manual, so plan to complete the migration before upgrading to Streams for Apache Kafka 3.0.

For more on using IdentityReplicationPolicy, see the following guides:

• OpenShift: Configuring Kafka MirrorMaker 2

• RHEL: Running MirrorMaker 2 in dedicated mode

Log4j2 logging for Kafka components

Kafka 4 transitions from Log4j1 to Log4j2 for logging. If you use custom logging configurations, review and update properties or file names as needed to ensure compatibility.

Java 11 no longer supported

From Streams for Apache Kafka 3.0, Java 11 is no longer supported for any components, including clients. Use Java 17 or 21 when developing new applications.

Dropped configuration options

The following configurations are no longer supported in Streams for Apache Kafka 3.0:

• Environment Variable Configuration Provider
  Replace io.strimzi.kafka.EnvVarConfigProvider with Kafka’s own provider: org.apache.kafka.common.config.provider.EnvVarConfigProvider.

• MirrorMaker Identity Replication Policy
  Replace io.strimzi.kafka.connect.mirror.IdentityReplicationPolicy with Kafka’s policy class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy.

• Storage class overrides (OpenShift version)
  Replace *.storage.overrides with KafkaNodePool resources. Use multiple node pools to assign different storage classes.

Updated configuration constraints and defaults

As part of Kafka 4.0, several configuration properties have updated constraints and default values, as described in KIP-1030.

Even if these properties are not explicitly configured in your deployment, the changes may impact performance or behavior. Review your current Kafka configuration before upgrading to Streams 3.0 and adjust any properties to align with the new constraints and defaults as necessary.

RHEL 10 adoption

RHEL 10, scheduled for release in early 2025, has been tested with Open JDK 17 and 21 for Streams for Apache Kafka 3.0.
Streams for Apache Kafka 3.0 will support RHEL 8, 9, and 10.

Key changes in RHEL 10 for Java:

• Java 21 (Open JDK 21) is the only version available through RPMs.

• Java 17 (Open JDK 17) is not available as an official Red Hat RPM. However, it can be installed using the upstream Temurin distribution. Note that this distribution is not maintained by Red Hat.

For optimal performance and security, consider upgrading to Open JDK 21 when moving to RHEL 10.

Client protocol support changes

With the release of Kafka 4.0, support for old client protocol API versions is being removed as part of KIP-896.

The minimum client version compatible with Kafka 4.0 is Apache Kafka 2.1. If you are using Kafka clients between versions 0.8 and 2.0, you must upgrade to at least Apache Kafka 2.1 before migrating to Streams 3.0. This change also applies to unofficial Kafka clients implemented in other languages, which may require updates to align with Kafka 4.0 protocol requirements.

Minimum supported client version

Currently, Streams for Apache Kafka only provides a Java client library. To ensure compatibility with Streams for Apache Kafka 3.0 and Kafka 4.0, client version support is as follows:

• The latest tested and supported version of the Kafka client aligns with kafka-clients-4.0.x.redhat-xxxx.

• The minimum tested and supported version aligns with kafka-clients-3.9.x.redhat-xxxx.

Older clients are not tested or supported in Streams for Apache Kafka 3.0, so we recommend using a tested and supported Kafka client version when upgrading to Streams for Apache Kafka 3.0.

Removal of message formats v0 and v1

With Kafka 4.0, support for message formats v0 and v1 is expected to be removed as part of KIP-724. These formats were used primarily in ZooKeeper-based Kafka clusters and are no longer relevant with the transition to KRaft mode.

Ensure all clients interacting with your Kafka cluster support message format v2 or later. With the probable removal of v0 and v1 message formats in Kafka 4.0, up-conversion for produce requests and down-conversion for fetch requests will no longer be supported. Clients using v0 or v1 message formats will be incompatible with Kafka 4.0 and Streams for Apache Kafka 3.0.

The message.format.version and log.message.format.version properties, previously used to configure compatibility with message formats in ZooKeeper-based clusters, are also being removed in Kafka 4.0. These properties are ignored in KRaft mode, so no additional configuration is required.