Chapter 4. Technology Previews

Important

Technology Preview features are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete; therefore, Red Hat does not recommend implementing any Technology Preview features in production environments. This Technology Preview feature provides early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. For more information about support scope, see Technology Preview Features Support Scope.

4.1. Kafka Static Quota plugin configuration

Use the Kafka Static Quota plugin to set throughput and storage limits on brokers in your Kafka cluster. You enable the plugin and set limits by configuring the Kafka resource. You can set a byte-rate threshold and storage quotas to put limits on the clients interacting with your brokers.

Example Kafka Static Quota plugin configuration

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
    config:
      client.quota.callback.class: io.strimzi.kafka.quotas.StaticQuotaCallback
      client.quota.callback.static.produce: 1000000
      client.quota.callback.static.fetch: 1000000
      client.quota.callback.static.storage.soft: 400000000000
      client.quota.callback.static.storage.hard: 500000000000
      client.quota.callback.static.storage.check-interval: 5

See Setting limits on brokers using the Kafka Static Quota plugin

4.2. Cruise Control for cluster rebalancing

Note

Cruise Control remains in Technology Preview, with some new enhancements.

You can deploy Cruise Control and use it to rebalance your Kafka cluster using optimization goals — defined constraints on CPU, disk, network load, and more. In a balanced Kafka cluster, the workload is more evenly distributed across the broker pods.

Cruise Control is configured and deployed as part of a Kafka resource. You can use the default optimization goals or modify them to suit your requirements. Example YAML configuration files for Cruise Control are provided in examples/cruise-control/.

When Cruise Control is deployed, you can create KafkaRebalance custom resources to:

  • Generate optimization proposals from multiple optimization goals
  • Rebalance a Kafka cluster based on an optimization proposal

Other Cruise Control features are not currently supported, including anomaly detection, notifications, write-your-own goals, and changing the topic replication factor.

See Cruise Control for cluster rebalancing

4.2.1. Enhancements to the Technology Preview

Refresh optimization proposals

You can now reuse existing KafkaRebalance resources with a status of Ready, which indicates that cluster rebalancing completed successfully. You can reuse the optimization goals defined in the KafkaRebalance resource, or change the goals.

To refresh an optimization proposal:

  1. Check the status of the KafkaRebalance resource: 

    oc describe kafkarebalance REBALANCE-NAME

  2. Apply the strimzi.io/rebalance=refresh annotation: 

    oc annotate kafkarebalance REBALANCE-NAME strimzi.io/rebalance=refresh

Cruise Control refreshes the optimization proposal to reflect the latest state of your Kafka cluster.

See Approving an optimization proposal

View the broker load in optimization proposals

Optimization proposals now consist of the broker load in addition to the summary status. The broker load is returned inside a ConfigMap and shows metrics for the load on each Kafka broker, including the CPU utilization, disk usage, network output rate, and more. The metrics are presented in three categories:

before
Current value before the optimization proposal is applied
after
Expected value after the optimization proposal is applied
difference
Difference between the after value and the before value

The broker load ConfigMap has the same name as the KafkaRebalance resource. The metrics are encoded as a JSON string. To view them in a human readable format, use jq or a similiar JSON parser. For example: 

oc get configmap MY-REBALANCE -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.'

See Optimization proposals overview