Release Notes for AMQ Streams 1.7 on RHEL

Red Hat AMQ 2021.q2

For use with AMQ Streams on Red Hat Enterprise Linux

Abstract

These release notes contain the latest information about new features, enhancements, fixes, and issues contained in the AMQ Streams 1.7 release.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.

Chapter 1. Features

The features added in this release, and that were not in previous releases of AMQ Streams, are outlined below.

Note

To view all the enhancements and bugs that are resolved in this release, see the AMQ Streams Jira project.

1.1. Kafka 2.7.0 support

AMQ Streams now supports Apache Kafka version 2.7.0.

AMQ Streams uses Kafka 2.7.0. Only Kafka distributions built by Red Hat are supported.

For upgrade instructions, see AMQ Streams and Kafka upgrades.

Refer to the Kafka 2.6.0 and Kafka 2.7.0 Release Notes for additional information.

Note

Kafka 2.6.x is supported only for the purpose of upgrading to AMQ Streams 1.7.

For more information on supported versions, see the Red Hat Knowledgebase article Red Hat AMQ 7 Component Details Page.

Kafka 2.7.0 requires the same ZooKeeper version as Kafka 2.6.x (ZooKeeper version 3.5.8). Therefore, you do not need to upgrade ZooKeeper when upgrading from AMQ Streams 1.6 to AMQ Streams 1.7.

Chapter 2. Enhancements

The enhancements added in this release are outlined below.

2.1. Kafka 2.7.0 enhancements

For an overview of the enhancements introduced with Kafka 2.7.0, refer to the Kafka 2.7.0 Release Notes.

2.2. OAuth 2.0 authentication and authorization

This release includes the following enhancements to OAuth 2.0 token-based authentication and authorization.

Checks on JWT access tokens

You can now configure two additional checks on JWT access tokens. Both of these checks are configured in the OAuth 2.0 authentication listener configuration.

Custom claim checks

Custom claim checks impose custom rules on the validation of JWT access tokens by Kafka brokers. They are defined using JsonPath filter queries.

If an access token does not contain the necessary data, it is rejected. When using introspection endpoint token validation, the custom check is applied to the introspection endpoint response JSON.

To configure custom claim checks, add the oauth.custom.claim.check option to the server.properties file and define a JsonPath filter query. Custom claim checks are disabled by default.

See Configuring OAuth 2.0 support for Kafka brokers

Audience checks

Your authorization server might provide aud (audience) claims in JWT access tokens.

When audience checks are enabled, the Kafka broker rejects tokens that do not contain the broker’s clientId in their aud claims.

To enable audience checks, set the oauth.check.audience option to true. Audience checks are disabled by default.

See Configuring OAuth 2.0 support for Kafka brokers

Support for OAuth 2.0 over SASL PLAIN authentication

You can now configure the PLAIN mechanism for OAuth 2.0 authentication between Kafka clients and Kafka brokers. Previously, the only supported authentication mechanism was OAUTHBEARER.

PLAIN is a simple authentication mechanism supported by all Kafka client tools (including developer tools such as kafkacat). AMQ Streams includes server-side callbacks that enable PLAIN to be used with OAuth 2.0 authentication. These capabilities are referred to as OAuth 2.0 over PLAIN.

Note

Red Hat recommends using OAUTHBEARER authentication for clients whenever possible. OAUTHBEARER provides a higher level of security than PLAIN because client credentials are never shared with Kafka brokers. Consider using PLAIN only with Kafka clients that do not support OAUTHBEARER.

When used with the provided OAuth 2.0 over PLAIN callbacks, Kafka clients can authenticate with Kafka brokers using either of the following methods:

  • Client ID and secret (by using the OAuth 2.0 client credentials mechanism)
  • A long-lived access token, obtained manually at configuration time

To use PLAIN, you must enable it in the server.properties file, in the OAuth authentication listener configuration.

See OAuth 2.0 authentication mechanisms and Configuring OAuth 2.0 support for Kafka brokers

Chapter 3. Technology Previews

Important

Technology Preview features are not supported with Red Hat production service-level agreements (SLAs) and might not be functionally complete; therefore, Red Hat does not recommend implementing any Technology Preview features in production environments. This Technology Preview feature provides early access to upcoming product innovations, enabling you to test functionality and provide feedback during the development process. For more information about support scope, see Technology Preview Features Support Scope.

3.1. Cruise Control for cluster rebalancing

Note

Cruise Control remains in Technology Preview, with some new enhancements.

You can install Cruise Control and use it to rebalance your Kafka cluster using optimization goals — defined constraints on CPU, disk, network load, and more. In a balanced Kafka cluster, the workload is more evenly distributed across the broker pods.

Cruise Control helps to reduce the time and effort involved in running an efficient and balanced Kafka cluster.

A zipped distribution of Cruise Control is available for download from the Customer Portal. To install Cruise Control, you configure each Kafka broker to use the provided Metrics Reporter. Then, you set Cruise Control properties, including optimization goals, and start Cruise Control using the provided script.

The Cruise Control server is hosted on a single machine for the whole Kafka cluster.

When Cruise Control is running, you can use the REST API to:

  • Generate dry run optimization proposals from multiple optimization goals
  • Initiate an optimization proposal to rebalance the Kafka cluster

Other Cruise Control features are not currently supported, including anomaly detection, notifications, write-your-own goals, and changing the topic replication factor.

See Cruise Control for cluster rebalancing.

3.1.1. Enhancements to the Technology Preview

The following enhancements have been added to the Technology Preview of Cruise Control for cluster rebalancing.

New goal: Minimum topic leaders per broker

You can use a new goal named MinTopicLeadersPerBrokerGoal.

For each topic in a defined group of topics, the goal ensures that each active broker has at least a certain number of leader replicas.

MinTopicLeadersPerBrokerGoal is a default goal and is preset as a hard goal.

See Optimization goals overview

Logging enhancement: Log4j 2

Log4j 2 is now used for Cruise Control logging.

You must update existing configurations for Cruise Control logging (in /opt/cruise-control/config/log4j.properties) from Log4j to Log4j 2 compatible syntax.

See Cruise Control configuration

Chapter 4. Deprecated features

There are no deprecated features for AMQ Streams 1.7.

Chapter 5. Fixed issues

The issues fixed in AMQ Streams 1.7 on RHEL are shown in the following table. For details of the issues fixed in Kafka 2.7.0, refer to the Kafka 2.7.0 Release Notes.

Issue NumberDescription

ENTMQST-2030

If the bin/kafka-acls.sh utility is used to add or remove an ACL, the operation is successful but a warning is generated

ENTMQST-2188

MirrorMaker: Enable synchronization of offsets to the consumer group on the target cluster

ENTMQST-2269

kafka-configs.sh is deprecating --zookeeper option but does not provide alternative functionality to list configs for users

Chapter 6. Known issues

There are no known issues for AMQ Streams 1.7 on RHEL.

Chapter 7. Supported integration products

AMQ Streams 1.7 supports integration with the following Red Hat products.

  • Red Hat Single Sign-On 7.4 and later  for OAuth 2.0 authentication and OAuth 2.0 authorization
  • Red Hat Debezium 1.4 and later  for monitoring databases and creating event streams

For information on the functionality these products can introduce to your AMQ Streams deployment, refer to the AMQ Streams 1.7 documentation.

Legal Notice

Copyright © 2021 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.