Jump To Close Expand all Collapse all Table of contents Using AMQ Streams on RHEL Making open source more inclusive 1. Overview of AMQ Streams Expand section "1. Overview of AMQ Streams" Collapse section "1. Overview of AMQ Streams" 1.1. Kafka capabilities 1.2. Kafka use cases 1.3. Supported Configurations 1.4. Document conventions 2. Getting started Expand section "2. Getting started" Collapse section "2. Getting started" 2.1. AMQ Streams distribution 2.2. Downloading an AMQ Streams Archive 2.3. Installing AMQ Streams 2.4. Data storage considerations Expand section "2.4. Data storage considerations" Collapse section "2.4. Data storage considerations" 2.4.1. Apache Kafka and ZooKeeper storage support 2.4.2. File systems 2.5. Running a single node AMQ Streams cluster 2.6. Using the cluster 2.7. Stopping the AMQ Streams services 2.8. Configuring AMQ Streams 3. Configuring ZooKeeper Expand section "3. Configuring ZooKeeper" Collapse section "3. Configuring ZooKeeper" 3.1. Basic configuration 3.2. ZooKeeper cluster configuration 3.3. Running multi-node ZooKeeper cluster 3.4. Authentication Expand section "3.4. Authentication" Collapse section "3.4. Authentication" 3.4.1. Authentication with SASL 3.4.2. Enabling Server-to-server authentication using DIGEST-MD5 3.4.3. Enabling Client-to-server authentication using DIGEST-MD5 3.5. Authorization 3.6. TLS 3.7. Additional configuration options 3.8. Logging 4. Configuring Kafka Expand section "4. Configuring Kafka" Collapse section "4. Configuring Kafka" 4.1. ZooKeeper 4.2. Listeners 4.3. Commit logs 4.4. Broker ID 4.5. Running a multi-node Kafka cluster 4.6. ZooKeeper authentication Expand section "4.6. ZooKeeper authentication" Collapse section "4.6. ZooKeeper authentication" 4.6.1. JAAS Configuration 4.6.2. Enabling ZooKeeper authentication 4.7. Authorization Expand section "4.7. Authorization" Collapse section "4.7. Authorization" 4.7.1. Simple ACL authorizer Expand section "4.7.1. Simple ACL authorizer" Collapse section "4.7.1. Simple ACL authorizer" 4.7.1.1. ACL rules 4.7.1.2. Principals 4.7.1.3. Authentication of users 4.7.1.4. Super users 4.7.1.5. Replica broker authentication 4.7.1.6. Supported resources 4.7.1.7. Supported operations 4.7.1.8. ACL management options 4.7.2. Enabling authorization 4.7.3. Adding ACL rules 4.7.4. Listing ACL rules 4.7.5. Removing ACL rules 4.8. ZooKeeper authorization Expand section "4.8. ZooKeeper authorization" Collapse section "4.8. ZooKeeper authorization" 4.8.1. ACL Configuration 4.8.2. Enabling ZooKeeper ACLs for a new Kafka cluster 4.8.3. Enabling ZooKeeper ACLs in an existing Kafka cluster 4.9. Encryption and authentication Expand section "4.9. Encryption and authentication" Collapse section "4.9. Encryption and authentication" 4.9.1. Listener configuration 4.9.2. TLS Encryption 4.9.3. Enabling TLS encryption 4.9.4. Authentication Expand section "4.9.4. Authentication" Collapse section "4.9.4. Authentication" 4.9.4.1. TLS client authentication 4.9.4.2. SASL authentication 4.9.5. Enabling TLS client authentication 4.9.6. Enabling SASL PLAIN authentication 4.9.7. Enabling SASL SCRAM authentication 4.9.8. Adding SASL SCRAM users 4.9.9. Deleting SASL SCRAM users 4.10. Using OAuth 2.0 token-based authentication Expand section "4.10. Using OAuth 2.0 token-based authentication" Collapse section "4.10. Using OAuth 2.0 token-based authentication" 4.10.1. OAuth 2.0 authentication mechanisms Expand section "4.10.1. OAuth 2.0 authentication mechanisms" Collapse section "4.10.1. OAuth 2.0 authentication mechanisms" 4.10.1.1. Configuring OAuth 2.0 with properties or variables 4.10.2. OAuth 2.0 Kafka broker configuration Expand section "4.10.2. OAuth 2.0 Kafka broker configuration" Collapse section "4.10.2. OAuth 2.0 Kafka broker configuration" 4.10.2.1. OAuth 2.0 client configuration on an authorization server 4.10.2.2. OAuth 2.0 authentication configuration in the Kafka cluster 4.10.2.3. Fast local JWT token validation configuration 4.10.2.4. OAuth 2.0 introspection endpoint configuration 4.10.3. Session re-authentication for Kafka brokers 4.10.4. OAuth 2.0 Kafka client configuration 4.10.5. OAuth 2.0 client authentication flow Expand section "4.10.5. OAuth 2.0 client authentication flow" Collapse section "4.10.5. OAuth 2.0 client authentication flow" 4.10.5.1. Example client authentication flows 4.10.6. Configuring OAuth 2.0 authentication Expand section "4.10.6. Configuring OAuth 2.0 authentication" Collapse section "4.10.6. Configuring OAuth 2.0 authentication" 4.10.6.1. Configuring Red Hat Single Sign-On as an OAuth 2.0 authorization server 4.10.6.2. Configuring OAuth 2.0 support for Kafka brokers 4.10.6.3. Configuring Kafka Java clients to use OAuth 2.0 4.11. Using OAuth 2.0 token-based authorization Expand section "4.11. Using OAuth 2.0 token-based authorization" Collapse section "4.11. Using OAuth 2.0 token-based authorization" 4.11.1. OAuth 2.0 authorization mechanism Expand section "4.11.1. OAuth 2.0 authorization mechanism" Collapse section "4.11.1. OAuth 2.0 authorization mechanism" 4.11.1.1. Kafka broker custom authorizer 4.11.2. Configuring OAuth 2.0 authorization support 4.12. Using OPA policy-based authorization Expand section "4.12. Using OPA policy-based authorization" Collapse section "4.12. Using OPA policy-based authorization" 4.12.1. Defining OPA policies 4.12.2. Connecting to the OPA 4.12.3. Configuring OPA authorization support 4.13. Logging Expand section "4.13. Logging" Collapse section "4.13. Logging" 4.13.1. Dynamically change logging levels for Kafka broker loggers 5. Topics Expand section "5. Topics" Collapse section "5. Topics" 5.1. Partitions and replicas 5.2. Message retention 5.3. Topic auto-creation 5.4. Topic deletion 5.5. Topic configuration 5.6. Internal topics 5.7. Creating a topic 5.8. Listing and describing topics 5.9. Modifying a topic configuration 5.10. Deleting a topic 6. Managing Kafka Expand section "6. Managing Kafka" Collapse section "6. Managing Kafka" 6.1. Tuning Kafka configuration Expand section "6.1. Tuning Kafka configuration" Collapse section "6.1. Tuning Kafka configuration" 6.1.1. Kafka broker configuration tuning Expand section "6.1.1. Kafka broker configuration tuning" Collapse section "6.1.1. Kafka broker configuration tuning" 6.1.1.1. Basic broker configuration 6.1.1.2. Replicating topics for high availability 6.1.1.3. Internal topic settings for transactions and commits 6.1.1.4. Improving request handling throughput by increasing I/O threads 6.1.1.5. Increasing bandwidth for high latency connections 6.1.1.6. Managing logs with data retention policies 6.1.1.7. Removing log data with cleanup policies 6.1.1.8. Managing disk utilization 6.1.1.9. Handling large message sizes 6.1.1.10. Controlling the log flush of message data 6.1.1.11. Partition rebalancing for availability 6.1.1.12. Unclean leader election 6.1.1.13. Avoiding unnecessary consumer group rebalances 6.1.2. Kafka producer configuration tuning Expand section "6.1.2. Kafka producer configuration tuning" Collapse section "6.1.2. Kafka producer configuration tuning" 6.1.2.1. Basic producer configuration 6.1.2.2. Data durability 6.1.2.3. Ordered delivery 6.1.2.4. Reliability guarantees 6.1.2.5. Optimizing throughput and latency 6.1.3. Kafka consumer configuration tuning Expand section "6.1.3. Kafka consumer configuration tuning" Collapse section "6.1.3. Kafka consumer configuration tuning" 6.1.3.1. Basic consumer configuration 6.1.3.2. Scaling data consumption using consumer groups 6.1.3.3. Message ordering guarantees 6.1.3.4. Optimizing throughput and latency 6.1.3.5. Avoiding data loss or duplication when committing offsets Expand section "6.1.3.5. Avoiding data loss or duplication when committing offsets" Collapse section "6.1.3.5. Avoiding data loss or duplication when committing offsets" 6.1.3.5.1. Controlling transactional messages 6.1.3.6. Recovering from failure to avoid data loss 6.1.3.7. Managing offset policy 6.1.3.8. Minimizing the impact of rebalances 6.2. Setting limits on brokers using the Kafka Static Quota plugin 6.3. Scaling Clusters Expand section "6.3. Scaling Clusters" Collapse section "6.3. Scaling Clusters" 6.3.1. Scaling Kafka clusters Expand section "6.3.1. Scaling Kafka clusters" Collapse section "6.3.1. Scaling Kafka clusters" 6.3.1.1. Adding brokers to a cluster 6.3.1.2. Removing brokers from the cluster 6.3.2. Reassignment of partitions Expand section "6.3.2. Reassignment of partitions" Collapse section "6.3.2. Reassignment of partitions" 6.3.2.1. Reassignment JSON file 6.3.2.2. Generating reassignment JSON files 6.3.2.3. Creating reassignment JSON files manually 6.3.3. Reassignment throttles 6.3.4. Scaling up a Kafka cluster 6.3.5. Scaling down a Kafka cluster 6.3.6. Scaling up a ZooKeeper cluster 6.3.7. Scaling down a ZooKeeper cluster 7. Monitoring your cluster using JMX Expand section "7. Monitoring your cluster using JMX" Collapse section "7. Monitoring your cluster using JMX" 7.1. JMX configuration options 7.2. Disabling the JMX agent 7.3. Connecting to the JVM from a different machine 7.4. Monitoring using JConsole 7.5. Important Kafka broker metrics Expand section "7.5. Important Kafka broker metrics" Collapse section "7.5. Important Kafka broker metrics" 7.5.1. Kafka server metrics 7.5.2. Kafka network metrics 7.5.3. Kafka log metrics 7.5.4. Kafka controller metrics 7.5.5. Yammer metrics 7.6. Producer MBeans Expand section "7.6. Producer MBeans" Collapse section "7.6. Producer MBeans" 7.6.1. MBeans matching kafka.producer:type=producer-metrics,client-id=* 7.6.2. MBeans matching kafka.producer:type=producer-metrics,client-id=*,node-id=* 7.6.3. MBeans matching kafka.producer:type=producer-topic-metrics,client-id=*,topic=* 7.7. Consumer MBeans Expand section "7.7. Consumer MBeans" Collapse section "7.7. Consumer MBeans" 7.7.1. MBeans matching kafka.consumer:type=consumer-metrics,client-id=* 7.7.2. MBeans matching kafka.consumer:type=consumer-metrics,client-id=*,node-id=* 7.7.3. MBeans matching kafka.consumer:type=consumer-coordinator-metrics,client-id=* 7.7.4. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* 7.7.5. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=* 7.7.6. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=* 7.8. Kafka Connect MBeans Expand section "7.8. Kafka Connect MBeans" Collapse section "7.8. Kafka Connect MBeans" 7.8.1. MBeans matching kafka.connect:type=connect-metrics,client-id=* 7.8.2. MBeans matching kafka.connect:type=connect-metrics,client-id=*,node-id=* 7.8.3. MBeans matching kafka.connect:type=connect-worker-metrics 7.8.4. MBeans matching kafka.connect:type=connect-worker-rebalance-metrics 7.8.5. MBeans matching kafka.connect:type=connector-metrics,connector=* 7.8.6. MBeans matching kafka.connect:type=connector-task-metrics,connector=*,task=* 7.8.7. MBeans matching kafka.connect:type=sink-task-metrics,connector=*,task=* 7.8.8. MBeans matching kafka.connect:type=source-task-metrics,connector=*,task=* 7.8.9. MBeans matching kafka.connect:type=task-error-metrics,connector=*,task=* 7.9. Kafka Streams MBeans Expand section "7.9. Kafka Streams MBeans" Collapse section "7.9. Kafka Streams MBeans" 7.9.1. MBeans matching kafka.streams:type=stream-metrics,client-id=* 7.9.2. MBeans matching kafka.streams:type=stream-task-metrics,client-id=*,task-id=* 7.9.3. MBeans matching kafka.streams:type=stream-processor-node-metrics,client-id=*,task-id=*,processor-node-id=* 7.9.4. MBeans matching kafka.streams:type=stream-[store-scope]-metrics,client-id=*,task-id=*,[store-scope]-id=* 7.9.5. MBeans matching kafka.streams:type=stream-record-cache-metrics,client-id=*,task-id=*,record-cache-id=* 8. Kafka Connect Expand section "8. Kafka Connect" Collapse section "8. Kafka Connect" 8.1. Kafka Connect in standalone mode Expand section "8.1. Kafka Connect in standalone mode" Collapse section "8.1. Kafka Connect in standalone mode" 8.1.1. Configuring Kafka Connect in standalone mode 8.1.2. Configuring connectors in Kafka Connect in standalone mode 8.1.3. Running Kafka Connect in standalone mode 8.2. Kafka Connect in distributed mode Expand section "8.2. Kafka Connect in distributed mode" Collapse section "8.2. Kafka Connect in distributed mode" 8.2.1. Configuring Kafka Connect in distributed mode 8.2.2. Configuring connectors in distributed Kafka Connect 8.2.3. Running distributed Kafka Connect 8.2.4. Creating connectors 8.2.5. Deleting connectors 8.3. Connector plug-ins 8.4. Adding connector plugins 9. Using AMQ Streams with MirrorMaker 2.0 Expand section "9. Using AMQ Streams with MirrorMaker 2.0" Collapse section "9. Using AMQ Streams with MirrorMaker 2.0" 9.1. MirrorMaker 2.0 data replication 9.2. Cluster configuration Expand section "9.2. Cluster configuration" Collapse section "9.2. Cluster configuration" 9.2.1. Bidirectional replication (active/active) 9.2.2. Unidirectional replication (active/passive) 9.2.3. Topic configuration synchronization 9.2.4. Data integrity 9.2.5. Offset tracking 9.2.6. Synchronizing consumer group offsets 9.2.7. Connectivity checks 9.3. ACL rules synchronization 9.4. Synchronizing data between Kafka clusters using MirrorMaker 2.0 9.5. Using MirrorMaker 2.0 in legacy mode 10. Kafka clients Expand section "10. Kafka clients" Collapse section "10. Kafka clients" 10.1. Adding Kafka clients as a dependency to your Maven project 11. Kafka Streams API overview Expand section "11. Kafka Streams API overview" Collapse section "11. Kafka Streams API overview" 11.1. Adding the Kafka Streams API as a dependency to your Maven project 12. Kafka Bridge Expand section "12. Kafka Bridge" Collapse section "12. Kafka Bridge" 12.1. Kafka Bridge overview Expand section "12.1. Kafka Bridge overview" Collapse section "12.1. Kafka Bridge overview" 12.1.1. Authentication and encryption 12.1.2. Requests to the Kafka Bridge Expand section "12.1.2. Requests to the Kafka Bridge" Collapse section "12.1.2. Requests to the Kafka Bridge" 12.1.2.1. Content Type headers 12.1.2.2. Embedded data format 12.1.2.3. Message format 12.1.2.4. Accept headers 12.1.3. Configuring loggers for the Kafka Bridge 12.1.4. Kafka Bridge API resources 12.1.5. Downloading a Kafka Bridge archive 12.1.6. Configuring Kafka Bridge properties 12.1.7. Installing the Kafka Bridge 12.2. Kafka Bridge quickstart Expand section "12.2. Kafka Bridge quickstart" Collapse section "12.2. Kafka Bridge quickstart" 12.2.1. Deploying the Kafka Bridge locally 12.2.2. Producing messages to topics and partitions 12.2.3. Creating a Kafka Bridge consumer 12.2.4. Subscribing a Kafka Bridge consumer to topics 12.2.5. Retrieving the latest messages from a Kafka Bridge consumer 12.2.6. Commiting offsets to the log 12.2.7. Seeking to offsets for a partition 12.2.8. Deleting a Kafka Bridge consumer 13. Using Kerberos (GSSAPI) authentication Expand section "13. Using Kerberos (GSSAPI) authentication" Collapse section "13. Using Kerberos (GSSAPI) authentication" 13.1. Setting up AMQ Streams to use Kerberos (GSSAPI) authentication 14. Cruise Control for cluster rebalancing Expand section "14. Cruise Control for cluster rebalancing" Collapse section "14. Cruise Control for cluster rebalancing" 14.1. Why use Cruise Control? 14.2. Downloading a Cruise Control archive 14.3. Deploying the Cruise Control Metrics Reporter 14.4. Configuring and starting Cruise Control 14.5. Optimization goals overview 14.6. Optimization proposals overview 14.7. Rebalance performance tuning overview 14.8. Cruise Control configuration 14.9. Generating optimization proposals 14.10. Initiating a cluster rebalance 14.11. Stopping an active cluster rebalance 15. Distributed tracing Expand section "15. Distributed tracing" Collapse section "15. Distributed tracing" 15.1. Overview of OpenTracing and Jaeger 15.2. Setting up tracing for Kafka clients Expand section "15.2. Setting up tracing for Kafka clients" Collapse section "15.2. Setting up tracing for Kafka clients" 15.2.1. Initializing a Jaeger tracer for Kafka clients 15.2.2. Instrumenting producers and consumers for tracing 15.2.3. Instrumenting Kafka Streams applications for tracing 15.3. Setting up tracing for MirrorMaker and Kafka Connect Expand section "15.3. Setting up tracing for MirrorMaker and Kafka Connect" Collapse section "15.3. Setting up tracing for MirrorMaker and Kafka Connect" 15.3.1. Enabling tracing for MirrorMaker 15.3.2. Enabling tracing for MirrorMaker 2.0 15.3.3. Enabling tracing for Kafka Connect 15.4. Enabling tracing for the Kafka Bridge 15.5. Environment variables for tracing 16. Kafka Exporter Expand section "16. Kafka Exporter" Collapse section "16. Kafka Exporter" 16.1. Consumer lag 16.2. Kafka Exporter alerting rule examples 16.3. Kafka Exporter metrics 16.4. Running Kafka Exporter 16.5. Presenting Kafka Exporter metrics in Grafana 17. AMQ Streams and Kafka upgrades Expand section "17. AMQ Streams and Kafka upgrades" Collapse section "17. AMQ Streams and Kafka upgrades" 17.1. Upgrade prerequisites 17.2. Upgrade process 17.3. Kafka versions 17.4. Upgrading to AMQ Streams 1.8 Expand section "17.4. Upgrading to AMQ Streams 1.8" Collapse section "17.4. Upgrading to AMQ Streams 1.8" 17.4.1. Upgrading Kafka brokers and ZooKeeper 17.4.2. Upgrading Kafka Connect 17.5. Upgrading Kafka Expand section "17.5. Upgrading Kafka" Collapse section "17.5. Upgrading Kafka" 17.5.1. Upgrading Kafka brokers to use the new inter-broker protocol version 17.5.2. Strategies for upgrading clients 17.5.3. Upgrading client applications to the new Kafka version 17.5.4. Upgrading Kafka brokers to use the new message format version 17.5.5. Upgrading consumers and Kafka Streams applications to cooperative rebalancing A. Broker configuration parameters B. Topic configuration parameters C. Consumer configuration parameters D. Producer configuration parameters E. Admin Client configuration parameters F. Kafka Connect configuration parameters G. Kafka Streams configuration parameters H. Using Your Subscription Legal Notice Settings Close Language: 简体中文 日本語 English Language: 简体中文 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Language and Page Formatting Options Language: 简体中文 日本語 English Language: 简体中文 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Using AMQ Streams on RHEL Red Hat AMQ 2021.q3For use with AMQ Streams 1.8 on Red Hat Enterprise LinuxLegal NoticeAbstract This guide describes how to install, configure, and manage Red Hat AMQ Streams to build a large-scale messaging network. Next