Jump To Close Expand all Collapse all Table of contents Using AMQ Streams on RHEL Making open source more inclusive 1. Overview of AMQ Streams Expand section "1. Overview of AMQ Streams" Collapse section "1. Overview of AMQ Streams" 1.1. Kafka capabilities 1.2. Kafka use cases 1.3. Supported Configurations 1.4. Document conventions 2. Getting started Expand section "2. Getting started" Collapse section "2. Getting started" 2.1. AMQ Streams distribution 2.2. Downloading an AMQ Streams Archive 2.3. Installing AMQ Streams 2.4. Data storage considerations Expand section "2.4. Data storage considerations" Collapse section "2.4. Data storage considerations" 2.4.1. Apache Kafka and ZooKeeper storage support 2.4.2. File systems 2.5. Running a single node AMQ Streams cluster 2.6. Using the cluster 2.7. Stopping the AMQ Streams services 2.8. Configuring AMQ Streams 3. Configuring ZooKeeper Expand section "3. Configuring ZooKeeper" Collapse section "3. Configuring ZooKeeper" 3.1. Basic configuration 3.2. ZooKeeper cluster configuration 3.3. Running multi-node ZooKeeper cluster 3.4. Authentication Expand section "3.4. Authentication" Collapse section "3.4. Authentication" 3.4.1. Authentication with SASL 3.4.2. Enabling Server-to-server authentication using DIGEST-MD5 3.4.3. Enabling Client-to-server authentication using DIGEST-MD5 3.5. Authorization 3.6. TLS 3.7. Additional configuration options 3.8. Logging 4. Configuring Kafka Expand section "4. Configuring Kafka" Collapse section "4. Configuring Kafka" 4.1. ZooKeeper 4.2. Listeners 4.3. Commit logs 4.4. Broker ID 4.5. Running a multi-node Kafka cluster 4.6. ZooKeeper authentication Expand section "4.6. ZooKeeper authentication" Collapse section "4.6. ZooKeeper authentication" 4.6.1. JAAS Configuration 4.6.2. Enabling ZooKeeper authentication 4.7. Authorization Expand section "4.7. Authorization" Collapse section "4.7. Authorization" 4.7.1. Simple ACL authorizer Expand section "4.7.1. Simple ACL authorizer" Collapse section "4.7.1. Simple ACL authorizer" 4.7.1.1. ACL rules 4.7.1.2. Principals 4.7.1.3. Authentication of users 4.7.1.4. Super users 4.7.1.5. Replica broker authentication 4.7.1.6. Supported resources 4.7.1.7. Supported operations 4.7.1.8. ACL management options 4.7.2. Enabling authorization 4.7.3. Adding ACL rules 4.7.4. Listing ACL rules 4.7.5. Removing ACL rules 4.8. ZooKeeper authorization Expand section "4.8. ZooKeeper authorization" Collapse section "4.8. ZooKeeper authorization" 4.8.1. ACL Configuration 4.8.2. Enabling ZooKeeper ACLs for a new Kafka cluster 4.8.3. Enabling ZooKeeper ACLs in an existing Kafka cluster 4.9. Encryption and authentication Expand section "4.9. Encryption and authentication" Collapse section "4.9. Encryption and authentication" 4.9.1. Listener configuration 4.9.2. TLS Encryption 4.9.3. Enabling TLS encryption 4.9.4. Authentication Expand section "4.9.4. Authentication" Collapse section "4.9.4. Authentication" 4.9.4.1. TLS client authentication 4.9.4.2. SASL authentication 4.9.5. Enabling TLS client authentication 4.9.6. Enabling SASL PLAIN authentication 4.9.7. Enabling SASL SCRAM authentication 4.9.8. Adding SASL SCRAM users 4.9.9. Deleting SASL SCRAM users 4.10. Using OAuth 2.0 token-based authentication Expand section "4.10. Using OAuth 2.0 token-based authentication" Collapse section "4.10. Using OAuth 2.0 token-based authentication" 4.10.1. OAuth 2.0 authentication mechanisms Expand section "4.10.1. OAuth 2.0 authentication mechanisms" Collapse section "4.10.1. OAuth 2.0 authentication mechanisms" 4.10.1.1. Configuring OAuth 2.0 with properties or variables 4.10.2. OAuth 2.0 Kafka broker configuration Expand section "4.10.2. OAuth 2.0 Kafka broker configuration" Collapse section "4.10.2. OAuth 2.0 Kafka broker configuration" 4.10.2.1. OAuth 2.0 client configuration on an authorization server 4.10.2.2. OAuth 2.0 authentication configuration in the Kafka cluster 4.10.2.3. Fast local JWT token validation configuration 4.10.2.4. OAuth 2.0 introspection endpoint configuration 4.10.3. Session re-authentication for Kafka brokers 4.10.4. OAuth 2.0 Kafka client configuration 4.10.5. OAuth 2.0 client authentication flow Expand section "4.10.5. OAuth 2.0 client authentication flow" Collapse section "4.10.5. OAuth 2.0 client authentication flow" 4.10.5.1. Example client authentication flows 4.10.6. Configuring OAuth 2.0 authentication Expand section "4.10.6. Configuring OAuth 2.0 authentication" Collapse section "4.10.6. Configuring OAuth 2.0 authentication" 4.10.6.1. Configuring Red Hat Single Sign-On as an OAuth 2.0 authorization server 4.10.6.2. Configuring OAuth 2.0 support for Kafka brokers 4.10.6.3. Configuring Kafka Java clients to use OAuth 2.0 4.11. Using OAuth 2.0 token-based authorization Expand section "4.11. Using OAuth 2.0 token-based authorization" Collapse section "4.11. Using OAuth 2.0 token-based authorization" 4.11.1. OAuth 2.0 authorization mechanism Expand section "4.11.1. OAuth 2.0 authorization mechanism" Collapse section "4.11.1. OAuth 2.0 authorization mechanism" 4.11.1.1. Kafka broker custom authorizer 4.11.2. Configuring OAuth 2.0 authorization support 4.12. Using OPA policy-based authorization Expand section "4.12. Using OPA policy-based authorization" Collapse section "4.12. Using OPA policy-based authorization" 4.12.1. Defining OPA policies 4.12.2. Connecting to the OPA 4.12.3. Configuring OPA authorization support 4.13. Logging Expand section "4.13. Logging" Collapse section "4.13. Logging" 4.13.1. Dynamically change logging levels for Kafka broker loggers 5. Topics Expand section "5. Topics" Collapse section "5. Topics" 5.1. Partitions and replicas 5.2. Message retention 5.3. Topic auto-creation 5.4. Topic deletion 5.5. Topic configuration 5.6. Internal topics 5.7. Creating a topic 5.8. Listing and describing topics 5.9. Modifying a topic configuration 5.10. Deleting a topic 6. Tuning client configuration Expand section "6. Tuning client configuration" Collapse section "6. Tuning client configuration" 6.1. Kafka producer configuration tuning Expand section "6.1. Kafka producer configuration tuning" Collapse section "6.1. Kafka producer configuration tuning" 6.1.1. Basic producer configuration 6.1.2. Data durability 6.1.3. Ordered delivery 6.1.4. Reliability guarantees 6.1.5. Optimizing throughput and latency 6.2. Kafka consumer configuration tuning Expand section "6.2. Kafka consumer configuration tuning" Collapse section "6.2. Kafka consumer configuration tuning" 6.2.1. Basic consumer configuration 6.2.2. Scaling data consumption using consumer groups 6.2.3. Message ordering guarantees 6.2.4. Optimizing throughput and latency 6.2.5. Avoiding data loss or duplication when committing offsets Expand section "6.2.5. Avoiding data loss or duplication when committing offsets" Collapse section "6.2.5. Avoiding data loss or duplication when committing offsets" 6.2.5.1. Controlling transactional messages 6.2.6. Recovering from failure to avoid data loss 6.2.7. Managing offset policy 6.2.8. Minimizing the impact of rebalances 7. Scaling Clusters Expand section "7. Scaling Clusters" Collapse section "7. Scaling Clusters" 7.1. Scaling Kafka clusters Expand section "7.1. Scaling Kafka clusters" Collapse section "7.1. Scaling Kafka clusters" 7.1.1. Adding brokers to a cluster 7.1.2. Removing brokers from the cluster 7.2. Reassignment of partitions Expand section "7.2. Reassignment of partitions" Collapse section "7.2. Reassignment of partitions" 7.2.1. Reassignment JSON file 7.2.2. Generating reassignment JSON files 7.2.3. Creating reassignment JSON files manually 7.3. Reassignment throttles 7.4. Scaling up a Kafka cluster 7.5. Scaling down a Kafka cluster 7.6. Scaling up a ZooKeeper cluster 7.7. Scaling down a ZooKeeper cluster 8. Monitoring your cluster using JMX Expand section "8. Monitoring your cluster using JMX" Collapse section "8. Monitoring your cluster using JMX" 8.1. JMX configuration options 8.2. Disabling the JMX agent 8.3. Connecting to the JVM from a different machine 8.4. Monitoring using JConsole 8.5. Important Kafka broker metrics Expand section "8.5. Important Kafka broker metrics" Collapse section "8.5. Important Kafka broker metrics" 8.5.1. Kafka server metrics 8.5.2. Kafka network metrics 8.5.3. Kafka log metrics 8.5.4. Kafka controller metrics 8.5.5. Yammer metrics 8.6. Producer MBeans Expand section "8.6. Producer MBeans" Collapse section "8.6. Producer MBeans" 8.6.1. MBeans matching kafka.producer:type=producer-metrics,client-id=* 8.6.2. MBeans matching kafka.producer:type=producer-metrics,client-id=*,node-id=* 8.6.3. MBeans matching kafka.producer:type=producer-topic-metrics,client-id=*,topic=* 8.7. Consumer MBeans Expand section "8.7. Consumer MBeans" Collapse section "8.7. Consumer MBeans" 8.7.1. MBeans matching kafka.consumer:type=consumer-metrics,client-id=* 8.7.2. MBeans matching kafka.consumer:type=consumer-metrics,client-id=*,node-id=* 8.7.3. MBeans matching kafka.consumer:type=consumer-coordinator-metrics,client-id=* 8.7.4. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* 8.7.5. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=* 8.7.6. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=* 8.8. Kafka Connect MBeans Expand section "8.8. Kafka Connect MBeans" Collapse section "8.8. Kafka Connect MBeans" 8.8.1. MBeans matching kafka.connect:type=connect-metrics,client-id=* 8.8.2. MBeans matching kafka.connect:type=connect-metrics,client-id=*,node-id=* 8.8.3. MBeans matching kafka.connect:type=connect-worker-metrics 8.8.4. MBeans matching kafka.connect:type=connect-worker-rebalance-metrics 8.8.5. MBeans matching kafka.connect:type=connector-metrics,connector=* 8.8.6. MBeans matching kafka.connect:type=connector-task-metrics,connector=*,task=* 8.8.7. MBeans matching kafka.connect:type=sink-task-metrics,connector=*,task=* 8.8.8. MBeans matching kafka.connect:type=source-task-metrics,connector=*,task=* 8.8.9. MBeans matching kafka.connect:type=task-error-metrics,connector=*,task=* 8.9. Kafka Streams MBeans Expand section "8.9. Kafka Streams MBeans" Collapse section "8.9. Kafka Streams MBeans" 8.9.1. MBeans matching kafka.streams:type=stream-metrics,client-id=* 8.9.2. MBeans matching kafka.streams:type=stream-task-metrics,client-id=*,task-id=* 8.9.3. MBeans matching kafka.streams:type=stream-processor-node-metrics,client-id=*,task-id=*,processor-node-id=* 8.9.4. MBeans matching kafka.streams:type=stream-[store-scope]-metrics,client-id=*,task-id=*,[store-scope]-id=* 8.9.5. MBeans matching kafka.streams:type=stream-record-cache-metrics,client-id=*,task-id=*,record-cache-id=* 9. Kafka Connect Expand section "9. Kafka Connect" Collapse section "9. Kafka Connect" 9.1. Kafka Connect in standalone mode Expand section "9.1. Kafka Connect in standalone mode" Collapse section "9.1. Kafka Connect in standalone mode" 9.1.1. Configuring Kafka Connect in standalone mode 9.1.2. Configuring connectors in Kafka Connect in standalone mode 9.1.3. Running Kafka Connect in standalone mode 9.2. Kafka Connect in distributed mode Expand section "9.2. Kafka Connect in distributed mode" Collapse section "9.2. Kafka Connect in distributed mode" 9.2.1. Configuring Kafka Connect in distributed mode 9.2.2. Configuring connectors in distributed Kafka Connect 9.2.3. Running distributed Kafka Connect 9.2.4. Creating connectors 9.2.5. Deleting connectors 9.3. Connector plug-ins 9.4. Adding connector plugins 10. Using AMQ Streams with MirrorMaker 2.0 Expand section "10. Using AMQ Streams with MirrorMaker 2.0" Collapse section "10. Using AMQ Streams with MirrorMaker 2.0" 10.1. MirrorMaker 2.0 data replication 10.2. Cluster configuration Expand section "10.2. Cluster configuration" Collapse section "10.2. Cluster configuration" 10.2.1. Bidirectional replication (active/active) 10.2.2. Unidirectional replication (active/passive) 10.2.3. Topic configuration synchronization 10.2.4. Data integrity 10.2.5. Offset tracking 10.2.6. Synchronizing consumer group offsets 10.2.7. Connectivity checks 10.3. ACL rules synchronization 10.4. Synchronizing data between Kafka clusters using MirrorMaker 2.0 10.5. Using MirrorMaker 2.0 in legacy mode 11. Kafka clients Expand section "11. Kafka clients" Collapse section "11. Kafka clients" 11.1. Adding Kafka clients as a dependency to your Maven project 12. Kafka Streams API overview Expand section "12. Kafka Streams API overview" Collapse section "12. Kafka Streams API overview" 12.1. Adding the Kafka Streams API as a dependency to your Maven project 13. Kafka Bridge Expand section "13. Kafka Bridge" Collapse section "13. Kafka Bridge" 13.1. Kafka Bridge overview Expand section "13.1. Kafka Bridge overview" Collapse section "13.1. Kafka Bridge overview" 13.1.1. Authentication and encryption 13.1.2. Requests to the Kafka Bridge Expand section "13.1.2. Requests to the Kafka Bridge" Collapse section "13.1.2. Requests to the Kafka Bridge" 13.1.2.1. Content Type headers 13.1.2.2. Embedded data format 13.1.2.3. Message format 13.1.2.4. Accept headers 13.1.3. Configuring loggers for the Kafka Bridge 13.1.4. Kafka Bridge API resources 13.1.5. Downloading a Kafka Bridge archive 13.1.6. Configuring Kafka Bridge properties 13.1.7. Installing the Kafka Bridge 13.2. Kafka Bridge quickstart Expand section "13.2. Kafka Bridge quickstart" Collapse section "13.2. Kafka Bridge quickstart" 13.2.1. Deploying the Kafka Bridge locally 13.2.2. Producing messages to topics and partitions 13.2.3. Creating a Kafka Bridge consumer 13.2.4. Subscribing a Kafka Bridge consumer to topics 13.2.5. Retrieving the latest messages from a Kafka Bridge consumer 13.2.6. Commiting offsets to the log 13.2.7. Seeking to offsets for a partition 13.2.8. Deleting a Kafka Bridge consumer 14. Using Kerberos (GSSAPI) authentication Expand section "14. Using Kerberos (GSSAPI) authentication" Collapse section "14. Using Kerberos (GSSAPI) authentication" 14.1. Setting up AMQ Streams to use Kerberos (GSSAPI) authentication 15. Cruise Control for cluster rebalancing Expand section "15. Cruise Control for cluster rebalancing" Collapse section "15. Cruise Control for cluster rebalancing" 15.1. Why use Cruise Control? 15.2. Downloading a Cruise Control archive 15.3. Deploying the Cruise Control Metrics Reporter 15.4. Configuring and starting Cruise Control 15.5. Optimization goals overview 15.6. Optimization proposals overview 15.7. Rebalance performance tuning overview 15.8. Cruise Control configuration 15.9. Generating optimization proposals 15.10. Initiating a cluster rebalance 15.11. Stopping an active cluster rebalance 16. Distributed tracing Expand section "16. Distributed tracing" Collapse section "16. Distributed tracing" 16.1. Overview of OpenTracing and Jaeger 16.2. Setting up tracing for Kafka clients Expand section "16.2. Setting up tracing for Kafka clients" Collapse section "16.2. Setting up tracing for Kafka clients" 16.2.1. Initializing a Jaeger tracer for Kafka clients 16.2.2. Instrumenting producers and consumers for tracing 16.2.3. Instrumenting Kafka Streams applications for tracing 16.3. Setting up tracing for MirrorMaker and Kafka Connect Expand section "16.3. Setting up tracing for MirrorMaker and Kafka Connect" Collapse section "16.3. Setting up tracing for MirrorMaker and Kafka Connect" 16.3.1. Enabling tracing for MirrorMaker 16.3.2. Enabling tracing for MirrorMaker 2.0 16.3.3. Enabling tracing for Kafka Connect 16.4. Enabling tracing for the Kafka Bridge 16.5. Environment variables for tracing 17. Kafka Exporter Expand section "17. Kafka Exporter" Collapse section "17. Kafka Exporter" 17.1. Consumer lag 17.2. Kafka Exporter alerting rule examples 17.3. Kafka Exporter metrics 17.4. Running Kafka Exporter 17.5. Presenting Kafka Exporter metrics in Grafana 18. AMQ Streams and Kafka upgrades Expand section "18. AMQ Streams and Kafka upgrades" Collapse section "18. AMQ Streams and Kafka upgrades" 18.1. Upgrade prerequisites 18.2. Upgrade process 18.3. Kafka versions 18.4. Upgrading to AMQ Streams 1.7 Expand section "18.4. Upgrading to AMQ Streams 1.7" Collapse section "18.4. Upgrading to AMQ Streams 1.7" 18.4.1. Upgrading Kafka brokers and ZooKeeper 18.4.2. Upgrading Kafka Connect 18.5. Upgrading Kafka Expand section "18.5. Upgrading Kafka" Collapse section "18.5. Upgrading Kafka" 18.5.1. Upgrading Kafka brokers to use the new inter-broker protocol version 18.5.2. Strategies for upgrading clients 18.5.3. Upgrading client applications to the new Kafka version 18.5.4. Upgrading Kafka brokers to use the new message format version 18.5.5. Upgrading consumers and Kafka Streams applications to cooperative rebalancing A. Broker configuration parameters B. Topic configuration parameters C. Consumer configuration parameters D. Producer configuration parameters E. Admin Client configuration parameters F. Kafka Connect configuration parameters G. Kafka Streams configuration parameters H. Using Your Subscription Legal Notice Settings Close Language: 日本語 English Language: 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Language and Page Formatting Options Language: 日本語 English Language: 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Using AMQ Streams on RHEL Red Hat AMQ 2021.q2For use with AMQ Streams 1.7 on Red Hat Enterprise LinuxLegal NoticeAbstract This guide describes how to install, configure, and manage Red Hat AMQ Streams to build a large-scale messaging network. Next