Jump To Close Expand all Collapse all Table of contents Using AMQ Streams on RHEL 1. Overview of AMQ Streams Expand section "1. Overview of AMQ Streams" Collapse section "1. Overview of AMQ Streams" 1.1. Kafka capabilities 1.2. Kafka use cases 1.3. Supported Configurations 1.4. Document conventions 2. Getting started Expand section "2. Getting started" Collapse section "2. Getting started" 2.1. AMQ Streams distribution 2.2. Downloading an AMQ Streams Archive 2.3. Installing AMQ Streams 2.4. Data storage considerations Expand section "2.4. Data storage considerations" Collapse section "2.4. Data storage considerations" 2.4.1. Apache Kafka and ZooKeeper storage support 2.4.2. File systems 2.5. Running a single node AMQ Streams cluster 2.6. Using the cluster 2.7. Stopping the AMQ Streams services 2.8. Configuring AMQ Streams 3. Configuring ZooKeeper Expand section "3. Configuring ZooKeeper" Collapse section "3. Configuring ZooKeeper" 3.1. Basic configuration 3.2. ZooKeeper cluster configuration 3.3. Running multi-node ZooKeeper cluster 3.4. Authentication Expand section "3.4. Authentication" Collapse section "3.4. Authentication" 3.4.1. Authentication with SASL 3.4.2. Enabling Server-to-server authentication using DIGEST-MD5 3.4.3. Enabling Client-to-server authentication using DIGEST-MD5 3.5. Authorization 3.6. TLS 3.7. Additional configuration options 3.8. Logging 4. Configuring Kafka Expand section "4. Configuring Kafka" Collapse section "4. Configuring Kafka" 4.1. ZooKeeper 4.2. Listeners 4.3. Commit logs 4.4. Broker ID 4.5. Running a multi-node Kafka cluster 4.6. ZooKeeper authentication Expand section "4.6. ZooKeeper authentication" Collapse section "4.6. ZooKeeper authentication" 4.6.1. JAAS Configuration 4.6.2. Enabling ZooKeeper authentication 4.7. Authorization Expand section "4.7. Authorization" Collapse section "4.7. Authorization" 4.7.1. Simple ACL authorizer Expand section "4.7.1. Simple ACL authorizer" Collapse section "4.7.1. Simple ACL authorizer" 4.7.1.1. ACL rules 4.7.1.2. Principals 4.7.1.3. Authentication of users 4.7.1.4. Super users 4.7.1.5. Replica broker authentication 4.7.1.6. Supported resources 4.7.1.7. Supported operations 4.7.1.8. ACL management options 4.7.2. Enabling authorization 4.7.3. Adding ACL rules 4.7.4. Listing ACL rules 4.7.5. Removing ACL rules 4.8. ZooKeeper authorization Expand section "4.8. ZooKeeper authorization" Collapse section "4.8. ZooKeeper authorization" 4.8.1. ACL Configuration 4.8.2. Enabling ZooKeeper ACLs for a new Kafka cluster 4.8.3. Enabling ZooKeeper ACLs in an existing Kafka cluster 4.9. Encryption and authentication Expand section "4.9. Encryption and authentication" Collapse section "4.9. Encryption and authentication" 4.9.1. Listener configuration 4.9.2. TLS Encryption 4.9.3. Enabling TLS encryption 4.9.4. Authentication Expand section "4.9.4. Authentication" Collapse section "4.9.4. Authentication" 4.9.4.1. TLS client authentication 4.9.4.2. SASL authentication 4.9.5. Enabling TLS client authentication 4.9.6. Enabling SASL PLAIN authentication 4.9.7. Enabling SASL SCRAM authentication 4.9.8. Adding SASL SCRAM users 4.9.9. Deleting SASL SCRAM users 4.10. Using OAuth 2.0 token-based authentication Expand section "4.10. Using OAuth 2.0 token-based authentication" Collapse section "4.10. Using OAuth 2.0 token-based authentication" 4.10.1. OAuth 2.0 authentication mechanism Expand section "4.10.1. OAuth 2.0 authentication mechanism" Collapse section "4.10.1. OAuth 2.0 authentication mechanism" 4.10.1.1. Configuring OAuth 2.0 with properties or variables 4.10.2. OAuth 2.0 Kafka broker configuration Expand section "4.10.2. OAuth 2.0 Kafka broker configuration" Collapse section "4.10.2. OAuth 2.0 Kafka broker configuration" 4.10.2.1. OAuth 2.0 client configuration on an authorization server 4.10.2.2. OAuth 2.0 authentication configuration in the Kafka cluster 4.10.2.3. Fast local JWT token validation configuration 4.10.2.4. OAuth 2.0 introspection endpoint configuration 4.10.3. OAuth 2.0 Kafka client configuration 4.10.4. OAuth 2.0 client authentication flow Expand section "4.10.4. OAuth 2.0 client authentication flow" Collapse section "4.10.4. OAuth 2.0 client authentication flow" 4.10.4.1. Example client authentication flows 4.10.5. Configuring OAuth 2.0 authentication Expand section "4.10.5. Configuring OAuth 2.0 authentication" Collapse section "4.10.5. Configuring OAuth 2.0 authentication" 4.10.5.1. Configuring Red Hat Single Sign-On as an OAuth 2.0 authorization server 4.10.5.2. Configuring OAuth 2.0 support for Kafka brokers 4.10.5.3. Configuring Kafka Java clients to use OAuth 2.0 4.11. Using OAuth 2.0 token-based authorization Expand section "4.11. Using OAuth 2.0 token-based authorization" Collapse section "4.11. Using OAuth 2.0 token-based authorization" 4.11.1. OAuth 2.0 authorization mechanism Expand section "4.11.1. OAuth 2.0 authorization mechanism" Collapse section "4.11.1. OAuth 2.0 authorization mechanism" 4.11.1.1. Kafka broker custom authorizer 4.11.2. Configuring OAuth 2.0 authorization support 4.12. Logging 5. Topics Expand section "5. Topics" Collapse section "5. Topics" 5.1. Partitions and replicas 5.2. Message retention 5.3. Topic auto-creation 5.4. Topic deletion 5.5. Topic configuration 5.6. Internal topics 5.7. Creating a topic 5.8. Listing and describing topics 5.9. Modifying a topic configuration 5.10. Deleting a topic 6. Scaling Clusters Expand section "6. Scaling Clusters" Collapse section "6. Scaling Clusters" 6.1. Scaling Kafka clusters Expand section "6.1. Scaling Kafka clusters" Collapse section "6.1. Scaling Kafka clusters" 6.1.1. Adding brokers to a cluster 6.1.2. Removing brokers from the cluster 6.2. Reassignment of partitions Expand section "6.2. Reassignment of partitions" Collapse section "6.2. Reassignment of partitions" 6.2.1. Reassignment JSON file 6.2.2. Generating reassignment JSON files 6.2.3. Creating reassignment JSON files manually 6.3. Reassignment throttles 6.4. Scaling up a Kafka cluster 6.5. Scaling down a Kafka cluster 6.6. Scaling up a ZooKeeper cluster 6.7. Scaling down a ZooKeeper cluster 7. Monitoring your cluster using JMX Expand section "7. Monitoring your cluster using JMX" Collapse section "7. Monitoring your cluster using JMX" 7.1. JMX configuration options 7.2. Disabling the JMX agent 7.3. Connecting to the JVM from a different machine 7.4. Monitoring using JConsole 7.5. Important Kafka broker metrics Expand section "7.5. Important Kafka broker metrics" Collapse section "7.5. Important Kafka broker metrics" 7.5.1. Kafka server metrics 7.5.2. Kafka network metrics 7.5.3. Kafka log metrics 7.5.4. Kafka controller metrics 7.5.5. Yammer metrics 7.6. Producer MBeans Expand section "7.6. Producer MBeans" Collapse section "7.6. Producer MBeans" 7.6.1. MBeans matching kafka.producer:type=producer-metrics,client-id=* 7.6.2. MBeans matching kafka.producer:type=producer-metrics,client-id=*,node-id=* 7.6.3. MBeans matching kafka.producer:type=producer-topic-metrics,client-id=*,topic=* 7.7. Consumer MBeans Expand section "7.7. Consumer MBeans" Collapse section "7.7. Consumer MBeans" 7.7.1. MBeans matching kafka.consumer:type=consumer-metrics,client-id=* 7.7.2. MBeans matching kafka.consumer:type=consumer-metrics,client-id=*,node-id=* 7.7.3. MBeans matching kafka.consumer:type=consumer-coordinator-metrics,client-id=* 7.7.4. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=* 7.7.5. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=* 7.7.6. MBeans matching kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*,partition=* 7.8. Kafka Connect MBeans Expand section "7.8. Kafka Connect MBeans" Collapse section "7.8. Kafka Connect MBeans" 7.8.1. MBeans matching kafka.connect:type=connect-metrics,client-id=* 7.8.2. MBeans matching kafka.connect:type=connect-metrics,client-id=*,node-id=* 7.8.3. MBeans matching kafka.connect:type=connect-worker-metrics 7.8.4. MBeans matching kafka.connect:type=connect-worker-rebalance-metrics 7.8.5. MBeans matching kafka.connect:type=connector-metrics,connector=* 7.8.6. MBeans matching kafka.connect:type=connector-task-metrics,connector=*,task=* 7.8.7. MBeans matching kafka.connect:type=sink-task-metrics,connector=*,task=* 7.8.8. MBeans matching kafka.connect:type=source-task-metrics,connector=*,task=* 7.8.9. MBeans matching kafka.connect:type=task-error-metrics,connector=*,task=* 7.9. Kafka Streams MBeans Expand section "7.9. Kafka Streams MBeans" Collapse section "7.9. Kafka Streams MBeans" 7.9.1. MBeans matching kafka.streams:type=stream-metrics,client-id=* 7.9.2. MBeans matching kafka.streams:type=stream-task-metrics,client-id=*,task-id=* 7.9.3. MBeans matching kafka.streams:type=stream-processor-node-metrics,client-id=*,task-id=*,processor-node-id=* 7.9.4. MBeans matching kafka.streams:type=stream-[store-scope]-metrics,client-id=*,task-id=*,[store-scope]-id=* 7.9.5. MBeans matching kafka.streams:type=stream-record-cache-metrics,client-id=*,task-id=*,record-cache-id=* 8. Kafka Connect Expand section "8. Kafka Connect" Collapse section "8. Kafka Connect" 8.1. Kafka Connect in standalone mode Expand section "8.1. Kafka Connect in standalone mode" Collapse section "8.1. Kafka Connect in standalone mode" 8.1.1. Configuring Kafka Connect in standalone mode 8.1.2. Configuring connectors in Kafka Connect in standalone mode 8.1.3. Running Kafka Connect in standalone mode 8.2. Kafka Connect in distributed mode Expand section "8.2. Kafka Connect in distributed mode" Collapse section "8.2. Kafka Connect in distributed mode" 8.2.1. Configuring Kafka Connect in distributed mode 8.2.2. Configuring connectors in distributed Kafka Connect 8.2.3. Running distributed Kafka Connect 8.2.4. Creating connectors 8.2.5. Deleting connectors 8.3. Connector plug-ins 8.4. Adding connector plugins 9. Using AMQ Streams with MirrorMaker 2.0 Expand section "9. Using AMQ Streams with MirrorMaker 2.0" Collapse section "9. Using AMQ Streams with MirrorMaker 2.0" 9.1. MirrorMaker 2.0 data replication 9.2. Cluster configuration Expand section "9.2. Cluster configuration" Collapse section "9.2. Cluster configuration" 9.2.1. Bidirectional replication 9.2.2. Topic configuration synchronization 9.2.3. Data integrity 9.2.4. Offset tracking 9.2.5. Connectivity checks 9.3. ACL rules synchronization 9.4. Synchronizing data between Kafka clusters using MirrorMaker 2.0 9.5. Using MirrorMaker 2.0 in legacy mode 10. Kafka clients Expand section "10. Kafka clients" Collapse section "10. Kafka clients" 10.1. Adding Kafka clients as a dependency to your Maven project 11. Kafka Streams API overview Expand section "11. Kafka Streams API overview" Collapse section "11. Kafka Streams API overview" 11.1. Adding the Kafka Streams API as a dependency to your Maven project 12. Kafka Bridge Expand section "12. Kafka Bridge" Collapse section "12. Kafka Bridge" 12.1. Kafka Bridge overview Expand section "12.1. Kafka Bridge overview" Collapse section "12.1. Kafka Bridge overview" 12.1.1. Authentication and encryption 12.1.2. Requests to the Kafka Bridge Expand section "12.1.2. Requests to the Kafka Bridge" Collapse section "12.1.2. Requests to the Kafka Bridge" 12.1.2.1. Content Type headers 12.1.2.2. Embedded data format 12.1.2.3. Accept headers 12.1.3. Configuring loggers for the Kafka Bridge 12.1.4. Kafka Bridge API resources 12.1.5. Downloading a Kafka Bridge archive 12.1.6. Configuring Kafka Bridge properties 12.1.7. Installing the Kafka Bridge 12.2. Kafka Bridge quickstart Expand section "12.2. Kafka Bridge quickstart" Collapse section "12.2. Kafka Bridge quickstart" 12.2.1. Deploying the Kafka Bridge locally 12.2.2. Producing messages to topics and partitions 12.2.3. Creating a Kafka Bridge consumer 12.2.4. Subscribing a Kafka Bridge consumer to topics 12.2.5. Retrieving the latest messages from a Kafka Bridge consumer 12.2.6. Commiting offsets to the log 12.2.7. Seeking to offsets for a partition 12.2.8. Deleting a Kafka Bridge consumer 13. Cruise Control for cluster rebalancing Expand section "13. Cruise Control for cluster rebalancing" Collapse section "13. Cruise Control for cluster rebalancing" 13.1. Why use Cruise Control? 13.2. Downloading a Cruise Control archive 13.3. Deploying the Cruise Control Metrics Reporter 13.4. Configuring and starting Cruise Control 13.5. Optimization goals overview 13.6. Optimization proposals overview 13.7. Cruise Control configuration 13.8. Generating optimization proposals 13.9. Initiating a cluster rebalance 13.10. Stopping an active cluster rebalance 14. Distributed tracing Expand section "14. Distributed tracing" Collapse section "14. Distributed tracing" 14.1. Overview of distributed tracing Expand section "14.1. Overview of distributed tracing" Collapse section "14.1. Overview of distributed tracing" 14.1.1. OpenTracing and Jaeger 14.2. Configuring Kafka clients for tracing Expand section "14.2. Configuring Kafka clients for tracing" Collapse section "14.2. Configuring Kafka clients for tracing" 14.2.1. Enabling a Jaeger tracer for Kafka clients 14.2.2. Instrumenting Kafka Producers and Consumers for tracing Expand section "14.2.2. Instrumenting Kafka Producers and Consumers for tracing" Collapse section "14.2.2. Instrumenting Kafka Producers and Consumers for tracing" 14.2.2.1. Custom span names in a Decorator pattern 14.2.2.2. Built-in span names 14.2.3. Instrumenting Kafka Streams applications for tracing 14.3. Configuring MirrorMaker and Kafka Connect for tracing Expand section "14.3. Configuring MirrorMaker and Kafka Connect for tracing" Collapse section "14.3. Configuring MirrorMaker and Kafka Connect for tracing" 14.3.1. Enabling tracing for MirrorMaker 14.3.2. Enabling tracing for Kafka Connect 14.4. Enabling tracing for the Kafka Bridge 14.5. Tracing environment variables 15. Kafka Exporter Expand section "15. Kafka Exporter" Collapse section "15. Kafka Exporter" 15.1. Consumer lag 15.2. Kafka Exporter alerting rule examples 15.3. Kafka Exporter metrics 15.4. Running Kafka Exporter 15.5. Presenting Kafka Exporter metrics in Grafana 16. AMQ Streams and Kafka upgrades Expand section "16. AMQ Streams and Kafka upgrades" Collapse section "16. AMQ Streams and Kafka upgrades" 16.1. Upgrade prerequisites 16.2. Upgrade process 16.3. Kafka versions 16.4. Upgrading to AMQ Streams 1.5 Expand section "16.4. Upgrading to AMQ Streams 1.5" Collapse section "16.4. Upgrading to AMQ Streams 1.5" 16.4.1. Upgrading ZooKeeper 16.4.2. Upgrading Kafka brokers 16.4.3. Upgrading Kafka Connect 16.5. Upgrading Kafka Expand section "16.5. Upgrading Kafka" Collapse section "16.5. Upgrading Kafka" 16.5.1. Upgrading Kafka brokers to use the new inter-broker protocol version 16.5.2. Strategies for upgrading clients 16.5.3. Upgrading client applications to the new Kafka version 16.5.4. Upgrading consumers and Kafka Streams applications to cooperative rebalancing 16.5.5. Upgrading Kafka brokers to use the new message format version A. Broker configuration parameters B. Topic configuration parameters C. Consumer configuration parameters D. Producer configuration parameters E. Admin Client configuration parameters F. Kafka Connect configuration parameters G. Kafka Streams configuration parameters H. Using Your Subscription Legal Notice Settings Close Language: 日本語 English Language: 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Language and Page Formatting Options Language: 日本語 English Language: 日本語 English Format: Multi-page Single-page PDF Format: Multi-page Single-page PDF Using AMQ Streams on RHEL Red Hat AMQ 7.7For use with AMQ Streams 1.5 on Red Hat Enterprise LinuxLegal NoticeAbstract This guide describes how to install, configure, and manage Red Hat AMQ Streams to build a large-scale messaging network. Next