As a developer of applications and services, you can use Kafka scripts to produce and consume messages for Kafka instances in Red Hat OpenShift Streams for Apache Kafka. This is a useful way to test and debug your Kafka instances. The Kafka scripts are a set of shell scripts that are included with the Apache Kafka distribution.
When you download and extract the Apache Kafka distribution, the bin/
directory of the distribution (or the bin\windows\
directory if you’re using Windows) contains a set of shell scripts that enable you to interact with your Kafka instance. With the scripts, you can produce and consume messages using your Kafka instances. You can also perform various other operations against the Kafka APIs to administer topics, consumer groups, and other resources.
<Kafka-distribution-dir>/bin/kafka-console-producer.sh
script, use the
<Kafka-distribution-dir>\bin\windows\kafka-console-producer.bat
script.
-
You have a running Kafka instance in OpenShift Streams for Apache Kafka (see Getting started with Red Hat OpenShift Streams for Apache Kafka).
-
You have a command-line terminal application.
-
JDK 11 or later is installed. (The latest LTS version of OpenJDK is recommended.)
-
You’ve downloaded the latest supported binary version of the Apache Kafka distribution. You can check your Kafka version using the following command:
$ ./kafka-console-producer.sh --version
Configuring the Kafka scripts to connect to a Kafka instance
To enable the Kafka scripts to access a Kafka instance, you must configure a connection using the generated credentials for your Red Hat OpenShift Application Services service account. In this task, you create a configuration file that specifies these credential values.
-
You have the generated credentials for your service account. To reset the credentials, use the Service Accounts page.
-
You’ve set permissions for your service account to access resources in the Kafka instance. To verify the current permissions, select your Kafka instance in the Red Hat OpenShift Streams for Apache Kafka web console and click the Access tab. To learn more about setting permissions, see Managing account access in Red Hat OpenShift Streams for Apache Kafka.
-
In your Kafka distribution, navigate to the
config/
directory. -
Create a file called
app-services.properties
. -
In the
app-services.properties
file, set the SASL connection mechanism and the Kafka instance client credentials as shown in the following configuration. Replace the values with your own credential information. OpenShift Streams for Apache Kafka supports the SASL/OAUTHBEARER mechanism for authentication, which is the recommended authentication mechanism to use.Setting server and credential valuessasl.mechanism=OAUTHBEARER security.protocol=SASL_SSL sasl.oauthbearer.token.endpoint.url= https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ scope="openid" \ clientId="<client-id>" \ clientSecret="<client-secret>" ; sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
-
Save the file. You’ll use this file in the next task to connect to your Kafka instance and produce messages.
Producing messages using Kafka scripts
In this task, you use the kafka-console-producer
script to produce messages to a Kafka topic.
-
You have a running Kafka instance in Red Hat OpenShift Streams for Apache Kafka (see Getting started with Red Hat OpenShift Streams for Apache Kafka).
-
You have the bootstrap server endpoint for your Kafka instance. To get the server endpoint, select your Kafka instance in the Red Hat OpenShift Streams for Apache Kafka web console, select the options icon (three vertical dots), and click Connection.
-
You’ve created the
app-services.properties
file to store your service account credentials.
-
On the command line, navigate to the
bin/
directory of your Kafka distribution. -
Use the
kafka-topics
script to create a Kafka topic, as shown in the following example. The example creates a topic calledmy-other-topic
with the default settings. Replace <bootstrap_server> with the bootstrap server endpoint for your own Kafka instance.For trial Kafka instances, the default value for the replication factor is 1.Using thekafka-topics
script to create a Kafka topic$ ./kafka-topics.sh --create --topic my-other-topic --partitions 1 --replication-factor 3 --bootstrap-server <bootstrap_server> --command-config ../config/app-services.properties
If you try to create a topic with a number of partitions that would cause the partition limit of the Kafka instance to be exceeded, you see an error message indicating this. For more information about partition limits for Kafka instances, see Red Hat OpenShift Streams for Apache Kafka service limits. -
Start the
kafka-console-producer
script, as shown in the following example. The example uses the SASL/OAUTHBEARER authentication mechanism with the credentials that you saved in theapp-services.properties
file. The command prepares a producer to send messages to themy-other-topic
topic that you previously created.Starting thekafka-console-producer
script$ ./kafka-console-producer.sh --topic my-other-topic --bootstrap-server "<bootstrap_server>" --producer.config ../config/app-services.properties
-
With the
kafka-console-producer
script running, enter messages that you want to produce to the Kafka topic.Example messages to produce to the Kafka topic>First message >Second message >Third message
-
Keep the producer running so that you can use it again later, when you create a consumer.
-
Verify that the
kafka-console-producer
script is still running without any errors in the terminal.
Consuming messages using Kafka scripts
In this task, you use the kafka-console-consumer
script to consume the messages that you previously produced with the kafka-console-producer
script.
-
You used the
kafka-console-producer
script to produce example messages to a topic.
-
Open a second terminal window or tab, separate from the producer.
-
On the command line, navigate to the
bin/
directory of your Kafka distribution. -
Start the
kafka-console-consumer
script, as shown in the following example. The example uses the SASL/OAUTHBEARER authentication mechanism with the credentials that you saved in theapp-services.properties
file. The command consumes and displays messages from themy-other-topic
topic.Starting thekafka-console-consumer
script$ ./kafka-console-consumer.sh --topic my-other-topic --bootstrap-server "<bootstrap_server>" --from-beginning --consumer.config ../config/app-services.properties
You see output like the following example:
First message Second message Third message
-
If your producer is still running in a separate terminal, continue entering messages in the producer terminal and observe the messages being consumed in the consumer terminal.
-
Verify that the
kafka-console-consumer
script is running without any errors in the terminal. -
Verify that the
kafka-console-consumer
script displays the messages from themy-other-topic
example topic.