Getting started with Red Hat OpenShift Streams for Apache Kafka

Guide
  • Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 19 October 2021
  • Published 30 March 2021

Getting started with Red Hat OpenShift Streams for Apache Kafka

Guide
Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 19 October 2021
  • Published 30 March 2021

Red Hat OpenShift Streams for Apache Kafka is currently available for Development Preview. Development Preview releases provide early access to a limited set of features that might not be fully tested and that might change in the final GA version. Users should not use Development Preview software in production or for business-critical workloads. Limited documentation is available for Development Preview releases and is typically focused on fundamental user goals.

As a developer of applications and services, you can use Red Hat OpenShift Streams for Apache Kafka to create and set up Kafka instances and connect your applications and services to these instances. OpenShift Streams for Apache Kafka is a managed cloud service that enables you to add Kafka data-streaming functionality in your applications without having to install, configure, run, and maintain your own Kafka clusters.

Prerequisites
  • You have a Red Hat account.

  • JDK 11 or later is installed.

  • For Windows, the latest version of Oracle JDK is installed.

Creating a Kafka instance in OpenShift Streams for Apache Kafka

Use the OpenShift Streams for Apache Kafka web console to create and configure a Kafka instance for your applications or services. A Kafka instance in OpenShift Streams for Apache Kafka includes a Kafka cluster, bootstrap server, and the configurations needed to connect to producer and consumer services.

Prerequisites
Procedure
  1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click Create Kafka instance.

  2. Enter a unique Name for the Kafka instance, such as my-first-kafka-instance. For this release of OpenShift Streams for Apache Kafka, all other fields are preconfigured.

  3. Click Create instance to start the creation process for your Kafka instance.

    Image of instance configuration details in Create Kafka instance window
    Figure 1. Kafka instance configuration details

    The new Kafka instance is listed in the instances table. When the instance Status is Ready, you can start using the Kafka instance. You can use the options icon (three vertical dots) to view instance details, connect to the instance, or delete the instance.

    Image of Kafka instance options menu
    Figure 2. Kafka instance options menu
    Although you can see Kafka instances created by other users in your organization, you can’t manage or connect to those instances. Only the user who creates an instance can edit or delete the instance, access the associated service account and topics, or connect to the instance.
Verification
  1. Verify that the new Kafka instance is listed in the instances table.

  2. Verify that the state of the new Kafka instance is shown as Ready.

Creating a service account to connect to a Kafka instance in OpenShift Streams for Apache Kafka

To connect your applications or services to a Kafka instance in the OpenShift Streams for Apache Kafka web console, you need to create a service account that’s associated with the instance. You also need to save the bootstrap server endpoint for the instance, the generated service account credentials, and the authentication token endpoint to a secure location. You’ll use the service account and Kafka instance information later when you configure your application.

Prerequisites
  • You’ve created a Kafka instance and the instance is in Ready state.

Procedure
  1. In the Kafka Instances page of the web console, for the relevant Kafka instance that you want to connect to, select the options icon (three vertical dots) and click Connection.

  2. In the Connection page, copy the Bootstrap server endpoint to a secure location. This is the bootstrap server endpoint that you’ll need for connecting to this Kafka instance.

  3. Click Create service account to set up the account that you’ll use to access this Kafka instance.

  4. Enter a unique service account name, such as my-service-account, add an optional description, and click Create.

  5. Copy the generated Client ID and Client Secret to a secure location. These are the credentials that you’ll use to connect to this Kafka instance.

    The generated credentials are displayed only one time, so ensure that you’ve successfully and securely saved the copied credentials before closing the credentials window.
  6. After you save the generated credentials to a secure location, select the confirmation check box in the credentials window and close the window.

  7. For the Authentication method, copy the SASL/OAUTHBEARER Token endpoint URL to a secure location. This is the endpoint that you’ll use with your service account credentials to authenticate the connection to this Kafka instance.

    SASL/PLAIN authentication is also available for tools and libraries that don’t support SASL/OAUTHBEARER, but SASL/OAUTHBEARER is recommended whenever possible. With SASL/PLAIN authentication, you use only the service account credentials to authenticate the connection to the Kafka instance.

    You’ll use the service account information that you saved to configure your application to connect to your Kafka instances when you’re ready. For example, if you plan to use Kafkacat to interact with your Kafka instance, you’ll use this information to set your bootstrap server and client environment variables.

  8. To review your service account information, reset your credentials, or delete the service account, use the left navigation menu to go to the Service Accounts page.

Verification
  1. Verify that the bootstrap server, client credentials, and authentication token endpoint are saved to a secure location.

  2. Verify that your service account was successfully created in the Service Accounts page.

Setting permissions for a service account in a Kafka instance in OpenShift Streams for Apache Kafka

After you create a service account to connect to a Kafka instance, you must also set the appropriate level of access for that new account in the Access Control List (ACL) of the Kafka instance. OpenShift Streams for Apache Kafka uses ACLs provided by Kafka that enable you to manage how other user accounts and service accounts are permitted to interact with the Kafka resources that you create.

Prerequisites
  • You’ve created a Kafka instance and the instance is in Ready state.

  • You’ve created a service account that you want to allow to access the running Kafka instance.

Procedure
  1. In the Kafka Instances page of the web console, click the name of the Kafka instance that you want the service account to access.

  2. Click the Access tab to view the current ACL for this instance.

  3. Click Manage access, use the Account drop-down menu to select the service account that you previously created, and click Next.

  4. Under Assign Permissions, use the drop-down menus to set the permissions shown in the following table for this service account. Click Add to add each new resource permission.

    These permissions enable applications associated with the service account to create and delete topics in the instance, to produce and consume messages in any topic in the instance, and to use any consumer group and any producer.

    Table 1. Example ACL permissions for a new service account

    Resource type

    Resource identifier and value

    Access type

    Operation

    Topic

    Is = *

    Allow

    All

    Consumer group

    Is = *

    Allow

    Read

    Transactional ID

    Is = *

    Allow

    All

  5. After you add these permissions for the service account, click Save to finish.

Verification
  • Verify that the new permissions for the service account are listed in the Access page of the Kafka instance.

Additional resources

Creating a Kafka topic in OpenShift Streams for Apache Kafka

After you create a Kafka instance, you can create Kafka topics to start producing and consuming messages in your services.

Prerequisites
  • You’ve created a Kafka instance and the instance is in Ready state.

Procedure
  1. In the Kafka Instances page of the web console, click the name of the Kafka instance that you want to add a topic to.

  2. Select the Topics tab, click Create topic, and follow the guided steps to define the topic details. Click Next to complete each step and click Finish to complete the setup.

    Image of wizard to create a topic
    Figure 3. Guided steps to define topic details
    • Topic name: Enter a unique topic name, such as my-first-kafka-topic.

    • Partitions: Set the number of partitions for this topic. This example sets the partition to 1 for a single partition. Partitions are distinct lists of messages within a topic and enable parts of a topic to be distributed over multiple brokers in the cluster. A topic can contain one or more partitions, enabling producer and consumer loads to be scaled.

    • Message retention: Set the message retention time and size to the relevant value and increment. This example sets the retention time to A week and the retention size to Unlimited. Message retention time is the amount of time that messages are retained in a topic before they are deleted or compacted, depending on the cleanup policy. Retention size is the maximum total size of all log segments in a partition before they are deleted or compacted.

    • Replicas: For this release of OpenShift Streams for Apache Kafka, the replicas are preconfigured. The number of partition replicas for the topic is set to 3 and the minimum number of follower replicas that must be in sync with a partition leader is set to 2. Replicas are copies of partitions in a topic. Partition replicas are distributed over multiple brokers in the cluster to ensure topic availability if a broker fails. When a follower replica is in sync with a partition leader, the follower replica can become the new partition leader if needed.

    After you complete the topic setup, the new Kafka topic is listed in the topics table. You can now start producing and consuming messages to and from this topic using services that you connect to this instance.

  3. In the topics table, on the right side of the Kafka topic, use the options icon (three vertical dots) to edit or delete the topic as needed.

    Image of topic options to edit or delete
    Figure 4. Edit or delete Kafka topic
Verification
  • Verify that the new Kafka topic is listed in the topics table.

Viewing metrics for a Kafka instance in OpenShift Streams for Apache Kafka

After you create Kafka topics in your Kafka instance, you can start producing and consuming messages using methods such as Kafka scripts, Kafkacat, or a Quarkus application.

After you produce and consume messages in your services, you can return to the Kafka instance in the web console and use the Dashboard page to view metrics for the instance and topics. The metrics help you understand the performance and data usage for your Kafka instance and topics.

Prerequisites
  • You’ve created a Kafka instance and the instance is in Ready state.

  • You’ve created topics in your Kafka instance.

Procedure
  • In the Kafka Instances page of the web console, click the name of the Kafka instance and select the Dashboard tab.

    When you create your Kafka instance and topics, the Dashboard page is initially empty. After you start producing and consuming messages in your services, you can return to this page to view related metrics.

    In some cases, after you start producing and consuming messages, you might need to wait several minutes for the latest metrics to appear. You might also need to wait until your instance and topics contain enough data for metrics to appear.

    After your topics contain data, the following metrics are available:

    • Kafka instance metrics

      • Used disk space: Used disk space is the amount of disk space used by the Kafka brokers in the instance. This metric enables you to assess available disk space relative to the limit. To reduce used disk space, you can adjust topic retention time or other topic properties as needed.

    • Topic metrics

      • Total bytes: Total bytes is the total incoming and outgoing bytes for all topics or for a selected topic in the Kafka instance. This metric enables you to assess data transfer in and out of your Kafka instance. To modify incoming and outgoing bytes, you can adjust topic message size or other topic properties as needed.

Verification
  • Verify that you know how to navigate to the instance Dashboard page to view metrics later.

Additional resources