Configuring consumer groups in Red Hat OpenShift Streams for Apache Kafka

Guide
  • Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 19 October 2021
  • Published 21 July 2021

Configuring consumer groups in Red Hat OpenShift Streams for Apache Kafka

Guide
Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 19 October 2021
  • Published 21 July 2021

Red Hat OpenShift Streams for Apache Kafka is currently available for Development Preview. Development Preview releases provide early access to a limited set of features that might not be fully tested and that might change in the final GA version. Users should not use Development Preview software in production or for business-critical workloads. Limited documentation is available for Development Preview releases and is typically focused on fundamental user goals.

Consumer groups share large data streams generated by producers from a given topic in a Kafka instance. Grouping consumers scales consumption to keep up with the rate of data produced. Consumers within a group don’t read data from the same partition, but can read data from one or more partitions.

As a developer of applications and services, you can view all the consumer groups that have access to a particular Kafka instance in OpenShift Streams for Apache Kafka. If required, use the OpenShift Streams for Apache Kafka web console to reset the offsets of consumer groups or delete consumer groups.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) or the Kafka kafka-consumer-groups.sh script to review and update consumer groups.

Connecting the Kafka consumer group script

If you’re using the kafka-consumer-groups.sh script, or any other Kafka scripts, you’ll need to use the --bootstrap-server and --command-config flags to connect to your Kafka instance.

Command to list all consumer groups
./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --list

The <bootstrap_server> is the bootstrap server endpoint for your Kafka instance. You copy this information for the Kafka instance in the OpenShift Streams for Apache Kafka web console by selecting the options menu (three vertical dots) and clicking Connection.

Kafka scripts require SASL/PLAIN authentication. Use an <authentication_properties> configuration file to configure the connection. Include the client ID and client secret generated when creating a service account to access your Kafka instance.

Authentication configuration properties
sasl.mechanism=PLAIN
security.protocol=SASL_SSL

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="<client_id>" \
  password="<client_secret>" \;
Additional resources

Adding a consumer to a consumer group

A group ID is required for a consumer to be able to join a consumer group. The group ID is the name of the consumer group. When you configure a consumer client application to access Kafka, you assign a group ID to associate it with a consumer group. All consumers with the same group ID belong to the consumer group with that ID. Consumer groups connected to a Kafka instance are displayed in the OpenShift Streams for Apache Kafka web console.

Procedure
  • Add the consumer group ID as a property to your consumer configuration. The group ID can’t be null or empty.

    Example group ID property configuration for a consumer
    group.id=my-group-id
    Table 1. Property attributes

    Type

    String

    Default value

    None

    Supported values

    Letters (Aa-Zz), numbers (0-9), underscores ( _ ), or hyphens ( - )

    If you’re using Kafka scripts to consume messages, you can use the kafka-console-consumer.sh tool to create a group ID:

    ./kafka-console-consumer.sh --bootstrap-server <bootstrap_server> --consumer.config <authentication_properties> --topic test-topic --group my-consumer-group

    The new consumer group is named my-consumer-group.

    If you add more consumers than the number of partitions in a consumer group, the additional consumers don’t help throughput but they function as standby consumers that can replace any consumers that stop functioning. The fewer consumers you use in order to meet your throughput target, the more resources you save.

Reviewing consumer group properties in OpenShift Streams for Apache Kafka

Use the OpenShift Streams for Apache Kafka web console to select a consumer group and review the consumer properties. You can look at the consumer groups associated with a Kafka instance or a specific topic of the instance.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to return information about a consumer group, as shown in the following example command:

Example CLI command to return consumer group information
rhoas kafka consumer-group describe [flags]

For a list of consumer group commands that you can use with the CLI, see the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to provide the same information:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --describe --group my-consumer-group
The Kafka scripts are part of the open source community version of Apache Kafka. The scripts are not a part of OpenShift Streams for Apache Kafka and are therefore not supported by Red Hat.
Prerequisites
  • You’re logged in to the OpenShift Streams for Apache Kafka web console.

  • You’ve created a Kafka instance with at least one Kafka topic in OpenShift Streams for Apache Kafka.

  • Consumer client applications connected to the Kafka instance have a consumer group ID.

Procedure
  1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the consumer groups that you want to review.

    Alternatively, select a consumer group for a specific topic.

    1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the topic.

    2. In the Topics page, click the name of the topic.

  2. In the Consumer groups page, click the name of the consumer group that you want to review.

    Alternatively, select the options icon (three vertical dots) for the relevant consumer group and click View partition offsets.

Consumer group properties in OpenShift Streams for Apache Kafka

The following consumer group properties are displayed in OpenShift Streams for Apache Kafka. Consumer group properties are used for monitoring in the OpenShift Streams for Apache Kafka web console and are not editable.

Consumer groups

Consumer Group ID
The consumer group ID is the unique identifier for the consumer group within the cluster. This is part of the consumer configuration for the application client.
Active Members
Active members shows the number of consumers in the group that are assigned to a topic partition in the Kafka instance. If you’re viewing information on consumer groups for a topic, these are the active members for the topic.
Partitions with lag
Partitions with lag shows the number of partitions where the assigned consumer has not caught up with the last message in the partition. The lag reflects the position of the consumer offset in relation to the end of the partition log.

Consumer offset positions

When you click a consumer group, you see the details of each member.

Partition
The partition number for the topic.
Client ID
The unique ID of the client application used to identify active consumers. If no client ID is shown, the partition is not currently being consumed.
Current offset
The current offset number for the consumer in the partition log. This is the position of the consumer in the partition and the latest read position.
Log end offset
The current offset number for the producer in the partition log. This is the end of the log and the latest write position.
Offset lag
The difference (delta) between the consumer and producer offset positions in the log.

Consumer lag indicators

Consumer lag for a given consumer group indicates the delay between the last message added in a partition and the message currently being picked up by that consumer. The lag reflects the position of the consumer offset in relation to the end of the partition log.

When you’re reviewing consumer group properties in the OpenShift Streams for Apache Kafka web console, look for the differences between Current offset and Log end offset. The difference shows as the Offset lag value.

For applications that rely on the processing of (near) real-time data, it’s critical that consumer lag doesn’t become too big. Suppose a topic streams 100 messages per second. A lag of 1000 messages between the producer offset (the topic partition head) and the last (current) offset that the consumer has read means a 10-second delay. The offset lag shows that a gap is opening up between the write and read positions.

To reduce lag, you typically add new consumers to a group. However, you can also increase the retention time for a message to remain in a topic. Extending the retention of data in the log gives the consumer a chance to catch up before data is flushed from the message log.

For more information about increasing topic retention time, see Configuring topics in OpenShift Streams for Apache Kafka.

Resetting consumer group offset positions

Use the OpenShift Streams for Apache Kafka web console to select consumer groups and reset partition offsets for a particular topic. A reset changes the offset position from which consumers read from the message log of a topic partition. To reset an offset position, the consumer group must have NO MEMBERS connected to a topic.

Choose one of the following options for the reset:

  • Earliest resets to the earliest offset at the start of the message log.

  • Latest resets to the latest offset at the end of the message log.

  • Custom offset resets to a specific offset in the message log.

By resetting the offset position you risk clients skipping or duplicating messages.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to reset consumer group offsets, as shown in the following example command:

Example CLI command to reset offsets for consumer groups
rhoas kafka consumer-group reset-offset --id my-consumer-group --offset earliest --topic topic1

The reset-offset CLI command has an additional reset option. You can use a timestamp value.

rhoas kafka consumer-group reset-offset --id my-consumer-group --offset timestamp --value "2021-06-23T09:07:21-07:00"

For a list of topic properties that you can update using the CLI, see the rhoas kafka topic update entry in the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to reset offsets:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --reset-offsets --group my-consumer-group --topic topic1 --to-latest

You can specify the reset for --all-topics or a single specified --topic.

Prerequisites
  • You’re logged in to the OpenShift Streams for Apache Kafka web console.

  • The consumer group you select must have no active members connected to the topic.

  • Consumers in the consumer group must be shut down (not consuming partitions).

Procedure
  1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the consumer group you’re updating.

    Alternatively, select a consumer group for a specific topic.

    1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the topic.

    2. In the Topics page, click the name of the topic.

  2. In the Consumer groups page, select the options icon (three vertical dots) for the relevant consumer group and click Reset Offset.

  3. Select a topic.

  4. Choose a new offset position from Absolute, Latest, or Earliest.

  5. Select one or more partitions to apply the offset reset.

  6. If you chose an absolute reset, enter the new custom offset number for the reset.

  7. Click Reset offset to finish.

Deleting a consumer group

Use the OpenShift Streams for Apache Kafka web console to delete consumer groups. The consumer group must have no active members connected to a topic. By deleting the consumer group, you remove the current state associated with the group.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to delete consumer groups, as shown in the following example command:

Example CLI command to delete a consumer group
rhoas kafka consumer-group delete my-consumer-group

For a list of topic properties that you can update using the CLI, see the rhoas kafka topic update entry in the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to delete consumer groups:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --delete --group my-consumer-group
Prerequisites
  • You’re logged in to the OpenShift Streams for Apache Kafka web console.

  • The consumer group you select must have no active members.

  • Consumers in the consumer group must be shut down (not consuming partitions).

Procedure
  1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the consumer group you’re updating.

    Alternatively, select a consumer group for a specific topic.

    1. In the OpenShift Streams for Apache Kafka web console, go to Streams for Apache Kafka > Kafka Instances and click the name of the Kafka instance that contains the topic.

    2. In the Topics page, click the name of the topic.

  2. In the Consumer groups page, select the options icon (three vertical dots) for the relevant consumer group and click Delete.

  3. Confirm the deletion by clicking Delete.