Configuring consumer groups in Red Hat OpenShift Streams for Apache Kafka

Guide
  • Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 22 February 2023
  • Published 21 July 2021

Configuring consumer groups in Red Hat OpenShift Streams for Apache Kafka

Guide
Red Hat OpenShift Streams for Apache Kafka 1
  • Updated 22 February 2023
  • Published 21 July 2021

Consumer groups share large data streams generated by producers from a given topic in a Kafka instance. Grouping consumers scales consumption to keep up with the rate of data produced. Consumers within a group don’t read data from the same partition, but can read data from one or more partitions.

As a developer of applications and services, you can view all the consumer groups that have access to a particular Kafka instance in Red Hat OpenShift Streams for Apache Kafka. If required, use the OpenShift Streams for Apache Kafka web console to reset the offsets of consumer groups or delete consumer groups.

Listing consumer groups

In addition to the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) or the Kafka kafka-consumer-groups.sh script to list consumer groups for your Kafka instance. The following subsections describe how to use these methods to list consumer groups.

The Kafka scripts are part of the open source community version of Apache Kafka. The scripts are not a part of OpenShift Streams for Apache Kafka and are therefore not supported by Red Hat.

Listing consumer groups using the CLI

To use the rhoas command-line interface (CLI) to list the consumer groups defined for your Kafka instance, enter the following command:

rhoas kafka consumer-group list

You now see output similar to the following example:

Consumer group ID    Active members    Partitions with lag    State
-------------------  ----------------  ---------------------  -------
consumergroup1                     2                     0    Empty
consumergroup2                     1                     0    Stable

Consumer group states

Consumer group states displayed in the OpenShift Streams for Apache Kafka web console and rhoas CLI can be one of the following values:

  • Empty: The group exists but has no members.

  • Stable: Rebalancing already occurred and consumers are consuming.

  • PreparingRebalance: The group is waiting for consumers to join, which requires the reassignment of partitions, so the Kafka instance is currently rebalancing.

  • CompletingRebalance: The Kafka instance is still rebalancing and reassigning partitions.

  • Dead: The group is due to be removed from this Kafka instance soon. This deletion might be due to inactivity, or the group is being migrated to a different group coordinator.

  • Unknown: The state is too new for it to be parsed.

Connecting the Kafka consumer group script

You can list consumer groups using the kafka-consumer-groups.sh script. To use this or any other Kafka scripts, you need to specify the --bootstrap-server and --command-config flags to connect to your Kafka instance. To list consumer groups, enter the following command:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --list

The <bootstrap_server> is the bootstrap server endpoint for your Kafka instance. You copy this information for the Kafka instance in the OpenShift Streams for Apache Kafka web console by selecting the options menu (three vertical dots) and clicking Connection.

Kafka scripts require SASL/PLAIN authentication. Use an <authentication_properties> configuration file to configure the connection. Include the client ID and client secret generated when creating a service account to access your Kafka instance.

Authentication configuration properties
sasl.mechanism=PLAIN
security.protocol=SASL_SSL

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
  username="<client_id>" \
  password="<client_secret>" \;
Additional resources

Adding a consumer to a consumer group

To create a consumer group, you specify a consumer group ID in your consumer application. You do not create consumer groups in the OpenShift Streams for Apache Kafka web console, or using the CLI.

A group ID is required for a consumer to be able to join a consumer group. The group ID is the name of the consumer group. When you configure a consumer client application to access a Kafka instance, you assign a group ID to associate it with a consumer group. All consumers with the same group ID belong to the consumer group with that ID.

Consumer groups connected to a Kafka instance are displayed in the OpenShift Streams for Apache Kafka web console.

Procedure
  • Add the consumer group ID as a property to your consumer configuration. The group ID can’t be null or empty.

    Example group ID property configuration for a consumer
    group.id=my-group-id
    Table 1. Property attributes

    Type

    String

    Default value

    None

    Supported values

    Letters (Aa-Zz), numbers (0-9), underscores ( _ ), or hyphens ( - )

    If you’re using Kafka scripts to consume messages, you can use the kafka-console-consumer.sh tool to create a group ID:

    ./kafka-console-consumer.sh --bootstrap-server <bootstrap_server> --consumer.config <authentication_properties> --topic test-topic --group my-consumer-group

    The new consumer group is named my-consumer-group.

    If you add more consumers than the number of partitions in a consumer group, the additional consumers don’t help throughput but they function as standby consumers that can replace any consumers that stop functioning. The fewer consumers you use in order to meet your throughput target, the more resources you save.

Reviewing consumer group properties in OpenShift Streams for Apache Kafka

Use the OpenShift Streams for Apache Kafka web console to select a consumer group and review the consumer properties. You can examine the consumer groups associated with a Kafka instance or a specific topic of the instance.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to return information about a consumer group, as shown in the following example command:

rhoas kafka consumer-group describe [flags]

For a list of consumer group commands that you can use with the CLI, see the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to provide the same information:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --describe --group my-consumer-group
Prerequisites
Procedure
  1. On the Kafka Instances page of the OpenShift Streams for Apache Kafka web console, click the name of the Kafka instance that contains the consumer groups that you want to review.

  2. Select the Consumer groups page.

  3. Select the options icon (three vertical dots) for the relevant consumer group and click View partition offsets.

You can also view consumer groups for a specific topic. In your Kafka instance, select the Topics page and then click the name of a topic. Select the Consumer groups page. For a given consumer group, select the options icon (three vertical dots) and click View partition offsets.

Consumer group properties in OpenShift Streams for Apache Kafka

The following consumer group properties are displayed in Red Hat OpenShift Streams for Apache Kafka. Consumer group properties are used for monitoring in the OpenShift Streams for Apache Kafka web console and are not editable.

Consumer groups

Consumer Group ID
The consumer group ID is the unique identifier for the consumer group within the cluster. This is part of the consumer configuration for the application client.
Active Members
Active members shows the number of consumers in the group that are assigned to a topic partition in the Kafka instance. If you’re viewing information about consumer groups for a topic, these are the active members for the topic.
Partitions with lag
Partitions with lag shows the number of partitions where the assigned consumer has not caught up with the last message in the partition. The lag reflects the position of the consumer offset in relation to the end of the partition log.

Consumer offset positions

When you click a consumer group, you see the details of each member.

Partition
The partition number for the topic.
Client ID
The unique ID of the client application used to identify active consumers. If no client ID is shown, the partition is not currently being consumed.
Current offset
The current offset number for the consumer in the partition log. This is the position of the consumer in the partition and the latest read position.
Log end offset
The current offset number for the producer in the partition log. This is the end of the log and the latest write position.
Offset lag
The difference (delta) between the consumer and producer offset positions in the log.

Consumer lag indicators

Consumer lag for a given consumer group indicates the delay between the last message added in a partition and the message currently being picked up by that consumer. The lag reflects the position of the consumer offset in relation to the end of the partition log.

When you’re reviewing consumer group properties in the OpenShift Streams for Apache Kafka web console, look for the differences between Current offset and Log end offset. The difference shows as the Offset lag value.

For applications that rely on the processing of (near) real-time data, it’s critical that consumer lag doesn’t become too big. Suppose a topic streams 100 messages per second. A lag of 1000 messages between the producer offset (the topic partition head) and the last (current) offset that the consumer has read means a 10-second delay. The offset lag shows that a gap is opening up between the write and read positions.

To reduce lag, you typically add new consumers to a group. However, you can also increase the retention time for a message to remain in a topic. Extending the retention of data in the log gives the consumer a chance to catch up before data is flushed from the message log.

For more information about increasing topic retention time, see Configuring topics in OpenShift Streams for Apache Kafka.

Resetting consumer group offset positions

Use the OpenShift Streams for Apache Kafka web console to select consumer groups and reset partition offsets for a particular topic. A reset changes the offset position from which consumers read from the message log of a topic partition. To reset an offset position, the consumer group must have NO MEMBERS connected to a topic.

Choose one of the following options for New offset:

  • absolute resets to a specific offset in the message log.

  • latest resets to the latest offset at the end of the message log.

  • earliest resets to the earliest offset at the start of the message log.

By resetting the offset position you risk clients skipping or duplicating messages.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to reset consumer group offsets, as shown in the following example command:

Example CLI command to reset offsets for consumer groups
rhoas kafka consumer-group reset-offset --id my-consumer-group --offset earliest --topic topic1

The reset-offset CLI command has an additional reset option. You can use a timestamp value.

rhoas kafka consumer-group reset-offset --id my-consumer-group --offset timestamp --value "2021-06-23T09:07:21-07:00"

For a list of topic properties that you can update using the CLI, see the rhoas kafka topic update entry in the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to reset offsets:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --reset-offsets --group my-consumer-group --topic topic1 --to-latest

You can specify the reset for --all-topics or a single specified --topic.

Prerequisites
  • The consumer group you select must have no active members connected to the topic.

  • Consumers in the consumer group must be shut down (not consuming partitions).

Procedure
  1. On the Kafka Instances page of the OpenShift Streams for Apache Kafka web console, click the name of the Kafka instance that contains the consumer group you’re updating.

    Alternatively, select a consumer group for a specific topic.

    1. On the Kafka Instances page of the OpenShift Streams for Apache Kafka web console, click the name of the Kafka instance that contains the topic.

    2. On the Topics page, click the name of the topic.

  2. On the Consumer groups page, select the options icon (three vertical dots) for the relevant consumer group and click Reset Offset.

  3. Select a topic.

  4. Choose a new offset position from Absolute, Latest, or Earliest.

  5. Select one or more partitions to apply the offset reset.

  6. If you chose an absolute reset, enter the new custom offset number for the reset.

  7. Click Reset offset to finish.

Deleting a consumer group

Use the OpenShift Streams for Apache Kafka web console to delete consumer groups. The consumer group must have no active members connected to a topic. By deleting the consumer group, you remove the current state associated with the group.

As an alternative to using the OpenShift Streams for Apache Kafka web console, you can use the rhoas command-line interface (CLI) to delete consumer groups, as shown in the following example command:

Example CLI command to delete a consumer group
rhoas kafka consumer-group delete my-consumer-group

For a list of topic properties that you can update using the CLI, see the rhoas kafka topic update entry in the CLI command reference (rhoas).

If you’re using Kafka scripts, you can use the kafka-consumer-groups.sh tool to delete consumer groups:

./kafka-consumer-groups.sh --bootstrap-server <bootstrap_server> --command-config <authentication_properties> --delete --group my-consumer-group
Prerequisites
  • The consumer group you select must have no active members.

  • Consumers in the consumer group must be shut down (not consuming partitions).

Procedure
  1. On the Kafka Instances page of the OpenShift Streams for Apache Kafka web console, click the name of the Kafka instance that contains the consumer group you’re updating.

    Alternatively, select a consumer group for a specific topic.

    1. On the Kafka Instances page of the OpenShift Streams for Apache Kafka web console, click the name of the Kafka instance that contains the topic.

    2. On the Topics page, click the name of the topic.

  2. On the Consumer groups page, select the options icon (three vertical dots) for the relevant consumer group and click Delete.

  3. Confirm the deletion by clicking Delete.