Red Hat OpenShift Streams for Apache Kafka Service Limits

Updated -

Resource limits are enforced across the Red Hat OpenShift Streams for Apache Kafka service. Use of the service that exceeds these limits is subject to corrective action from Red Hat. For more information, see the following terms and conditions.

Red Hat recommends that you use the monitor API to alert you when you have exceeded your service limits and your applications are being throttled. If you get an alert, follow the guidelines shown in the service limits table for production instances.

To understand how OpenShift Streams for Apache Kafka throttles applications that exceed the service limits, see Red Hat OpenShift Streams for Apache Kafka throttles Kafka producers.

Streaming unit limits

In OpenShift Streams for Apache Kafka, you can create Kafka instances with sizes of 1 or 2 streaming units. An instance with a larger size can handle higher loads and process more events has more storage and can handle more clients and connections. Kafka instances larger than 1 streaming unit are available only as a Technology Preview at this time.

For a given streaming unit value, the capacity is enforced by limits on various dimensions of the Kafka instance. Some of these limits can be increased by contacting Red Hat, some are fixed limits and cannot be exceeded, and some are dynamic and dependent on the usage level of other dimensions. You can achieve the maximum capacity for a dimension by optimizing your application for that dimension and not exceeding the limits for other dimensions.

Production instance service limits

Your chosen Kafka instance size imposes limits on the dimensions shown in the following table.

Table 1. Service limits per Kafka instance size

Dimension 1 Streaming Unit Limit 2 Streaming Units Limit Summary
Ingress Up to 50 MiB/s Up to 100 MiB/s The maximum rate at which producers can send data to the Kafka instance. This is a combined ingress throughput limit for the whole Kafka instance and includes the protocol overhead. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit results in the blocking of your producers for a short period of time.
Egress Up to 100 MiB/s Up to 200 MiB/s The maximum rate at which consumers can read data from the Kafka instance. This is a combined egress throughput limit for the whole Kafka instance and includes the protocol overhead. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit results in the blocking of your consumers for a short period of time.
Storage Up to 1000 GiB Up to 2000 GiB The post-replication storage limit for the whole Kafka instance. The storage is evenly divided among the brokers. Open Shift Streams for Apache Kafka currently enforces limits on storage usage through the Kafka quota plug-in. If any broker reaches its storage limit, then the quota plug-in throttles all publishers, even those publishing to topics on brokers that have remaining disk space. It’s important to partition sensibly so OpenShift Streams for Apache Kafka has the ability to spread those partitions throughout the cluster.

Exceeding the storage limit means that your Kafka instance gradually throttles producers and eventually fully prevents them from producing messages. You need to free up some storage or expand the storage size to automatically unblock producers. Red Hat monitoring (such as Canary) and workload balancing systems consume small amounts of capacity. For more information about workload balancing, see the Red Hat OpenShift Streams for Apache Kafka Service Definition. You can check your current storage usage and monitor available storage across each broker by monitoring disk space metrics. One way to control storage use is to configure topic retention options. You can also request storage expansion for your Kafka instance by contacting Red Hat Support.
Partitions Up to 1500 Up to 3000 The maximum number of partitions you can have in a single Kafka instance. Creating a larger number of partitions will have a negative impact on latency. This is a pre-replication count that does not include follower replicas partitions.
Client connections Up to 3000 Up to 6000 The total number of TCP connections that can be open to a Kafka instance. This number also includes the connections from our internal monitoring system which is approximately 3 times the number of brokers. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit means you can’t make new client connections.

A client application opens multiple connections to a Kafka instance (typically one data connection per broker and one additional metadata connection to a random broker). The actual number of connections varies between the different usage patterns. To avoid reaching this limit, reduce the number of clients connected to the same Kafka instance. You can check your current client connections usage by monitoring cluster metrics.
Connection attempts per second Up to 100 Up to 200 The rate of successful and unsuccessful authentication attempts to the Kafka instance. This limit is divided evenly among all brokers and enforced per broker. Exceeding this limit means you can’t make new connections. To avoid reaching this limit, increase the client retry interval through the reconnect.backoff.ms configuration. You can check your current client connections usage by monitoring cluster metrics.
Maximum message size Up to 1 MiB Up to 1 MiB The maximum size of a message that can be sent to the Kafka instance.
Service accounts per organization 50 50 The maximum number of service accounts per organization. This limit is shared among all service accounts within the organization.
Availability zone Multi Multi All standard instances use multi-availability zones.

Trial instance service limits

The limits that apply to trial instances in OpenShift Streams for Apache Kafka are shown in the following table.

Table 2. Service limits per Kafka trial instance

Dimension Limit
Trial instance duration 48 hours
Trial instances per user 1
Availability zone Single
Topic replicas 1
Ingress Up to 1 MiB/s
Egress Up to 1 MiB/s
Storage Up to 10 GiB
Partitions Up to 100
Client connections Up to 100
Connection rate Up to 50 connections/s
Message size Up to 1 MiB