Red Hat OpenShift Streams for Apache Kafka Service Limits
Resource limits are enforced across the Red Hat OpenShift Streams for Apache Kafka service. Use of the service that exceeds these limits is subject to corrective action from Red Hat. For more information, see the following terms and conditions.
Red Hat recommends that you use the monitor API to alert you when you have exceeded your service limits and your applications are being throttled. If you get an alert, follow the guidelines shown in the service limits table for production instances.
To understand how OpenShift Streams for Apache Kafka throttles applications that exceed the service limits, see Red Hat OpenShift Streams for Apache Kafka throttles Kafka producers.
Streaming unit limits
In OpenShift Streams for Apache Kafka, you can create Kafka instances with sizes of 1 or 2 streaming units. An instance with a larger size can handle higher loads and process more events has more storage and can handle more clients and connections. Kafka instances larger than 1 streaming unit are available only as a Technology Preview at this time.
For a given streaming unit value, the capacity is enforced by limits on various dimensions of the Kafka instance. Some of these limits can be increased by contacting Red Hat, some are fixed limits and cannot be exceeded, and some are dynamic and dependent on the usage level of other dimensions. You can achieve the maximum capacity for a dimension by optimizing your application for that dimension and not exceeding the limits for other dimensions.
Production instance service limits
Your chosen Kafka instance size imposes limits on the dimensions shown in the following table.
Table 1. Service limits per Kafka instance size
| Dimension | 1 Streaming Unit Limit | 2 Streaming Units Limit | Summary |
|---|---|---|---|
| Ingress | Up to 50 MiB/s | Up to 100 MiB/s | The maximum rate at which producers can send data to the Kafka instance. This is a combined ingress throughput limit for the whole Kafka instance and includes the protocol overhead. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit results in the blocking of your producers for a short period of time. |
| Egress | Up to 100 MiB/s | Up to 200 MiB/s | The maximum rate at which consumers can read data from the Kafka instance. This is a combined egress throughput limit for the whole Kafka instance and includes the protocol overhead. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit results in the blocking of your consumers for a short period of time. |
| Storage | Up to 1000 GiB | Up to 2000 GiB | The post-replication storage limit for the whole Kafka instance. The storage is evenly divided among the brokers. Open Shift Streams for Apache Kafka currently enforces limits on storage usage through the Kafka quota plug-in. If any broker reaches its storage limit, then the quota plug-in throttles all publishers, even those publishing to topics on brokers that have remaining disk space. It’s important to partition sensibly so OpenShift Streams for Apache Kafka has the ability to spread those partitions throughout the cluster. Exceeding the storage limit means that your Kafka instance gradually throttles producers and eventually fully prevents them from producing messages. You need to free up some storage or expand the storage size to automatically unblock producers. Red Hat monitoring (such as Canary) and workload balancing systems consume small amounts of capacity. For more information about workload balancing, see the Red Hat OpenShift Streams for Apache Kafka Service Definition. You can check your current storage usage and monitor available storage across each broker by monitoring disk space metrics. One way to control storage use is to configure topic retention options. You can also request storage expansion for your Kafka instance by contacting Red Hat Support. |
| Partitions | Up to 1500 | Up to 3000 | The maximum number of partitions you can have in a single Kafka instance. Creating a larger number of partitions will have a negative impact on latency. This is a pre-replication count that does not include follower replicas partitions. |
| Client connections | Up to 3000 | Up to 6000 | The total number of TCP connections that can be open to a Kafka instance. This number also includes the connections from our internal monitoring system which is approximately 3 times the number of brokers. This limit is evenly distributed among all brokers and enforced per broker. Exceeding this limit means you can’t make new client connections. A client application opens multiple connections to a Kafka instance (typically one data connection per broker and one additional metadata connection to a random broker). The actual number of connections varies between the different usage patterns. To avoid reaching this limit, reduce the number of clients connected to the same Kafka instance. You can check your current client connections usage by monitoring cluster metrics. |
| Connection attempts per second | Up to 100 | Up to 200 | The rate of successful and unsuccessful authentication attempts to the Kafka instance. This limit is divided evenly among all brokers and enforced per broker. Exceeding this limit means you can’t make new connections. To avoid reaching this limit, increase the client retry interval through the reconnect.backoff.ms configuration. You can check your current client connections usage by monitoring cluster metrics. |
| Maximum message size | Up to 1 MiB | Up to 1 MiB | The maximum size of a message that can be sent to the Kafka instance. |
| Service accounts per organization | 50 | 50 | The maximum number of service accounts per organization. This limit is shared among all service accounts within the organization. |
| Availability zone | Multi | Multi | All standard instances use multi-availability zones. |
Trial instance service limits
The limits that apply to trial instances in OpenShift Streams for Apache Kafka are shown in the following table.
Table 2. Service limits per Kafka trial instance
| Dimension | Limit |
|---|---|
| Trial instance duration | 48 hours |
| Trial instances per user | 1 |
| Availability zone | Single |
| Topic replicas | 1 |
| Ingress | Up to 1 MiB/s |
| Egress | Up to 1 MiB/s |
| Storage | Up to 10 GiB |
| Partitions | Up to 100 |
| Client connections | Up to 100 |
| Connection rate | Up to 50 connections/s |
| Message size | Up to 1 MiB |