Red Hat OpenShift Streams for Apache Kafka Service Definition

Updated -

Introduction

Red Hat OpenShift Streams for Apache Kafka is provided as a cloud service. The underlying platform is provisioned, operated, managed, and maintained by Red Hat.

OpenShift Streams for Apache Kafka runs on Red Hat’s OpenShift platform. The service provides a production-grade managed streams processing platform for enterprises and organizations. Based on hybrid cloud, the service is highly available and scalable.

Deployment models

OpenShift Streams for Apache Kafka supports the deployment of Kafka instances on OpenShift instances spanning multiple regions and cloud providers. These OpenShift instances are administered and accessible only to Red Hat personnel.

Customers access OpenShift Streams for Apache Kafka as a service through public interfaces. The interfaces sit above the OpenShift layer. Red Hat provides the following components for provisioning and monitoring tasks:

The OpenShift Streams for Apache Kafka service has a control plane and data plane. The control plane comprises multi-tenant components that are available to all OpenShift Streams for Apache Kafka customers. The data plane comprises single-tenant Kafka instances that are not shared with any other customers of the service.

Control plane elements

  • Identity and access management
  • Metrics and monitoring
  • User interface
  • Service API

Data plane elements

  • Kafka brokers
  • ZooKeeper
  • Topics and topic partitions
  • Consumer groups

Cloud provider and region availability

OpenShift Streams for Apache Kafka is currently available for the following cloud providers and regions:

  • Amazon Web Services (AWS)
    • us-east-1
    • eu-west-1
    • Additional regions by request

Cluster visibility and secure access

All Kafka clusters created through the OpenShift Streams for Apache Kafka service are provisioned for secure access. Endpoints are TLS-enabled. External access is subject to authentication and authorization restrictions, which are configurable through the service’s identity and access management component.

Guidance on environments

Red Hat recommends that customers use a separate Kafka instance for each of their environments. For example, a customer can use a separate Kafka instance for their development, stage, and production environments. Customers can also use automated build and deploy pipelines to migrate between these environments.

Service configuration

Kafka instances created through OpenShift Streams for Apache Kafka are optimized for high availability and reliability. The following default configurations are not modifiable by users:

  • Each Kafka instance consists of a number of brokers, with instances distributed across a minimum of three AZs (Availability Zones) in the target region.
  • All topics have a replication factor of 3, with partition replicas placed in separate AZs.
  • All topics have a minimum in-sync replica count of 2.

The ZooKeeper environment supporting Kafka instances is completely managed by Red Hat and not accessible to users.

Operations related to provisioning and administering resources in the service are supported through the OpenShift Streams for Apache Kafka control plane interfaces (user interface, API, and CLI). Resource operations available through the Kafka protocol are restricted to prevent duplication or inappropriate modifications to Kafka instances. For example, it’s not possible to alter broker configuration. These restrictions do not limit data plane access, for example, when producing and consuming messages.

For a complete description of the operations available with the OpenShift Streams for Apache Kafka components, see the product documentation.

OpenShift Service Registry

Red Hat OpenShift Service Registry is available as a freely entitled service to the users of the following services:

OpenShift Service Registry with OpenShift Streams for Apache Kafka allows users to store and manage the schemas used by their Kafka producers and consumers. OpenShift Service Registry also helps to reduce message size and improve overall performance. By storing the schemas in OpenShift Service Registry and passing only a schema ID rather than the whole schema in every message, the message size is reduced. For more details, see the Red Hat OpenShift Service Registry Service Definition.

Workload balancing

A cluster imbalance situation can happen for several reasons.

  • Brokers, nodes, and storage volumes fail unexpectedly and trigger automatic recovery.
  • Planned maintenance performs rolling upgrades to brokers.
  • During the application lifecycle, new topics are created and old ones are deleted.
  • Changes in the application usage patterns shift message load distributions among topics.

These kinds of scenarios can result in the topic partition placement not matching the desired topic placement policy and lead to cluster imbalance. This cluster imbalance could prevent applications from using a Kafka Instance to its full capacity.

The workload balancing feature in OpenShift Streams continuously collects metrics about the state of the cluster and partition layout. It detects anomalies such as partition imbalance and addresses them by reassigning partitions on the available brokers, thereby rebalancing the load.

How different sized Kafka instances balance workload

In OpenShift Streams for Apache Kafka, users can specify a Kafka instance size of 1 or 2 streaming units. How these different instance sizes balance workload is explained in the subsections that follow.

1-streaming-unit Kafka instances

The data and the load on a 1-streaming-unit Kafka instance are always evenly distributed among all the brokers in the form of partitions and leaders respectively.

2-streaming-unit Kafka instances

Kafka instances comprising 2 streaming units share the load in the cluster between a larger number of individual brokers. This allocation allows the cluster to serve more requests as a whole. 2-streaming-units Kafka instances can run into imbalance if the message partitioning logic distributes data unevenly across partitions. This imbalance might result in uneven data distribution in the cluster and could reduce the ability to use Kafka to the advertised storage limits.

To prevent this imbalance, users can check the size of each partition from the OpenShift Streams for Apache Kafka console and identify unevenly distributed partitions. Alternatively, users can monitor the size of partitions and brokers from the metrics endpoint and use a platform like Prometheus and Grafana (or both) to raise alerts.

Service limits

All Kafka instances created through OpenShift Streams for Apache Kafka have predefined limits that represent the maximum capacity of a single Kafka instance.
For information about the current service limits, users can refer to Red Hat OpenShift Streams for Apache Kafka Service Limits.

Red Hat actively monitors and enforces these limits in Kafka instances. Customer usage that goes beyond these limits is subject to corrective action from Red Hat personnel, including but not limited to customer notification for correction, throttling, and service suspension.

Service limits are scoped to individual Kafka instances and not to a customer account.

Customers wishing to utilize higher resource capacity can create multiple Kafka instances using a single account and partition workload to each instance accordingly.

In the event that the limits provided are deemed insufficient, contact Red Hat Support to discuss further options.

Performance

Performance of Kafka-based applications conforms to a combination of broker and client (producer and consumer) settings. Kafka instances created through OpenShift Streams for Apache Kafka are capable of scaling to their defined service limits.

In the event that application and client performance do not meet expectations from a customer perspective, customers can contact Red Hat Support. Red Hat support can help diagnose and resolve performance issues.

Identity and access Management

The OpenShift Streams for Apache Kafka service supports authentication and authorization for users (user identity) and Kafka client applications (service accounts).

User identity

User identity is based on a customer or organization's Red Hat identity. The user identity is used to access the OpenShift Streams for Apache Kafka service user interface, APIs, and CLI.

Service accounts

Client applications access Kafka instances through an authenticated service account. A service account provides the credentials -- client ID and secret -- for a client to access a Kafka instance. OpenShift Streams for Apache Kafka components support service account creation. Creating service accounts for each client isolates and identifies client access to the Kafka instance.

OpenShift Streams for Apache Kafka supports the ability to create multiple service accounts to isolate application access to a Kafka instance.

Authentication

The service supports the SASL/OAUTHBEARER (recommended) and SASL/PLAIN (for clients that do not support SASL/OAUTHBEARER) protocols for client authentication.

Supported clients and messaging protocols

Kafka instances provided through OpenShift Streams for Apache Kafka support the Apache Kafka messaging protocol used by a wide range of Kafka clients.

Red Hat tests and validates a number of popular Kafka clients as part of the service release process.

Other Kafka-compatible client tools should also work with OpenShift Streams for Apache Kafka. Responsibility for testing and validating these clients is a customer responsibility.

For more information on supported clients, customers can refer to Supported and Compatible Configurations OpenShift Streams for Apache Kafka.

Security and compliance

The OpenShift Streams for Apache Kafka service is deployed on a fleet of OpenShift instances.

The underlying OpenShift Dedicated managed environment is certified against the ISO 2700 international information security standard. Other certifications, such as PCI and FedRAMP, are also planned. For more information on OpenShift Dedicated, customers can refer to the product documentation.

The OpenShift Streams for Apache Kafka service itself will also be certified against a number of these security and compliance protocols at a future date.

Metrics and logging

As a cloud service, OpenShift Streams for Apache Kafka collects a broad range of metrics on Kafka instances spanning multiple components.

Service metrics

Service metrics are internal only. They are used by Red Hat to provide and maintain the service at agreed levels. Service metrics are accessible to Red Hat authorized personnel only.

Customer metrics

Customer metrics are a subset of the overall metrics gathered. Metrics that are highly relevant to customers are provided directly in the OpenShift Streams for Apache Kafka user interface. They are also available through the API.

Metrics are provided for Kafka instances, topics, and consumer groups.

Cluster metrics

  • Bytes in and out per listener

  • Available disk space

  • TLS versions and cipher suites

  • Kafka client type and version

  • Connection count

  • Successful and failed authentications

Topic metrics

  • Bytes in and out per topic

  • Messages in per topic

  • Log size

  • First and last offset

  • Partition states

Consumer group metrics

  • Active members

  • Partitions with consumer lag

  • Current offset for a consumer

  • Log end offset for the partition

  • Offset lag between the producer and consumer of a partition

Retention periods

OpenShift Streams for Apache Kafka customer metrics are retained for a maximum of 7 days. At this time, there is no way to extend this period. Customers interested in storing metrics for longer time periods can integrate with the metrics REST API (/api/kafkas_mgmt/v1/kafkas/{id}/metrics/query) using third-party monitoring tools.

Service logging

System logs for all components of the OpenShift Streams for Apache Kafka service, such as Kafka brokers and ZooKeeper, are internal and available only to Red Hat personnel. Red Hat does not provide user access to component logs.

Updates and upgrades to the service

Red Hat will make a commercially reasonable effort to notify customers prior to service impacting updates and upgrades. The determination of the need for a service update and the timing thereof are the sole responsibility of Red Hat.

Customers do not have control over when a service update occurs.

Upgrades to the version of Kafka used in OpenShift Streams for Apache Kafka are considered part of a service update.

Upgrades aim to preserve overall service availability, but may temporarily impact performance.

Kafka clients have built-in compatibility across Kafka versions, so Kafka upgrades should not be service impacting events for OpenShift Streams for Apache Kafka users.

Red Hat recommends that customers align their client versions with the current version of Kafka used in the service. Validating compatibility between different client and broker versions is a customer responsibility.

Service availability

Red Hat maintains a 99.95% availability for its General Availability cloud services, including the underlying OpenShift Dedicated managed environment.

For more information, customers can refer to Appendix 4 (Online Subscription Services) of the Red Hat Enterprise Agreements and Product Appendices.

High availability and disaster recovery

Kafka instances created through OpenShift Streams for Apache Kafka use settings that replicate partitions and distribute brokers across availability zones. This configuration supports data reliability and minimizes the risk of data loss. While the risk of data loss is minimized in this architecture, data loss can still occur. For example, all availability zones used by a Kafka instance might fail at the same time.

Additional backup and recovery procedures are the responsibility of the customer.

User-initiated operations might involve the deletion of Kafka instances or topics. Recovery from any potentially destructive operations are the sole responsibility of the customer.

Resource and responsibility assignment

Although Red Hat manages the Red Hat OpenShift Streams for Apache Kafka service, the customer has responsibility for a number of key resources and assets.

The following table shows what aspects of the OpenShift Streams for Apache Kafka service are the responsibility of the customer, of Red Hat, and of the underlying cloud provider.

Resource Incident and operations management Change management Identity and access management Security and regulation compliance Disaster recovery
Customer data Customer Customer Customer Customer Customer
Customer applications Customer Customer Customer Customer Customer
Developer services Customer Customer Customer Customer Customer
Platform monitoring Red Hat Red Hat Red Hat Red Hat Red Hat
Logging Red Hat Red Hat Red Hat Red Hat Red Hat
Networking Red Hat Red Hat Red Hat Red Hat Red Hat
Capacity management Red Hat Red Hat Red Hat Red Hat Red Hat
Service storage Red Hat Red Hat Red Hat Red Hat Red Hat
Physical infrastructure and security Red Hat/Cloud Provider Red Hat/Cloud Provider Red Hat/Cloud Provider Red Hat/Cloud Provider Red Hat/Cloud Provider

Getting Support

As a premium offering by Red Hat, users have full access to the Red Hat Customer Portal with 24x7 production and developer-level support for our General Availability services.

Whenever users have a question or issue, they can file a ticket that specifies Red Hat OpenShift Streams for Apache Kafka.

Production Support Terms of Service

Additional resources