Chapter 1. Configuration overview

AMQ Streams simplifies the process of running Apache Kafka in an OpenShift cluster.

This guide describes how to configure and manage an AMQ Streams deployment.

1.1. Configuring custom resources

Use custom resources to configure your AMQ Streams deployment.

You can use custom resources to configure and create instances of the following components:

  • Kafka clusters
  • Kafka Connect clusters
  • Kafka MirrorMaker
  • Kafka Bridge
  • Cruise Control

You can also use custom resource configuration to manage your instances or modify your deployment to introduce additional features. This might include configuration that supports the following:

  • Securing client access to Kafka brokers
  • Accessing Kafka brokers from outside the cluster
  • Creating topics
  • Creating users (clients)
  • Controlling feature gates
  • Changing logging frequency
  • Allocating resource limits and requests
  • Introducing features, such as AMQ Streams Drain Cleaner, Cruise Control, or distributed tracing.

The Custom resource API reference describes the properties you can use in your configuration.

1.2. Configuring listeners to connect to Kafka brokers

Listeners are used to connect to Kafka brokers. AMQ Streams provides a generic GenericKafkaListener schema with properties to configure listeners through the Kafka resource.

The GenericKafkaListener provides a flexible approach to listener configuration. You can specify properties to configure internal listeners for connecting within the OpenShift cluster, or external listeners for connecting outside the OpenShift cluster.

Each listener is defined as an array in the Kafka resource. You can configure as many listeners as required, as long as their names and ports are unique.

You might want to configure multiple external listeners, for example, to handle access from networks that require different authentication mechanisms. Or you might need to join your OpenShift network to an outside network. In which case, you can configure internal listeners (using the useServiceDnsDomain property) so that the OpenShift service DNS domain (typically .cluster.local) is not used.

For more information on the configuration options available for listeners, see the GenericKafkaListener schema reference.

Configuring listeners to secure access to Kafka brokers

You can configure listeners for secure connection using authentication. For more information, see Securing access to Kafka brokers.

Configuring external listeners for client access outside OpenShift

You can configure external listeners for client access outside an OpenShift environment using a specified connection mechanism, such as a loadbalancer. For more information on the configuration options for connecting an external client, see Accessing Kafka from external clients outside of the OpenShift cluster.

Listener certificates

You can provide your own server certificates, called Kafka listener certificates, for TLS listeners or external listeners which have TLS encryption enabled. For more information, see Kafka listener certificates.

Note

If you scale your Kafka cluster while using external listeners, it might trigger a rolling update of all Kafka brokers. This depends on the configuration.

1.3. Document Conventions

User-replaced values

User-replaced values, also known as replaceables, are shown in italics with angle brackets (< >). Underscores ( _ ) are used for multi-word values. If the value refers to code or commands, monospace is also used.

For example, in the following code, you will want to replace <my_namespace> with the name of your namespace: 

sed -i 's/namespace: .*/namespace: <my_namespace>/' install/cluster-operator/*RoleBinding*.yaml

1.4. Additional resources