Chapter 3. Loading configuration values from external sources

Use configuration provider plugins to load configuration data from external sources. The providers operate independently of AMQ Streams. You can use them to load configuration data for all Kafka components, including producers and consumers. Use them, for example, to provide the credentials for Kafka Connect connector configuration.

OpenShift Configuration Provider

The OpenShift Configuration Provider plugin loads configuration data from OpenShift secrets or config maps.

Suppose you have a Secret object that’s managed outside the Kafka namespace, or outside the Kafka cluster. The OpenShift Configuration Provider allows you to reference the values of the secret in your configuration without extracting the files. You just need to tell the provider what secret to use and provide access rights. The provider loads the data without needing to restart the Kafka component, even when using a new Secret or ConfigMap object. This capability avoids disruption when a Kafka Connect instance hosts multiple connectors.

Environment Variables Configuration Provider

The Environment Variables Configuration Provider plugin loads configuration data from environment variables.

The values for the environment variables can be mapped from secrets or config maps. You can use the Environment Variables Configuration Provider, for example, to load certificates or JAAS configuration from environment variables mapped from OpenShift secrets.

Note

OpenShift Configuration Provider can’t use mounted files. For example, it can’t load values that need the location of a truststore or keystore. Instead, you can mount config maps or secrets into a Kafka Connect pod as environment variables or volumes. You can use the Environment Variables Configuration Provider to load values for environment variables. You add configuration using the externalConfiguration property in KafkaConnect.spec. You don’t need to set up access rights with this approach. However, Kafka Connect will need a restart when using a new Secret or ConfigMap for a connector. This will cause disruption to all the Kafka Connect instance’s connectors.

3.1. Loading configuration values from a config map

This procedure shows how to use the OpenShift Configuration Provider plugin.

In the procedure, an external ConfigMap object provides configuration properties for a connector.

Prerequisites

  • An OpenShift cluster is available.
  • A Kafka cluster is running.
  • The Cluster Operator is running.

Procedure

  1. Create a ConfigMap or Secret that contains the configuration properties.

    In this example, a ConfigMap object named my-connector-configuration contains connector properties:

    Example ConfigMap with connector properties

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: my-connector-configuration
    data:
      option1: value1
      option2: value2

  2. Specify the OpenShift Configuration Provider in the Kafka Connect configuration.

    The specification shown here can support loading values from secrets and config maps.

    Example Kafka Connect configuration to enable the OpenShift Configuration Provider

    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaConnect
    metadata:
      name: my-connect
      annotations:
        strimzi.io/use-connector-resources: "true"
    spec:
      # ...
      config:
        # ...
        config.providers: secrets,configmaps 1
        config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider 2
        config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider 3
      # ...

    1
    The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from config.providers, taking the form config.providers.${alias}.class.
    2
    KubernetesSecretConfigProvider provides values from secrets.
    3
    KubernetesConfigMapConfigProvider provides values from config maps.
  3. Create or update the resource to enable the provider.

    oc apply -f <kafka_connect_configuration_file>
  4. Create a role that permits access to the values in the external config map.

    Example role to access values from a config map

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: connector-configuration-role
    rules:
    - apiGroups: [""]
      resources: ["configmaps"]
      resourceNames: ["my-connector-configuration"]
      verbs: ["get"]
    # ...

    The rule gives the role permission to access the my-connector-configuration config map.

  5. Create a role binding to permit access to the namespace that contains the config map.

    Example role binding to access the namespace that contains the config map

    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: connector-configuration-role-binding
    subjects:
    - kind: ServiceAccount
      name: my-connect-connect
      namespace: my-project
    roleRef:
      kind: Role
      name: connector-configuration-role
      apiGroup: rbac.authorization.k8s.io
    # ...

    The role binding gives the role permission to access the my-project namespace.

    The service account must be the same one used by the Kafka Connect deployment. The service account name format is <cluster_name>-connect, where <cluster_name> is the name of the KafkaConnect custom resource.

  6. Reference the config map in the connector configuration.

    Example connector configuration referencing the config map

    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaConnector
    metadata:
      name: my-connector
      labels:
        strimzi.io/cluster: my-connect
    spec:
      # ...
      config:
        option: ${configmaps:my-project/my-connector-configuration:option1}
        # ...
    # ...

    Placeholders for the property values in the config map are referenced in the connector configuration. The placeholder structure is configmaps:<path_and_file_name>:<property>. KubernetesConfigMapConfigProvider reads and extracts the option1 property value from the external config map.

3.2. Loading configuration values from environment variables

This procedure shows how to use the Environment Variables Configuration Provider plugin.

In the procedure, environment variables provide configuration properties for a connector. A database password is specified as an environment variable.

Prerequisites

  • An OpenShift cluster is available.
  • A Kafka cluster is running.
  • The Cluster Operator is running.

Procedure

  1. Specify the Environment Variables Configuration Provider in the Kafka Connect configuration.

    Define environment variables using the externalConfiguration property.

    Example Kafka Connect configuration to enable the Environment Variables Configuration Provider

    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaConnect
    metadata:
      name: my-connect
      annotations:
        strimzi.io/use-connector-resources: "true"
    spec:
      # ...
      config:
        # ...
        config.providers: env 1
        config.providers.env.class: io.strimzi.kafka.EnvVarConfigProvider 2
      # ...
      externalConfiguration:
        env:
          - name: DB_PASSWORD 3
            valueFrom:
              secretKeyRef:
                name: db-creds 4
                key: dbPassword 5
      # ...

    1
    The alias for the configuration provider is used to define other configuration parameters. The provider parameters use the alias from config.providers, taking the form config.providers.${alias}.class.
    2
    EnvVarConfigProvider provides values from environment variables.
    3
    The DB_PASSWORD environment variable takes a password value from a secret.
    4
    The name of the secret containing the predefined password.
    5
    The key for the password stored inside the secret.
  2. Create or update the resource to enable the provider.

    oc apply -f <kafka_connect_configuration_file>
  3. Reference the environment variable in the connector configuration.

    Example connector configuration referencing the environment variable

    apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaConnector
    metadata:
      name: my-connector
      labels:
        strimzi.io/cluster: my-connect
    spec:
      # ...
      config:
        option: ${env:DB_PASSWORD}
        # ...
    # ...