Chapter 4. Configuring secrets for Alertmanager

The OpenShift Container Platform monitoring stack includes Alertmanager, which routes alerts from Prometheus to endpoint receivers. If you need to authenticate with a receiver so that Alertmanager can send alerts to it, you can configure Alertmanager to use a secret that contains authentication credentials for the receiver.

For example, you can configure Alertmanager to use a secret to authenticate with an endpoint receiver that requires a certificate issued by a private Certificate Authority (CA). You can also configure Alertmanager to use a secret to authenticate with a receiver that requires a password file for Basic HTTP authentication. In either case, authentication details are contained in the Secret object rather than in the ConfigMap object.

4.1. Adding a secret to the Alertmanager configuration

You can add secrets to the Alertmanager configuration for core platform monitoring components by editing the cluster-monitoring-config config map in the openshift-monitoring project.

After you add a secret to the config map, the secret is mounted as a volume at /etc/alertmanager/secrets/<secret_name> within the alertmanager container for the Alertmanager pods.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components in the openshift-monitoring project:

    • You have access to the cluster as a user with the cluster-admin cluster role.
    • You have created the cluster-monitoring-config config map.
    • You have created the secret to be configured in Alertmanager in the openshift-monitoring project.

  • If you are configuring components that monitor user-defined projects:

    • A cluster administrator has enabled monitoring for user-defined projects.
    • You have access to the cluster as a user with the cluster-admin cluster role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the secret to be configured in Alertmanager in the openshift-user-workload-monitoring project.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the ConfigMap object.

    • To add a secret configuration to Alertmanager for core platform monitoring:

      1. Edit the cluster-monitoring-config config map in the openshift-monitoring project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add a secrets: section under data/config.yaml/alertmanagerMain with the following configuration: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      secrets: 1
      - <secret_name_1> 2
      - <secret_name_2>
        1
        This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.
        2
        The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

        The following sample config map settings configure Alertmanager to use two Secret objects named test-secret-basic-auth and test-secret-api-token: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      secrets:
      - test-secret-basic-auth
      - test-secret-api-token

    • To add a secret configuration to Alertmanager for user-defined project monitoring:

      1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add a secrets: section under data/config.yaml/alertmanager/secrets with the following configuration: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    alertmanager:
      secrets: 1
      - <secret_name_1> 2
      - <secret_name_2>
        1
        This section contains the secrets to be mounted into Alertmanager. The secrets must be located within the same namespace as the Alertmanager object.
        2
        The name of the Secret object that contains authentication credentials for the receiver. If you add multiple secrets, place each one on a new line.

        The following sample config map settings configure Alertmanager to use two Secret objects named test-secret and test-secret-api-token: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    alertmanager:
      enabled: true
      secrets:
      - test-secret
      - test-api-receiver-token
        Note

        Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

  2. Save the file to apply the changes to the ConfigMap object. The new configuration is applied automatically.

4.2. Attaching additional labels to your time series and alerts

You can attach custom labels to all time series and alerts leaving Prometheus by using the external labels feature of Prometheus.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the cluster-admin cluster role.
    • You have created the cluster-monitoring-config ConfigMap object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the cluster-admin cluster role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the user-workload-monitoring-config ConfigMap object.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the ConfigMap object:

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

      1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Define a map of labels you want to add for every metric under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        <key>: <value> 1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning
        • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
        • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.

        For example, to add metadata about the region and environment to all time series and alerts, use the following example: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        region: eu
        environment: prod

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

      1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Define a map of labels you want to add for every metric under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        <key>: <value> 1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning
        • Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
        • Do not use cluster or managed_cluster as key names. Using them can cause issues where you are unable to see data in the developer dashboards.
        Note

        In the openshift-user-workload-monitoring project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting externalLabels for prometheus in the user-workload-monitoring-config ConfigMap object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use the following example: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        region: eu
        environment: prod

  2. Save the file to apply the changes. The new configuration is applied automatically.

    Note

    Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Additional resources