Chapter 3. Configuring external Alertmanager instances

The OpenShift Container Platform monitoring stack includes a local Alertmanager instance that routes alerts from Prometheus. You can add external Alertmanager instances to route alerts for core OpenShift Container Platform projects or user-defined projects.

If you add the same external Alertmanager configuration for multiple clusters and disable the local instance for each cluster, you can then manage alert routing for multiple clusters by using a single external Alertmanager instance.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components in the openshift-monitoring project:

    • You have access to the cluster as a user with the cluster-admin cluster role.
    • You have created the cluster-monitoring-config config map.
  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the cluster-admin cluster role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the user-workload-monitoring-config config map.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the ConfigMap object.

    • To configure additional Alertmanagers for routing alerts from core OpenShift Container Platform projects:

      1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
      2. Add an additionalAlertmanagerConfigs: section under data/config.yaml/prometheusK8s.
      3. Add the configuration details for additional Alertmanagers in this section:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              additionalAlertmanagerConfigs:
              - <alertmanager_specification>

        For <alertmanager_specification>, substitute authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig). The following sample config map configures an additional Alertmanager using a bearer token with client TLS authentication:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: cluster-monitoring-config
          namespace: openshift-monitoring
        data:
          config.yaml: |
            prometheusK8s:
              additionalAlertmanagerConfigs:
              - scheme: https
                pathPrefix: /
                timeout: "30s"
                apiVersion: v1
                bearerToken:
                  name: alertmanager-bearer-token
                  key: token
                tlsConfig:
                  key:
                    name: alertmanager-tls
                    key: tls.key
                  cert:
                    name: alertmanager-tls
                    key: tls.crt
                  ca:
                    name: alertmanager-tls
                    key: tls.ca
                staticConfigs:
                - external-alertmanager1-remote.com
                - external-alertmanager1-remote2.com
    • To configure additional Alertmanager instances for routing alerts from user-defined projects:

      1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
      2. Add a <component>/additionalAlertmanagerConfigs: section under data/config.yaml/.
      3. Add the configuration details for additional Alertmanagers in this section:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            <component>:
              additionalAlertmanagerConfigs:
              - <alertmanager_specification>

        For <component>, substitute one of two supported external Alertmanager components: prometheus or thanosRuler.

        For <alertmanager_specification>, substitute authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig). The following sample config map configures an additional Alertmanager using Thanos Ruler with a bearer token and client TLS authentication:

        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: user-workload-monitoring-config
          namespace: openshift-user-workload-monitoring
        data:
          config.yaml: |
            thanosRuler:
              additionalAlertmanagerConfigs:
              - scheme: https
                pathPrefix: /
                timeout: "30s"
                apiVersion: v1
                bearerToken:
                  name: alertmanager-bearer-token
                  key: token
                tlsConfig:
                  key:
                    name: alertmanager-tls
                    key: tls.key
                  cert:
                    name: alertmanager-tls
                    key: tls.crt
                  ca:
                    name: alertmanager-tls
                    key: tls.ca
                staticConfigs:
                - external-alertmanager1-remote.com
                - external-alertmanager1-remote2.com
  2. Save the file to apply the changes to the ConfigMap object. The new component placement configuration is applied automatically.

    Note

    Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

  3. Save the file to apply the changes to the ConfigMap object. The new component placement configuration is applied automatically.