Chapter 3. Configuring external alertmanager instances

The OpenShift Container Platform monitoring stack includes a local Alertmanager instance that routes alerts from Prometheus. You can add external Alertmanager instances by configuring the cluster-monitoring-config config map in either the openshift-monitoring project or the user-workload-monitoring-config project.

If you add the same external Alertmanager configuration for multiple clusters and disable the local instance for each cluster, you can then manage alert routing for multiple clusters by using a single external Alertmanager instance.

Prerequisites

  • You have installed the OpenShift CLI (oc).

  • If you are configuring core OpenShift Container Platform monitoring components in the openshift-monitoring project:

    • You have access to the cluster as a user with the cluster-admin role.
    • You have created the cluster-monitoring-config config map.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the cluster-admin role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the user-workload-monitoring-config config map.

Procedure

  1. Edit the ConfigMap object.

    • To configure additional Alertmanagers for routing alerts from core OpenShift Container Platform projects:

      1. Edit the cluster-monitoring-config config map in the openshift-monitoring project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config
      2. Add an additionalAlertmanagerConfigs: section under data/config.yaml/prometheusK8s.

      3. Add the configuration details for additional Alertmanagers in this section: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      additionalAlertmanagerConfigs:
      - <alertmanager_specification>

        For <alertmanager_specification>, substitute authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig). The following sample config map configures an additional Alertmanager using a bearer token with client TLS authentication: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      additionalAlertmanagerConfigs:
      - scheme: https
        pathPrefix: /
        timeout: "30s"
        apiVersion: v1
        bearerToken:
          name: alertmanager-bearer-token
          key: token
        tlsConfig:
          key:
            name: alertmanager-tls
            key: tls.key
          cert:
            name: alertmanager-tls
            key: tls.crt
          ca:
            name: alertmanager-tls
            key: tls.ca
        staticConfigs:
        - external-alertmanager1-remote.com
        - external-alertmanager1-remote2.com

    • To configure additional Alertmanager instances for routing alerts from user-defined projects:

      1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config
      2. Add a <component>/additionalAlertmanagerConfigs: section under data/config.yaml/.

      3. Add the configuration details for additional Alertmanagers in this section: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>:
      additionalAlertmanagerConfigs:
      - <alertmanager_specification>

        For <component>, substitute one of two supported external Alertmanager components: prometheus or thanosRuler.

        For <alertmanager_specification>, substitute authentication and other configuration details for additional Alertmanager instances. Currently supported authentication methods are bearer token (bearerToken) and client TLS (tlsConfig). The following sample config map configures an additional Alertmanager using Thanos Ruler with a bearer token and client TLS authentication: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
   thanosRuler:
     additionalAlertmanagerConfigs:
    - scheme: https
      pathPrefix: /
      timeout: "30s"
      apiVersion: v1
      bearerToken:
        name: alertmanager-bearer-token
        key: token
      tlsConfig:
        key:
          name: alertmanager-tls
          key: tls.key
        cert:
          name: alertmanager-tls
          key: tls.crt
        ca:
          name: alertmanager-tls
          key: tls.ca
      staticConfigs:
      - external-alertmanager1-remote.com
      - external-alertmanager1-remote2.com
        Note

        Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

  2. Save the file to apply the changes to the ConfigMap object. The new component placement configuration is applied automatically.

3.1. Attaching additional labels to your time series and alerts

Using the external labels feature of Prometheus, you can attach custom labels to all time series and alerts leaving Prometheus.

Prerequisites

  • If you are configuring core OpenShift Container Platform monitoring components:

    • You have access to the cluster as a user with the cluster-admin role.
    • You have created the cluster-monitoring-config ConfigMap object.

  • If you are configuring components that monitor user-defined projects:

    • You have access to the cluster as a user with the cluster-admin role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the user-workload-monitoring-config ConfigMap object.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the ConfigMap object:

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors core OpenShift Container Platform projects:

      1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Define a map of labels you want to add for every metric under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        <key>: <value> 1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        For example, to add metadata about the region and environment to all time series and alerts, use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        region: eu
        environment: prod

    • To attach custom labels to all time series and alerts leaving the Prometheus instance that monitors user-defined projects:

      1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Define a map of labels you want to add for every metric under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        <key>: <value> 1
        1
        Substitute <key>: <value> with a map of key-value pairs where <key> is a unique name for the new label and <value> is its value.
        Warning

        Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.

        Note

        In the openshift-user-workload-monitoring project, Prometheus handles metrics and Thanos Ruler handles alerting and recording rules. Setting externalLabels for prometheus in the user-workload-monitoring-config ConfigMap object will only configure external labels for metrics and not for any rules.

        For example, to add metadata about the region and environment to all time series and alerts related to user-defined projects, use: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      externalLabels:
        region: eu
        environment: prod

  2. Save the file to apply the changes. The new configuration is applied automatically.

    Note

    Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

Additional resources

3.2. Setting log levels for monitoring components

You can configure the log level for Alertmanager, Prometheus Operator, Prometheus, Thanos Querier, and Thanos Ruler.

The following log levels can be applied to the relevant component in the cluster-monitoring-config and user-workload-monitoring-config ConfigMap objects:

  • debug. Log debug, informational, warning, and error messages.
  • info. Log informational, warning, and error messages.
  • warn. Log warning and error messages only.
  • error. Log error messages only.

The default log level is info.

Prerequisites

  • If you are setting a log level for Alertmanager, Prometheus Operator, Prometheus, or Thanos Querier in the openshift-monitoring project:

    • You have access to the cluster as a user with the cluster-admin role.
    • You have created the cluster-monitoring-config ConfigMap object.

  • If you are setting a log level for Prometheus Operator, Prometheus, or Thanos Ruler in the openshift-user-workload-monitoring project:

    • You have access to the cluster as a user with the cluster-admin role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.
    • You have created the user-workload-monitoring-config ConfigMap object.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the ConfigMap object:

    • To set a log level for a component in the openshift-monitoring project:

      1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project: 

        $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

      2. Add logLevel: <log_level> for a component under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    <component>: 1
      logLevel: <log_level> 2
        1
        The monitoring stack component for which you are setting a log level. For default platform monitoring, available component values are prometheusK8s, alertmanagerMain, prometheusOperator, and thanosQuerier.
        2
        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

    • To set a log level for a component in the openshift-user-workload-monitoring project:

      1. Edit the user-workload-monitoring-config ConfigMap object in the openshift-user-workload-monitoring project: 

        $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

      2. Add logLevel: <log_level> for a component under data/config.yaml: 

        apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    <component>: 1
      logLevel: <log_level> 2
        1
        The monitoring stack component for which you are setting a log level. For user workload monitoring, available component values are prometheus, prometheusOperator, and thanosRuler.
        2
        The log level to set for the component. The available values are error, warn, info, and debug. The default value is info.

  2. Save the file to apply the changes. The pods for the component restarts automatically when you apply the log-level change.

    Note

    Configurations applied to the user-workload-monitoring-config ConfigMap object are not activated unless a cluster administrator has enabled monitoring for user-defined projects.

    Warning

    When changes are saved to a monitoring config map, the pods and other resources in the related project might be redeployed. The running monitoring processes in that project might also be restarted.

  3. Confirm that the log-level has been applied by reviewing the deployment or pod configuration in the related project. The following example checks the log level in the prometheus-operator deployment in the openshift-user-workload-monitoring project: 

    $ oc -n openshift-user-workload-monitoring get deploy prometheus-operator -o yaml |  grep "log-level"

    Example output

            - --log-level=debug

  4. Check that the pods for the component are running. The following example lists the status of pods in the openshift-user-workload-monitoring project: 

    $ oc -n openshift-user-workload-monitoring get pods
    Note

    If an unrecognized loglevel value is included in the ConfigMap object, the pods for the component might not restart successfully.

Additional resources

3.3. Disabling the default Grafana deployment

By default, a read-only Grafana instance is deployed with a collection of dashboards displaying cluster metrics. The Grafana instance is not user-configurable.

You can disable the Grafana deployment, causing the associated resources to be deleted from the cluster. You might do this if you do not need these dashboards and want to conserve resources in your cluster. You will still be able to view metrics and dashboards included in the web console. Grafana can be safely enabled again at any time.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have created the cluster-monitoring-config ConfigMap object.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring project: 

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add enabled: false for the grafana component under data/config.yaml: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    grafana:
      enabled: false

  3. Save the file to apply the changes. The resources will begin to be removed automatically when you apply the change.

    Warning

    This change results in some components, including Prometheus and the Thanos Querier, being restarted. This might lead to previously collected metrics being lost if you have not yet followed the steps in the "Configuring persistent storage" section.

  4. Check that the Grafana pod is no longer running. The following example lists the status of pods in the openshift-monitoring project: 

    $ oc -n openshift-monitoring get pods
    Note

    It may take a few minutes after applying the change for these pods to terminate.

Additional resources

3.4. Disabling the local Alertmanager

A local Alertmanager that routes alerts from Prometheus instances is enabled by default in the openshift-monitoring project of the OpenShift Container Platform monitoring stack.

If you do not need the local Alertmanager, you can disable it by configuring the cluster-monitoring-config config map in the openshift-monitoring project.

Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.
  • You have created the cluster-monitoring-config config map.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. Edit the cluster-monitoring-config config map in the openshift-monitoring project: 

    $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

  2. Add enabled: false for the alertmanagerMain component under data/config.yaml: 

    apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    alertmanagerMain:
      enabled: false
  3. Save the file to apply the changes. The Alertmanager instance is disabled automatically when you apply the change.

Additional resources

3.5. Next steps