How to add additional labels to AlertManager Rules in OpenShift

Solution Unverified - Updated -

Environment

  • OpenShift 4.x
  • Gatekeeper

Issue

Prometheus / AlertManager alerts are forwarded to a central federating service.

  • Is it possible to group alerts from each cluster together?
  • Can I make alerts of the same name unique?

Resolution

This solution requires Gatekeeper to be present with the mutating functionality enabled.
* NOTE: Gatekeeper is an add-on to the OpenShift Container Platform and is available through the Operator Hub. This article assumes that Gatekeeper is already installed and does not cover how to install Gatekeeper. *

  1. Apply the Gatekeeper Rule to add label(s) to all PrometheusRule objects within the cluster:
  $ cat <<EOF | oc apply -f -
  apiVersion: mutations.gatekeeper.sh/v1alpha1
  kind: Assign
  metadata:
    name: configure-prometheus-rule-labels
  spec:
    applyTo:
    - groups: ["monitoring.coreos.com"]
      kinds: ["PrometheusRule"]
      versions: ["v1"]
    match:
      scope: Namespaced
      kinds:
      - apiGroups: ["monitoring.coreos.com"]
        kinds: ["PrometheusRule"]
    location: "spec.groups[name:*].rules[alert:*].labels.test"
    parameters:
      assign:
        value: "value"
  EOF

NOTE: This will add a label with the key of test and a value of value. Please replace these placeholders with the desired values.

  1. Create a PrometheusRule object or follow steps in [Optional] secton.

  2. Confirm the label is applied to the created PrometheusRules.

    $ oc get prometheusrule -o yaml | grep <expectedLabel>

[Optional] Apply Labels to OpenShift default PrometheusRule

  1. Backup all PrometheusRules from the Cluster and delete all Operator-controlled OpenShift PrometheusRules.

    $ oc get prometheusrule -A -o yaml > promrules.yaml
$ for NS in `oc get ns -o name | cut -d "/" -f 2 | grep openshift` ; do echo "$NS" ; oc delete prometheusrules --all -n $NS ;  done

  2. Wait for the PrometheusRule to be recreated. This can take a number of minutes.

Root Cause

OpenShifts core PrometheusRule objects are controlled by the OpenShift Operators. The Operators are all independently configured and many do not provide the ability to perform changes to their PrometheusRules.

Gatekeeper is a validating and mutating webhook feature that enforces CRD-based policies executed by Open Policy Agent. These mutating features can be used to manipulate objects in OpenShift as they are created.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments