How to Setup Email Receivers for User-defined Projects in RHOCP 4?

Resolution

This a general step-by-step example of setting up email receivers and rules for user-defined projects in OpenShift by using AlertmanagerConfig:

1. Enable Monitoring for User-Defined Projects: As a cluster administrator, you can enable monitoring for user-defined projects by setting the enableUserWorkload: true field in the cluster monitoring ConfigMap object.

$ oc get cm cluster-monitoring-config -n openshift-monitoring -o yaml
apiVersion: v1
data:
  config.yaml: |
    enableUserWorkload: true
    alertmanagerMain: {}
    prometheusK8s: {}
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring

2. Enable Alert Routing: To use the default platform Alertmanager instance or a separate Alertmanager instance only for user-defined projects, enable alert routing for user-defined projects. This process is described in [1].

$ oc get cm user-workload-monitoring-config -n openshift-user-workload-monitoring -o yaml
apiVersion: v1
data:
  config.yaml: |
    alertmanager:
      enabled: true
      enableAlertmanagerConfig: true
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring

3. Grant Permissions: Grant users permission to configure alert routing for user-defined projects as per Document. [2].

4. Create AlertmanagerConfig: After the above steps, create the AlertmanagerConfig custom resource in the user-defined project namespace. This custom resource will define the alert receivers (like email, PagerDuty, etc.) and routing rules. Follow the example here to create an AlertmanagerConfig with an email receiver.

Create a new project custom-alert.

oc new-project custom-alert

Create a secret for SMTP password.

oc create secret generic smtp-password-secret --from-literal password=$SUPERSTORNGPASSWORD -n custom-alert

Replace the $SUPERSTORNGPASSWORD with your email account password

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom-alert
  namespace: custom-alert
spec:
  receivers:
  - name: 'email_receiver'
    emailConfigs:
    - to: 'your-email@example.com'
      from: 'alertmanager@example.com'
      smarthost: 'smtp.example.com:587'
      authUsername: 'your-email@example.com'
      authPassword:
        name: 'smtp-password-secret'
        key: 'password'
  route:
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'email_receiver'

Replace 'your-email@example.com' with the email address and 'smtp.example.com:587' with the email server and port.

5. SMTP Testing: You could use the following command to test the SMTP authentication with a test email.

$ swaks --to your-email@example.com --from your-email@example.com --server smtp.example.com:587 --auth LOGIN --auth-user "your-email@example.com" --auth-password $SUPERSTORNGPASSWORD --tls

If it is successful, a test email will be received with the tile as sending timestamps and content being This is a test mailing.

6. Create Alerting Rule: Once the AlertmanagerConfig resource in place, create your alerting rules. Follow this example to create an alerting rule.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prometheus-example-rules
  namespace: custom-alert
spec:
  groups:
  - name: example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)

7. Apply Configuration to the Cluster: After creating the YAML files with the configurations, apply them to the cluster.

$ oc apply -f prometheus-example-rules.yaml

8. Check email box: An email alerts with title like [FIRING:1] (ExampleAlert custom-alert).

Note: Remember to replace the relevant placeholders in the examples with the actual values.