How to Setup Email Receivers for User-defined Projects in OpenShift

Resolution

This a general step-by-step example of setting up email receivers and rules for user-defined projects in OpenShift by using AlertmanagerConfig:

1. Enable Monitoring for User-Defined Projects: As a cluster administrator, you can enable monitoring for user-defined projects by setting the enableUserWorkload: true field in the cluster monitoring ConfigMap object.

$ oc get cm cluster-monitoring-config -n openshift-monitoring -o yaml
apiVersion: v1
data:
  config.yaml: |
    enableUserWorkload: true
    alertmanagerMain: {}
    prometheusK8s: {}
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring

2. Enable Alert Routing: To use the default platform Alertmanager instance or a separate Alertmanager instance only for user-defined projects, you need to enable alert routing for user-defined projects. This process is described in [1].

$ oc get cm user-workload-monitoring-config -n openshift-user-workload-monitoring -o yaml
apiVersion: v1
data:
  config.yaml: |
    alertmanager:
      enabled: true
      enableAlertmanagerConfig: true
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring

3. Grant Permissions: You need to grant users permission to configure alert routing for user-defined projects as per Document. [2].

4. Create AlertmanagerConfig: After the above steps, you can create the AlertmanagerConfig custom resource in your user-defined project namespace. This custom resource will define your alert receivers (like email, PagerDuty, etc.) and routing rules. You can follow the example here to create an AlertmanagerConfig with an email receiver.
   Create a new project custom-alert.

oc new-project custom-alert

Create a secret for SMTP password.

oc create secret generic smtp-password-secret --from-literal password=$SUPERSTORNGPASSWORD -n custom-alert

Replace the $SUPERSTORNGPASSWORD with your email account password

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: custom-alert
  namespace: custom-alert
spec:
  receivers:
  - name: 'email_receiver'
    emailConfigs:
    - to: 'your-email@example.com'
      from: 'alertmanager@example.com'
      smarthost: 'smtp.example.com:587'
      authUsername: 'your-email@example.com'
      authPassword:
        name: 'smtp-password-secret'
        key: 'password'
  route:
    groupBy: ['job']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: 'email_receiver'

You can replace 'your-email@example.com' with your email address and 'smtp.example.com:587' with your email server and port.

5. SMTP Testing: You could use the following command to test the SMTP authentication with a test email.

$ swaks --to your-email@example.com --from your-email@example.com --server smtp.example.com:587 --auth LOGIN --auth-user "your-email@example.com" --auth-password $SUPERSTORNGPASSWORD --tls

If it is successful, you should receive a test email with the tile as sending timestamps and content being This is a test mailing.

6. Create Alerting Rule: Once you have the AlertmanagerConfig resource in place, you can now create your alerting rules. Follow this example to create an alerting rule.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prometheus-example-rules
  namespace: custom-alert
spec:
  groups:
  - name: example.rules
    rules:
    - alert: ExampleAlert
      expr: vector(1)

7. Apply Configuration to the Cluster: After creating your YAML files with the configurations, you can apply them to your cluster.

$ oc apply -f prometheus-example-rules.yaml

8. Check email box: You should receive email alerts with title like [FIRING:1] (ExampleAlert custom-alert).

Note: Remember to replace the relevant placeholders in the examples with your actual values.