How to Setup Email Receivers for User-defined Projects in OpenShift
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.11+
- Red Hat OpenShift on AWS (ROSA)
- 4.11+
- Red Hat OpenShift Dedicated (OSD)
- 4.11+
- Azure Red Hat OpenShift (ARO)
- 4.11+
Issue
- We have enabled monitoring for user-defined projects, Alerts are also getting triggered but the notifications are not being sent to the email address configured.
- How project admin can create alerting rule for a project.
Resolution
This a general step-by-step example of setting up email receivers and rules for user-defined projects in OpenShift by using AlertmanagerConfig:
- Enable Monitoring for User-Defined Projects: As a cluster administrator, you can enable monitoring for user-defined projects by setting the
enableUserWorkload: true
field in the cluster monitoring ConfigMap object.
$ oc get cm cluster-monitoring-config -n openshift-monitoring -o yaml
apiVersion: v1
data:
config.yaml: |
enableUserWorkload: true
alertmanagerMain: {}
prometheusK8s: {}
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
- Enable Alert Routing: To use the default platform Alertmanager instance or a separate Alertmanager instance only for user-defined projects, you need to enable alert routing for user-defined projects. This process is described in [1].
$ oc get cm user-workload-monitoring-config -n openshift-user-workload-monitoring -o yaml
apiVersion: v1
data:
config.yaml: |
alertmanager:
enabled: true
enableAlertmanagerConfig: true
kind: ConfigMap
metadata:
name: user-workload-monitoring-config
namespace: openshift-user-workload-monitoring
-
Grant Permissions: You need to grant users permission to configure alert routing for user-defined projects as per Document. [2].
-
Create AlertmanagerConfig: After the above steps, you can create the
AlertmanagerConfig
custom resource in your user-defined project namespace. This custom resource will define your alert receivers (like email, PagerDuty, etc.) and routing rules. You can follow the example here to create anAlertmanagerConfig
with an email receiver.
Create a new projectcustom-alert
.
oc new-project custom-alert
Create a secret for SMTP password.
oc create secret generic smtp-password-secret --from-literal password=$SUPERSTORNGPASSWORD -n custom-alert
Replace the $SUPERSTORNGPASSWORD
with your email account password
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: custom-alert
namespace: custom-alert
spec:
receivers:
- name: 'email_receiver'
emailConfigs:
- to: 'your-email@example.com'
from: 'alertmanager@example.com'
smarthost: 'smtp.example.com:587'
authUsername: 'your-email@example.com'
authPassword:
name: 'smtp-password-secret'
key: 'password'
route:
groupBy: ['job']
groupWait: 30s
groupInterval: 5m
repeatInterval: 12h
receiver: 'email_receiver'
You can replace 'your-email@example.com'
with your email address and 'smtp.example.com:587'
with your email server and port.
- SMTP Testing: You could use the following command to test the SMTP authentication with a test email.
$ swaks --to your-email@example.com --from your-email@example.com --server smtp.example.com:587 --auth LOGIN --auth-user "your-email@example.com" --auth-password $SUPERSTORNGPASSWORD --tls
If it is successful, you should receive a test email with the tile as sending timestamps and content being This is a test mailing
.
- Create Alerting Rule: Once you have the AlertmanagerConfig resource in place, you can now create your alerting rules. Follow this example to create an alerting rule.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: prometheus-example-rules
namespace: custom-alert
spec:
groups:
- name: example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
- Apply Configuration to the Cluster: After creating your YAML files with the configurations, you can apply them to your cluster.
$ oc apply -f prometheus-example-rules.yaml
- Check email box: You should receive email alerts with title like
[FIRING:1] (ExampleAlert custom-alert)
.
Note: Remember to replace the relevant placeholders in the examples with your actual values.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments