Prometheus pods keep restarting on ROSA after modifying the cluster-monitoring-config configmap
Environment
- Red Hat OpenShift Service on AWS (ROSA) 4.x
Issue
- The prometheus pods keep restarting on
ROSAafter modifying the platform's monitoring stack via the configmapcluster-monitoring-configin the namespaceopenshift-monitoring.
Resolution
- The
user-workload-monitoringshould be used/configured as per docConfiguring the monitoring stack
Root Cause
- Using configurations other than those described in this link Configuring the monitoring stack, will disappear because the cluster-monitoring-operator reconciles any differences. The Operator resets everything to the defined state by default and by design.
Diagnostic Steps
- The following command should show the pods being recreated every hour or so:
$ oc get pods -n openshift-monitoring
$ oc get events -n openshift-monitoring
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments