OpenShift Cluster Monitoring degraded on hosted cluster when MultiCluster Observability Addon is enabled
Issue
After installing with or importing an OpenShift cluster into Red Hat Advanced Cluster Management 2.15.0, monitoring on the hosted cluster is in degraded state:
$ oc get clusteroperator monitoring
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
monitoring 4.18.27 False True True 6d19h UpdatingPrometheus: Prometheus "openshift-monitoring/k8s": NoPodReady: shard 0: pod prometheus-k8s-0: containers with incomplete status: [init-config-reloader]...
Looking at the Prometheus pods in namespace openshift-monitoring, both pods are stuck at the Init step:
$ oc --namespace openshift-monitoring get pods --selector prometheus=k8s
prometheus-k8s-0 0/6 Init:0/1 0 2m27s
prometheus-k8s-1 0/6 Init:0/1 0 2m24s
Events in the openshift-monitoring namespace show failed secret mounts:
$ oc --namespace openshift-monitoring get events
LAST SEEN TYPE REASON OBJECT MESSAGE
57m Warning FailedMount pod/prometheus-k8s-0 MountVolume.SetUp failed for volume "secret-hub-alertmanager-router-ca" : secret "hub-alertmanager-router-ca" not found
53m Warning FailedMount pod/prometheus-k8s-0 MountVolume.SetUp failed for volume "secret-observability-alertmanager-accessor" : secret "observability-alertmanager-accessor" not found
However, secrets containing hub-alertmanager-router-ca and observability-alertmanager-accessor exist:
$ oc --namespace openshift-monitoring get secrets | grep hub-alertmanager-router-ca
hub-alertmanager-router-ca-xyz123 Opaque 1 6d
$ oc --namespace openshift-monitoring get secrets | grep observability-alertmanager-accessor
observability-alertmanager-accessor-xyz123 Opaque 1 6d
Environment
- Red Hat Advanced Cluster Management for Kubernetes (RHACM) 2.15.0
- MultiCluster Observability Addon (MCOA) enabled
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.