OpenShift Cluster Monitoring degraded on hosted cluster when MultiCluster Observability Addon is enabled

Solution Verified - Updated -

Issue

After installing with or importing an OpenShift cluster into Red Hat Advanced Cluster Management 2.15.0, monitoring on the hosted cluster is in degraded state:

$ oc get clusteroperator monitoring
NAME         VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
monitoring   4.18.27   False       True          True       6d19h   UpdatingPrometheus: Prometheus "openshift-monitoring/k8s": NoPodReady: shard 0: pod prometheus-k8s-0: containers with incomplete status: [init-config-reloader]...

Looking at the Prometheus pods in namespace openshift-monitoring, both pods are stuck at the Init step:

$ oc --namespace openshift-monitoring get pods --selector prometheus=k8s
prometheus-k8s-0                                         0/6     Init:0/1    0          2m27s
prometheus-k8s-1                                         0/6     Init:0/1    0          2m24s

Events in the openshift-monitoring namespace show failed secret mounts:

$ oc --namespace openshift-monitoring get events
LAST SEEN   TYPE      REASON             OBJECT                       MESSAGE
57m         Warning   FailedMount        pod/prometheus-k8s-0         MountVolume.SetUp failed for volume "secret-hub-alertmanager-router-ca" : secret "hub-alertmanager-router-ca" not found
53m         Warning   FailedMount        pod/prometheus-k8s-0         MountVolume.SetUp failed for volume "secret-observability-alertmanager-accessor" : secret "observability-alertmanager-accessor" not found

However, secrets containing hub-alertmanager-router-ca and observability-alertmanager-accessor exist:

$ oc --namespace openshift-monitoring get secrets | grep hub-alertmanager-router-ca
hub-alertmanager-router-ca-xyz123            Opaque                    1      6d

$ oc --namespace openshift-monitoring get secrets | grep observability-alertmanager-accessor
observability-alertmanager-accessor-xyz123   Opaque                    1      6d

Environment

  • Red Hat Advanced Cluster Management for Kubernetes (RHACM) 2.15.0
  • MultiCluster Observability Addon (MCOA) enabled

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content