[BUG] Cluster upgrade to 4.6.30 version failed due to issues with thanos-ruler-user-workload pods in openshift-user-workload-monitoring namespace.

Solution In Progress - Updated -

Issue

  • Cluster monitoring operator degraded during 4.6.30 cluster upgrade.
 - lastTransitionTime: '2021-06-01T16:04:55Z'
    message: 'Failed to rollout the stack. Error: running task Updating User Workload Thanos Ruler failed: waiting for ThanosRuler object changes failed: waiting for Thanos Ruler openshift-user-workload-monitoring/user-workload: expected 2 replicas, got 0 updated replicas'
    reason: UpdatingUserWorkloadThanosRulerFailed
    status: 'True'
    type: Degraded
  • Both pods of thanos-ruler-user-workload are in crashloopbackoff state due to the observed error message:
level=error ts=2021-06-01T17:04:27.144903541Z caller=main.go:212 err="lookup SRV records \"_web._tcp.alertmanager-operated.openshift-monitoring.svc\": lookup _web._tcp.alertmanager-operated.openshift-monitoring.svc on 172.21.0.10:53: cannot unmarshal DNS message\nrule command failed\nmain.main\n\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:212\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1374"

Environment

  • Red Hat OpenShift Container Platform
    • 4.6.30

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content