[BUG] Cluster upgrade to 4.6.30 version failed due to issues with thanos-ruler-user-workload pods in openshift-user-workload-monitoring namespace.
Issue
- Cluster monitoring operator degraded during
4.6.30cluster upgrade.
- lastTransitionTime: '2021-06-01T16:04:55Z'
message: 'Failed to rollout the stack. Error: running task Updating User Workload Thanos Ruler failed: waiting for ThanosRuler object changes failed: waiting for Thanos Ruler openshift-user-workload-monitoring/user-workload: expected 2 replicas, got 0 updated replicas'
reason: UpdatingUserWorkloadThanosRulerFailed
status: 'True'
type: Degraded
- Both pods of thanos-ruler-user-workload are in crashloopbackoff state due to the observed error message:
level=error ts=2021-06-01T17:04:27.144903541Z caller=main.go:212 err="lookup SRV records \"_web._tcp.alertmanager-operated.openshift-monitoring.svc\": lookup _web._tcp.alertmanager-operated.openshift-monitoring.svc on 172.21.0.10:53: cannot unmarshal DNS message\nrule command failed\nmain.main\n\t/go/src/github.com/improbable-eng/thanos/cmd/thanos/main.go:212\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1374"
Environment
- Red Hat OpenShift Container Platform
- 4.6.30
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.