Cluster upgrade stuck as Monitoring operator is in degraded state with alertmanager-main-x pod in CreateContainerError state

Solution Verified - Updated -

Issue

  • Upgrade stuck from RHOCP 4.7.x to 4.7.y.
  • alertmanager-main-x pod is not starting with the below error:
  - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c31a553a10235882871ceb358b12e82d0a7d28996873d2a0b70bdd878a3d5146
    imageID: ""
    lastState: {}
    name: alertmanager
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: |
          container create failed: time="2021-05-25T09:11:24Z" level=error msg="container_linux.go:366: starting container process caused: exec: \"/bin/alertmanager\": stat /bin/alertmanager: no such file or directory"
        reason: CreateContainerError
  • Monitoring operator is in a degraded state with the below errors:
 lastTransitionTime: '2021-05-18T02:17:04Z'
    message: 'Failed to rollout the stack. Error: running task Updating Alertmanager
      failed: waiting for Alertmanager object changes failed: waiting for Alertmanager
      openshift-monitoring/main: expected 3 replicas, got 0 updated replicas'
    reason: UpdatingAlertmanagerFailed

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content