ARO cluster or OCP cluster in Azure upgraded but Machine Config Pools Degraded with error: Marking Degraded due to: unexpected on-disk state

Solution Unverified - Updated -

Issue

  • After upgrading an ARO cluster or and OCP cluster installed in Azure, the node versions are inconsistent:

    $ oc get nodes
    NAME                      STATUS                    ROLES   AGE    VERSION
    aro-master-2              Ready                     master  403d   v1.21.1+6438632
    aro-master-0              Ready                     master  403d   v1.21.1+6438632
    aro-master-1              Ready                     master  403d   v1.21.1+6438632
    aro-worker-regionx-xxxxx  Ready                     worker  6h4m   v1.20.0+bbbc079
    aro-worker-regionx-xxxxx  Ready                     worker  3h20m  v1.20.0+bbbc079
    aro-worker-regionx-xxxxx  Ready,SchedulingDisabled  worker  2h2m   v1.20.0+bbbc079
    aro-worker-regionx-xxxxx  Ready                     worker  23d    v1.20.0+bbbc079
    aro-worker-regionx-xxxxx  Ready                     worker  5h31m  v1.20.0+bbbc079
    
  • A machine config pool is degraded and shows the errors specified in "Diagnostic Steps" section:

    Marking Degraded due to: unexpected on-disk state validating against rendered-worker-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: expected target osImageURL "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz", have "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
    
  • The machine-config-daemon shows the following error for the azure.sock file:

    error: During /etc merge: Copying azure.sock: Cannot copy non-regular/non-symlink file: azure.sock
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Azure Red Hat OpenShift (ARO)
    • 4
  • OpenShift Managed (Azure)
    • 4
  • Azure Key Vault Provider for Secrets Store CSI Driver (secrets-store-csi-driver-provider-azure)
  • Azure

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content