Machine-Config Cluster operator degraded due to error "unexpected on-disk state validating against rendered-master-XXX: expected target osImageURL "quay.io...@XXX " have "quay.io...@XXX...possible root cause: error: Child process exited with code 1

Issue

• During Cluster upgrade, machine-config cluster operator enter into degraded state due to osImageURL mismatch on render:

master: 'pool is degraded because nodes fail with "1 nodes are reporting degraded
      status on sync": "Node <node_name> is reporting:
      \"unexpected on-disk state validating against rendered-master-xxxx:
      expected target osImageURL \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:527a76ea6df69f9cb36276cc1822c74b47beb3b30f1357feacfe2119758d8550\\\",
      have \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:068f0728a42fa333660d2c4f70f04d8fc56036aad0a7a0b5095bdab21903bdfc\\\"
      (\\\"d818d560661c081002e3e162a8e04f042d38c1ba15081eea0a8957b3dd1bf8a5\\\")\""'

• Reported render has proper osImageURL defined

 osImageURL: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:527a76ea6df69f9cb36276cc1822c74b47beb3b30f1357feacfe2119758d8550  ====> Value described on machine-config operator as osImageURL expected

• machine config pool has UPDATING and DEGRADED set in true

NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-xxxxxxx   False     True       True       3              0                   0                     1                      2y

• ostree-finalize-staged.service journal shows next error:

E0216 16:18:34.580324    2438 writer.go:226] Marking Degraded due to: unexpected on-disk state validating against rendered-master-ef29d3a1ee7134527b09aec48e0ed9d3: expected target osImageURL "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:527a76ea6df69f9cb36276cc1822c74b47beb3b30f1357feacfe2119758d8550", have "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:068f0728a42fa333660d2c4f70f04d8fc56036aad0a7a0b5095bdab21903bdfc" ("d818d560661c081002e3e162a8e04f042d38c1ba15081eea0a8957b3dd1bf8a5"); possible root cause: error: Child process exited with code 1