Image can't be pulled due to digest mismatch, expected sha256 not found

Solution In Progress - Updated -

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.7.24
  • Red Hat OpenShift Container Storage (OCS) 4
  • Kernel-4.18.0-305.10.2.el8_4.x86_64

Issue

  • Pod creation fail due to digest mismatch on images modified after the upgrade to OCP 4.7.24

Resolution

  • Seems that performing a new build after deleting the data file identified as corrupted in the registry might be a work around.
  • Issue is still under investigation in bugzilla 1999591

Root Cause

Diagnostic Steps

  • Pod can't be created due to error during the image pull
28s         Normal    Pulling                          pod/mytest-56-6g54x                      Pulling image "image-registry.openshift-image-registry.svc:5000/test-tech/mytest@sha256:f79ef4b427dd3f27842903e91e23ebde9a0c6f59f271010ac6f1c4900da6c418"
43s         Warning   Failed                           pod/mytest-56-6g54x                      Failed to pull image "image-registry.openshift-image-registry.svc:5000/test-tech/mytest@sha256:f79ef4b427dd3f27842903e91e23ebde9a0c6f59f271010ac6f1c4900da6c418": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage895446108/4": error happened during read: Digest did not match, expected sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51, got sha256:b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce
26s         Warning   Failed                           pod/mytest-56-6g54x                      Error: ErrImagePull
13s         Normal    BackOff                          pod/mytest-56-6g54x                      Back-off pulling image "image-registry.openshift-image-registry.svc:5000/test-tech/mytest@sha256:f79ef4b427dd3f27842903e91e23ebde9a0c6f59f271010ac6f1c4900da6c418"
26s         Warning   Failed                           pod/mytest-56-6g54x                      Failed to pull image "image-registry.openshift-image-registry.svc:5000/test-tech/mytest@sha256:f79ef4b427dd3f27842903e91e23ebde9a0c6f59f271010ac6f1c4900da6c418": rpc error: code = Unknown desc = Error writing blob: error storing blob to file "/var/tmp/storage877899811/1": error happened during read: Digest did not match, expected sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51, got sha256:b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce
  • The image pull to local from OCP internal registry fail
# podman pull image-registry.openshift-image-registry.svc:5000/test-tech/mytest --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull image-registry.openshift-image-registry.svc:5000/test-tech/mytest --log-level debug)
...
Copying blob b799f9bfa86e [====================================>-] 44.0MiB / 44.7MiB
Copying blob b95c0dd0dc0d done
Copying blob 6f590a1c98e3 done
Copying blob 37e36a5f1ac1 done
DEBU[0036] Error pulling image ref //image-registry.openshift-image-registry.svc:5000/test-tech/mytest:latest: Error writing blob: error storing blob to file "/var/tmp/storage240344691/2": error happened during read: Digest did not match, expected sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51, got sha256:b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce
  Digest did not match, expected sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51, got sha256:b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce
Error: Error writing blob: error storing blob to file "/var/tmp/storage240344691/2": error happened during read: Digest did not match, expected sha256:4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51, got sha256:b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce
  • Hash is not matching in the repository
$ oc exec -ti image-registry-45xxx46fb7dd4-xyzdw -- ls -l /registry/docker/registry/v2/blobs/sha256/4c/4c25b3090c2685271afcffc2a4db73f15ab11a0124bfcde6085c934a4e6f4a51
total 53628
-rw-r--r--. 1 1000320000 root 54915004 Aug 26 06:50 data

$ sha256sum data
b662914768a0deb941633ae5956d6106ab63e2f25a4a51b327d4a299042fa0ce  data
  • Get the list of blobs corrupted, execute the search from an image-registry pod:
$ find /registry/docker/registry/v2/blobs/sha256/ -name data | while read -r f; do HASH=${f%/data}; HASH=${HASH##*/}; [ "$(sha256sum "$f" | cut -d' ' -f1)" == "$HASH" ] || echo $HASH; done

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments