Image registry pulls incorrect digest when mirroring enabled

Solution Verified - Updated -

Issue

  • The internal openshift image registry is proxying to an incorrect digest for all openshift image streams when mirroring is enabled.

  • In a cluster with image mirroring configured, when a pod is created with image as image-registry.openshift-image-registry.svc:5000/openshift/cli:latest (example), the pod fails due to ImagePullBackOff.

25m         Normal    Pulling                pod/ldap-group-sync   Pulling image "image-registry.openshift-image-registry.svc:5000/openshift/cli"
25m         Warning   Failed                 pod/ldap-group-sync   Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/cli": rpc error: code = Unknown desc = Error parsing image configuration: Error fetching blob: invalid status code from registry 500 (Internal Server Error)
25m         Warning   Failed                 pod/ldap-group-sync   Error: ErrImagePull
22m         Normal    BackOff                pod/ldap-group-sync  Back-off pulling image "image-registry.openshift-image-registry.svc:5000/openshift/cli"
25m         Warning   Failed                 pod/ldap-group-sync   Error: ImagePullBackOff
2021-09-24T10:55:41.872730246Z time="2021-09-24T10:55:41.872606439Z" level=error msg="response completed with error" err.code=unknown err.detail="Get \"https://mirror-registry/mirror/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f\": unauthorized: Not Authorized." err.message="unknown error" go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=c5e9175e-2b9c-4559-8c6e-0755e3306210 http.request.method=GET http.request.remoteaddr="10.x.x.x:1000" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=889.386496ms http.response.status=500 http.response.written=275 openshift.auth.user="system:serviceaccount:ldap-sync:ldap-group-syncer" vars.digest="sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" vars.name=openshift/cli

2021-09-24T10:55:41.872781776Z time="2021-09-24T10:55:41.872738525Z" level=info msg=response go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d5db8919-e7d9-40cd-bf29-5dbad08b0e6e http.request.method=GET http.request.remoteaddr="240.0.2.1:38736" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=889.558678ms http.response.status=500 http.response.written=275

2021-09-24T10:55:42.002776272Z time="2021-09-24T10:55:42.002690679Z" level=error msg="Background mirroring failed: error committing to storage: Get \"https://mirror-registry/mirror/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f\": unauthorized: Not Authorized." go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=c5e9175e-2b9c-4559-8c6e-0755e3306210 http.request.method=GET http.request.remoteaddr="10.x.x.x:1000" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:ldap-sync:ldap-group-syncer" vars.digest="sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" vars.name=openshift/cli

Environment

  • Red Hat OpenShift Container Platform(RHOCP)
    • 4.8.10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content