Image registry pulls incorrect digest when mirroring enabled
Issue
-
The internal openshift image registry is proxying to an incorrect digest for all openshift image streams when mirroring is enabled.
-
In a cluster with image mirroring configured, when a pod is created with image as
image-registry.openshift-image-registry.svc:5000/openshift/cli:latest
(example), the pod fails due to ImagePullBackOff.
25m Normal Pulling pod/ldap-group-sync Pulling image "image-registry.openshift-image-registry.svc:5000/openshift/cli"
25m Warning Failed pod/ldap-group-sync Failed to pull image "image-registry.openshift-image-registry.svc:5000/openshift/cli": rpc error: code = Unknown desc = Error parsing image configuration: Error fetching blob: invalid status code from registry 500 (Internal Server Error)
25m Warning Failed pod/ldap-group-sync Error: ErrImagePull
22m Normal BackOff pod/ldap-group-sync Back-off pulling image "image-registry.openshift-image-registry.svc:5000/openshift/cli"
25m Warning Failed pod/ldap-group-sync Error: ImagePullBackOff
2021-09-24T10:55:41.872730246Z time="2021-09-24T10:55:41.872606439Z" level=error msg="response completed with error" err.code=unknown err.detail="Get \"https://mirror-registry/mirror/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f\": unauthorized: Not Authorized." err.message="unknown error" go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=c5e9175e-2b9c-4559-8c6e-0755e3306210 http.request.method=GET http.request.remoteaddr="10.x.x.x:1000" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=889.386496ms http.response.status=500 http.response.written=275 openshift.auth.user="system:serviceaccount:ldap-sync:ldap-group-syncer" vars.digest="sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" vars.name=openshift/cli
2021-09-24T10:55:41.872781776Z time="2021-09-24T10:55:41.872738525Z" level=info msg=response go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d5db8919-e7d9-40cd-bf29-5dbad08b0e6e http.request.method=GET http.request.remoteaddr="240.0.2.1:38736" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=889.558678ms http.response.status=500 http.response.written=275
2021-09-24T10:55:42.002776272Z time="2021-09-24T10:55:42.002690679Z" level=error msg="Background mirroring failed: error committing to storage: Get \"https://mirror-registry/mirror/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f\": unauthorized: Not Authorized." go.version=go1.16.6 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=c5e9175e-2b9c-4559-8c6e-0755e3306210 http.request.method=GET http.request.remoteaddr="10.x.x.x:1000" http.request.uri="/v2/openshift/cli/blobs/sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" http.request.useragent="cri-o/1.21.2-15.rhaos4.8.gitcdc4f56.el8 go/go1.16.6 os/linux arch/amd64" openshift.auth.user="system:serviceaccount:ldap-sync:ldap-group-syncer" vars.digest="sha256:84c9f2afcca866cd0246d0df94f38ec3512bfe83fa5fa772204cc636490c1e5f" vars.name=openshift/cli
Environment
- Red Hat OpenShift Container Platform(RHOCP)
- 4.8.10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.