Unable to push/pull images to/from Azure Blob Storage backed internal image registry: Error writing blob: failed to read from destination repository <project>/<image>: 403 (Forbidden)

Solution In Progress - Updated -

Environment

  • Red Hat OpenShift Container Platform(RHOCP)
    • 4.x
  • Azure Blob Storage

Issue

  • Issue pushing an image to OpenShift internal registry when the registry is configured with Azure Blob Storage.
  • Getting below error while pushing image with Skopeo
  FATA[0001] Error trying to reuse blob sha256:<digest> at destination: failed to read from destination repository <project>/<image>: 403 (Forbidden)
  • Below error while pushing image using Podman
  Error writing blob: failed to read from destination repository <project>/<image>: 403 (Forbidden)

Resolution

  • Disable backend redirection of the request and route all data through the image registry.
$ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"disableRedirect":true}}' --type=merge

Root Cause

  • The disableRedirect controls whether to route all data through the Registry, rather than redirecting to the backend. Read this for more information.

Diagnostic Steps

$ curl -k -I -L --request 'GET' --header "Authorization: Bearer ${TOKEN}" https://default-route-openshift-image-registry.apps.cluster.example.com/v2/foo/alpine/blobs/sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba

HTTP/1.1 307 Temporary Redirect
Content-Type: application/octet-stream
Docker-Distribution-Api-Version: registry/2.0
Location: https://imageregistrysharedvtnwm.blob.core.windows.net/shared-services-m8qhb-image-registry-xlvnnmqtjpoqsuwjokfrvtevv//docker/registry/v2/blobs/sha256/54/540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba/data?se=2021-04-29T16%3A39%3A09Z&sig=z4lZvGZ4W7gaeGiyrGFj65cq1D1nV0o7pMPmFVaA1%2Bw%3D&sp=r&sr=b&sv=2016-05-31
X-Registry-Supports-Signatures: 1
Date: Thu, 29 Apr 2021 16:19:09 GMT
Content-Length: 0
Set-Cookie: 34727b82525ebf0ec2f2=d8fcb3d091cc51ca9674c90; path=/; HttpOnly; Secure; SameSite=None

HTTP/1.1 403 This request is not authorized to perform this operation.
Content-Length: 246
Content-Type: application/xml
Server: Microsoft-HTTPAPI/2.0
x-ms-request-id: 5d070f67-401e-0072-2013-3d423e000000
Date: Thu, 29 Apr 2021 16:19:08 GMT

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments