Error fetching blob: invalid status code from registry 403 (Forbidden) in ARO

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)

Issue

  • When pulling images from internal registry from cluster A to internal registry of cluster B fails with message:

    skopeo inspect --debug --creds skopeo:$TOKEN --tls-verify=false docker://default-route-openshift-image-registry.apps.myaro.supercluster.com/myrepo/my-image
FATA[0001] Error fetching blob: invalid status code from registry 403 (Forbidden)

Resolution

When exposing the internal registry in ARO is needed to run an extra step to disable the redirection:

$ oc patch config.imageregistry.operator.openshift.io/cluster --patch='[{"op": "add", "path": "/spec/disableRedirect", "value": true}]' --type=json

Root Cause

Disclaimer: Links contained herein to an external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

The reason behind this is that the Image Registry uses an Azure Storage Account as a backend storage and that Azure Storage Account is Firewall'd off from the internet.

The default behavior for the Image Registry is to redirect the client to read/write directly to that backend. Since it is only accessible from within the ARO Cluster Networks, this means that we need to disableRedirect, in order for the Image Registry pods themselves proxy the connection to the storage account.

For more information, please, refer to the How are my storage accounts secured? section under the FAQ where this behavior is explained in more detail. A couple of interesting sentences could be:

  • Storage accounts are set to private access only.
  • Storage accounts access is limited with firewall rules via Azure network security groups (NSGs), which filter network traffic to and from your storage accounts.

The Configure the built-in container registry for Azure Red Hat OpenShift 4 contains more information about exposing the internal registry in Azure Red Hat OpenShift(ARO)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments