ARO storage lockdown issues with the internal registry

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4
  • OpenShift Managed (Azure)
    • 4

Issue

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • There is a known side-effect of the image registry storage lockdown feature deployed that prevents some users from accessing the image registry externally, which will manifest as a "podman push/pull" or an "oc image mirror" AuthorizationFailure error:

    Trying to pull default-route-openshift-image-registry.apps.example.eastus.aroapp.io/openshift/tools:latest...
Error: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden)

Resolution

Establish that the issue is related to the storage lockdown, i.e. accessing the registry externally as per their usual workflow that now results in an AuthorizationFailure (see the Diagnostic Steps section). If that is the case, please patch the imageregistry in the ARO cluster using the following command:

$ oc patch config.imageregistry.operator.openshift.io cluster --type=json -p='[{"op": "add", "path": "/spec/disableRedirect", "value": true}]'

Root Cause

There is a known side-effect of the image registry storage lockdown feature deployed that prevents some users from accessing the image registry externally.

Diagnostic Steps

Pulling or pushing images to registry should fail with error as below:

$ podman pull default-route-openshift-image-registry.apps.example.eastus.aroapp.io/openshift/tools:latest 
Trying to pull default-route-openshift-image-registry.apps.example.eastus.aroapp.io/openshift/tools:latest...
Error: parsing image configuration: Error fetching blob: invalid status code from registry 403 (Forbidden)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments