Failed to pull image from azurecr.io domain in ARO

Solution Unverified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4
  • OpenShift Managed (Azure)
    • 4

Issue

  • The ARO cluster is trying to pull images from arosvc.azurecr.io or arosvc.[azure_region].data.azurecr.io, but that registries are not configured.

  • There are the following errors in the logs trying to pulling images to arosvc.azurecr.io or arosvc.[azure_region].data.azurecr.io:

    Failed to pull image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rpc error: code = Unknown desc = Error parsing image configuration: Get https://arosvc.[azure_region].data.azurecr.io?[...]

    dial tcp: lookup arosvc.[azure_region].data.azurecr.io on 10.0.0.1:53: no such host

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

The arosvc.azurecr.io and arosvc.[azure_region].data.azurecr.io registries are mirrors of quay.io used by ARO clusters, and it's needed to allow the access to those registries from the ARO cluster.

Refer to Minimum Required FQDN / application rules and allow the access to those FQDN /applications. Also, refer to Troubleshoot network issues with registry for additional information about the container registry troubleshooting in Azure.

Root Cause

In ARO clusters, the quay.io registry is mirrored by arosvc.azurecr.io and/or arosvc.[azure_region].data.azurecr.io, and it's needed to allow the access from the cluster.

Diagnostic Steps

Check the registry mirrors configured in the ARO cluster:

$ oc get imagecontentsourcepolicies -o yaml
[...]
  spec:
    repositoryDigestMirrors:
    - mirrors:
      - arosvc.azurecr.io/openshift-release-dev/ocp-release
      source: quay.io/openshift-release-dev/ocp-release
[...]
  spec:
    repositoryDigestMirrors:
    - mirrors:
      - arosvc.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev
      source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments