Firewall changes for container image pulls 2022/2023

Updated -

The Red Hat container image registries are changing, and this means docker / podman users who may pull images from the Red Hat registry may need to adjust firewall settings. Please be sure to make this adjustment by May 1, 2023. OpenShift users are not affected as they already accesses Quay.io since June 2022.

What to change

Currently all image manifests and filesystem blobs are served directly from registry.redhat.io and registry.access.redhat.com. The coming change will mean filesystem blobs are served from Quay.io instead. To avoid problems pulling container images, you will need to allow outbound TCP connections (ports 80 and 443) to these hostnames:

  • cdn.quay.io
  • cdn01.quay.io
  • cdn02.quay.io
  • cdn03.quay.io

This change should be made to any firewall configuration that specifically allows outbound connections to registry.redhat.io or registry.access.redhat.com. After making this change you will be able to continue pulling images from registry.redhat.io and registry.access.redhat.com as before. You do not need a Quay.io login, or to interact with the Quay.io registry directly in any way, in order to continue pulling Red Hat container images.

This change is needed for all products which may pull images from Red Hat registries. Outbound connections to these hosts may already be allowed in your firewall configuration as a result of having previously followed the OpenShift installation instructions, or due to otherwise needing to use the Quay.io registry. Other products synchronizing or downloading container images from the Red Hat registry, such as Red Hat Ansible Automation Platform (AAP) or Red Hat Satellite, may need changes to their relevant firewall or proxy to allow outbound access to the hosts listed above.

We recommend using the hostnames instead of IP addresses when configuring firewall rules. See this article for more information.

Why this change is coming

Since June 2022 the Red Hat OpenShift operator index images (redhat/redhat/operator-index) have been served from registry.redhat.io using Quay.io as the backend. OpenShift itself already needs access to the Quay.io registry and CDN hosts as explained in its installation instructions, and so this change required no action from customers at that time.

We are extending this to all Red Hat container images. This allows customers to benefit from the high availability of the Quay.io registry while simplifying the way Red Hat delivers container images and paving the way for future enhancements.

Testing

You can confirm image pulls will continue to work ahead of the registry change. To do so, pull the registry.redhat.io/redhat/redhat-operator-index:v4.12 image, which already has filesystem blobs hosted on Quay.io. To do this, run the following commands using your Customer Portal credentials:

# podman login registry.redhat.io
# podman pull registry.redhat.io/redhat/redhat-operator-index:v4.12
# echo $?

If the image was pulled successfully the echo $? command will show "0".

Additional information

Other than this change, many things are staying the same as before:

  • Container image pull specifications are unchanged, so you can continue to pull images from registry.redhat.io and registry.access.redhat.com.
  • Red Hat container images will continue to be signed in the same way, and with the same keys.
  • Container image manifests are served directly from registry.redhat.io and registry.access.redhat.com. Redirects to the Quay.io CDN are only for config and filesystem blobs.
  • Pulling an image by its sha256 digest must still be done using its schema 2 digest (see this earlier article).
  • Image tags, schema 2 digests, image IDs, and signatures remain unchanged.
  • Images pulled before the change is made remain valid and do not need to be re-pulled.
  • No changes relating to ImageContentSourcePolicy are needed for OpenShift or Kubernetes clusters.
  • For OpenShift or Kubernetes clusters, no node restarts, configuration changes, cache changes, or upgrades of any kind are needed.
  • The registry.connect.redhat.com registry is not affected by this change.

The Red Hat registries, registry.redhat.io and registry.access.redhat.com, return a header in the HTTP 302 redirect response which allows access only to specific content in the Quay.io CDN for a short period of time. Due to this mechanism, no image content can be pulled directly from the Quay.io CDN hosts – it must instead be pulled using a container registry. The Red Hat registries only provide access to Red Hat container images.

Allowing outbound connections to the hostnames mentioned above may resolve the following issues, depending on the characteristics of the firewall you use:

  • Connection refused when pulling images
  • I/O timeout when pulling images
  • ImagePullBackOff status when pulling images within an OpenShift or Kubernetes cluster

Here are example errors you might see from "podman pull" with different firewall configurations:

Trying to pull [...]...
WARN[0033] Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout
WARN[0065] Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout
WARN[0099] Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout
Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: i/o timeout

Trying to pull [...]...
WARN[0033] Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused
WARN[0065] Failed, retrying in 1s ... (2/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused
WARN[0099] Failed, retrying in 1s ... (3/3). Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused
Error: copying system image from manifest list: parsing image configuration: Get "https://cdn02.quay.io/sha256/[...]": dial tcp [...]: connect: connection refused

Related changes

The HTTP responses for requests for image manifests will include a docker-content-digest header, as described in the registry API specification.

While modern container client software uses the V2 schema 2 image manifest format, manifests in the older V2 schema 1 format are still available by tag. These have the same image content as before the change, but with some technical differences to the manifest formatting and filesystem layer blob compression. One result of this is a V2 schema 1 manifest pulled before the change will have a different manifest digest when compared to pulling from the same tag after the change. Another is that some filesystem layer blob digests will also be different before and after the change (despite identical uncompressed content). The "diff_ids" in the config blob refer to the uncompressed content and remain unchanged.

Fetching the list of tags in a repository will result in paginated responses: the first 50 tags will be returned initially, with a header indicating the URL to use for the next page. This pagination scheme is detailed in the Docker Registry HTTP API V2 specification and compliant tooling (such as skopeo) already understands how to use it.

Getting help

Your Red Hat account team or Red Hat partner is available for guidance. Alternatively, reach out to our support experts: https://access.redhat.com/support/.

Comments