Hosted Control Planes stuck in ValidReleaseImage because of Container Image rewrite failure

Solution Verified - Updated -

Issue

  • In disconnected environment, the Image Registry is configured for pullthrough and the ImageDigestMirrorSet is configured with the upstream URL as part of the repository like registry.example.org/quay.io/openshift-release-dev/ocp-v4.0-art-dev. When updating the HostedCluster resource, conciling of the resource starts to failing because multiple Image Registry rewrites are taking place as observed in the control-plane-operator logs:

    {"level":"error","ts":"2025-09-23T16:38:55Z","msg":"reconciliation is blocked","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","HostedCluster":{"name":"demo","namespace":"clusters"},"namespace":"clusters","name":"demo","reconcileID":"4cc37694-2b8e-4205-a3df-e21c9256d7fb","message":"failed to lookup release image: failed to extract release metadata: failed to obtain root manifest for registry.example.org/registry.example.org/registry.example.org/quay.io/openshift-release-dev/ocp-release@sha256:2db093f063ad5310fa4e5ed2d2eda4bad5215c47092b72d1cfafbcfdbf1f4dd2: unauthorized: project registry.example.org not found: project registry.example.org not found","error":"release image is invalid","stacktrace":"github.com/openshift/hypershift/hypershift-operator/controllers/hostedcluster.(*HostedClusterReconciler).reconcile\n\t/hypershift/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go:1200\ngithub.com/openshift/hypershift/hypershift-operator/controllers/hostedcluster.(*HostedClusterReconciler).Reconcile\n\t/hypershift/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go:343\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"}
    
  • Reconciling or Updates of HostedCluster resource is failing, when the ImageDigestMirrorSet does contain the actual upstream Image Registry URL as part of the repository definition like registry.example.org/quay.io/openshift-release-dev/ocp-v4.0-art-dev.

Environment

  • Red Hat OpenShift Container Platform 4 - Hosted Control Planes

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content