Hosted Control Planes stuck in ValidReleaseImage because of Container Image rewrite failure
Issue
-
In disconnected environment, the Image Registry is configured for pullthrough and the
ImageDigestMirrorSetis configured with the upstream URL as part of the repository likeregistry.example.org/quay.io/openshift-release-dev/ocp-v4.0-art-dev. When updating theHostedClusterresource, conciling of the resource starts to failing because multiple Image Registry rewrites are taking place as observed in thecontrol-plane-operatorlogs:{"level":"error","ts":"2025-09-23T16:38:55Z","msg":"reconciliation is blocked","controller":"hostedcluster","controllerGroup":"hypershift.openshift.io","controllerKind":"HostedCluster","HostedCluster":{"name":"demo","namespace":"clusters"},"namespace":"clusters","name":"demo","reconcileID":"4cc37694-2b8e-4205-a3df-e21c9256d7fb","message":"failed to lookup release image: failed to extract release metadata: failed to obtain root manifest for registry.example.org/registry.example.org/registry.example.org/quay.io/openshift-release-dev/ocp-release@sha256:2db093f063ad5310fa4e5ed2d2eda4bad5215c47092b72d1cfafbcfdbf1f4dd2: unauthorized: project registry.example.org not found: project registry.example.org not found","error":"release image is invalid","stacktrace":"github.com/openshift/hypershift/hypershift-operator/controllers/hostedcluster.(*HostedClusterReconciler).reconcile\n\t/hypershift/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go:1200\ngithub.com/openshift/hypershift/hypershift-operator/controllers/hostedcluster.(*HostedClusterReconciler).Reconcile\n\t/hypershift/hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go:343\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:224"} -
Reconciling or Updates of
HostedClusterresource is failing, when theImageDigestMirrorSetdoes contain the actual upstream Image Registry URL as part of the repository definition likeregistry.example.org/quay.io/openshift-release-dev/ocp-v4.0-art-dev.
Environment
- Red Hat OpenShift Container Platform 4 - Hosted Control Planes
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.