Cluster upgrade of disconnected environment fails when using container images from another mirror repository in OpenShift 4.

Issue

• Cluster upgrade of disconnected environment fails when using container images from another mirror repository.

• A different repository is created in the mirror registry from the repository that mirrored the current version of the container image, and the container image of the version to be upgraded has been mirrored to the repository. (Separate repository paths for each version) Running oc adm upgrade with the --to-image option specifying the repository image, the upgrade process does not progress and does not complete.

  $ curl -L -H  "Authorization: Bearer <token> " ${LOCAL_REGISTRY}/v2/_catalog | jq .
  {
    "repositories":[
      "{LOCAL_REPOSITORY_A}",           <---- the repository that mirrored the current version
      "{LOCAL_REPOSITORY_B}"            <---- the repository that mirrored the upgrade version
    ]
  }
  
  $ oc adm upgrade --allow-explicit-upgrade --to-image ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY_B}@<sha256>    <---- specify the repository that mirrored the upgrade version
  
  $ oc get clusterversion
  NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
  version   4.X.Y            True                True                        1h           Working towards 4.X.Z:  AAA of BBB done ( XX % complete)    <---- the upgrade process does not progress.
  

• After running oc adm upgrade, the status of the etcd-operator pod and the kube-apiserver-operator pod has been ImagePullBackOff.

  $ oc get pods -n openshift-etcd-operator
  NAME                             READY   STATUS             RESTARTS   AGE
  etcd-operator-XXXXXXXXXXXXXXXX   0/1     ImagePullBackOff   0          11h
  
  $ oc get pods -n openshift-kube-apiserver-operator
  NAME                                      READY   STATUS             RESTARTS   AGE
  kube-apiserver-operator-XXXXXXXXXXXXXXX   0/1     ImagePullBackOff   0          12h
  
  $ oc get events -n openshift-etcd-operator
  LAST SEEN   TYPE      REASON    OBJECT                               MESSAGE
  30m         Warning   Failed    pod/etcd-operator-XXXXXXXXXXXXXXXX   Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>": rpc error: code = Unknown desc = (Mirrors also failed: [{LOCAL_REGISTRY}/{LOCAL_REPOSITORY_A}@<sha256>: reading manifest <sha256> in {LOCAL_REGISTRY}/{LOCAL_REPOSITORY_A}: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 3.224.198.181:443: i/o timeout
  5m          Normal    BackOff   pod/etcd-operator-XXXXXXXXXXXXXXXX   Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>"
  
  $ oc get events -n openshift-kube-apiserver-operator
  LAST SEEN   TYPE      REASON    OBJECT
  33m         Warning   Failed    pod/kube-apiserver-operator-XXXXXXXXXXXXXXX   Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>": rpc error: code = Unknown desc = (Mirrors also failed: [{LOCAL_REGISTRY}/{LOCAL_REPOSITORY_A}@<sha256>: reading manifest <sha256> in {LOCAL_REGISTRY}/{LOCAL_REPOSITORY_A}: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 54.144.203.57:443: i/o timeout
  8m          Normal    BackOff   pod/kube-apiserver-operator-XXXXXXXXXXXXXXX   Back-off pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@<sha256>"