Upgrade Disconnected ARO clusters

Solution Verified - Updated -

Environment

  • Azure Red Hat Openshift (ARO)
    • 4.x

Issue

  • While initiating ARO Disconnected Cluster upgrade using oc adm upgrade getting an error through CLI:
Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:518177a34452837920f1e77944f6afa08864537260c9f742b8c88b6157e4fXXX" failure=The update cannot be verified: unable to verify sha256:518177a34452837920f1e77944f6afa08864537260c9f742b8c88b6157e4fXXX against keyrings: verifier-public-key-redhat

  • Unable to upgrade ARO cluster using Private Azure Red Hat OpenShift cluster with no Internet Connectivity and ACR(Azure Container Registry)

  • How to upgrade disconnected ARO clusters

Resolution

Check upgrade path

NOTE: This step is VERY important. In a future step, you need to have already validated that the version you are upgrading to is safe to do so.

  1. First check which version your cluster is at:
oc get clusterversion version

Note the server version.

NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.40   True        False         14h     Cluster version is 4.10.40
  1. Verify you are selecting a valid version to upgrade to. Go to ocpupgradegraph
  • Under Channel, select the stable minor version that you want to upgrade the cluster to.
  • On the next screen, start by selecting the version your cluster is at.
  • Then select the version you want to upgrade to. Ensure there is a green line showing the upgrade path is recommended.

Upgrade the cluster

NOTE: In step 2 below, You are explicitly telling the cluster to upgrade to an image digest value and must use the --force flag because the cluster has no ability to validate the image digest value without Internet connectivity.
Please ensure you have completed the step to check the upgrade path so that you are upgrading the cluster to a version with a supported path from the current cluster version you’re on.

  1. Retrieve the image digest of the OpenShift version you want to upgrade to:
export VERSION=4.11.28 # Update to your desired version 
curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/"${VERSION}"/release.txt | grep "Pull From:"

Expected Output:

Pull From: quay.io/openshift-release-dev/ocp-release@sha256:85238bc3eddb88e958535597dbe8ec6f2aa88aa1713c2e1ee7faf88d1fefdac0
  1. Perform the Upgrade

Set the image to the desired values from the above command.

oc adm upgrade --allow-explicit-upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:1c3913a65b0a10b4a0650f54e545fe928360a94767acea64c0bd10faa52c945a --force
  1. Check the status of the scheduled upgrade
oc get clusterversion version

When the upgrade is completed you will see the following:

NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.28    True        False         161m    Cluster version is 4.11.28

Root Cause

  • One of the great features of ARO is that you can create ‘disconnected’ clusters with no connectivity to the Internet. Out of the box, the ARO service mirrors all the code repositories to build OpenShift clusters to Azure Container Registry. This means ARO is built without having to reach out to the Internet as the images to build OpenShift are pulled via the Azure private network.

  • Disconnected ARO cluster does not have internet access. Thus, we need to use appropriate targeted cluster version mirror image of desired cluster version from openshift mirror image link

Diagnostic Steps

$ oc adm upgrade

  Channel: stable-4.x
  warning: Cannot display available updates:
  Reason: RemoteFailed
  Message: Unable to retrieve available updates: Get "https://api.openshift.com/api/upgrades_info/v1/graph?
  arch=amd64&channel=stable-4.11&id=8ed36052-fbb4-40f5-a681-3a8969440a0c&version=4.11.x": dial tcp
  xx.xxx.xx.xxx:443: connect: connection timed out

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments