Pre-upgrade checks for ARO clusters

Prior to upgrading your Azure Red Hat OpenShift cluster, it is advisable to perform pre-upgrade checks as part of your planning.

Red Hat's Managed Cloud Services team offers pre-upgrade checks for ARO clusters by prior arrangement based on analysis of Customer-provided must-gather archives, however recent feature enhancements of the oc CLI now provide a self-service option for identifying issues that may prevent a cluster upgrade from commencing.

If it is intended to upgrade multiple minor versions sequentially, for example 4.15 to 4.16 and then to 4.17, the please consult the OpenShift Container Platform update graph to identify the recommended intermediate release.

Self-service checks using oc CLI

1. Download the latest OpenShift client command-line interface (oc) from the Red Hat Hybrid Cloud Console Downloads section.
2. Ensure that you set the upgrade channel to the target channel:

$ oc adm upgrade channel stable-4.16

3. Run the upgrade check:

$ oc adm upgrade

Note: this is a read-only check for any DEGRADED=True states of cluster operators and does not initiate any changes to your cluster.

4. View the cluster status and available updates.

• Example output with no issues found:

Cluster version is 4.15.49

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.16 (available channels: candidate-4.15, candidate-4.16, eus-4.16, fast-4.15, fast-4.16, stable-4.15, stable-4.16)

Recommended updates:

  VERSION     IMAGE
  4.16.45     quay.io/openshift-release-dev/ocp-release@sha256:6d097cefdcc369ba328e25fdfbfcdeaabcecad38be4b3c12a99f2222f384a31c
  4.16.44     quay.io/openshift-release-dev/ocp-release@sha256:f41be65a929742f584bb4299dfa48135b093b3cc86dc76c3ab77265574bdd4fe
  4.16.43     quay.io/openshift-release-dev/ocp-release@sha256:0ac57a43408916ba0b3812d3d3e8beb2a8451cb0cf1f0e23bf96f65896ff426a
  4.16.42     quay.io/openshift-release-dev/ocp-release@sha256:2fa965ede9ab6530be8a2e780b484d487c2b3ea4354e84cec680c22823c255ab
  4.16.41     quay.io/openshift-release-dev/ocp-release@sha256:e6d37b140f32d560dcdac97e9a0a34f1ec1a96af6f987f769eaa801369f1f610
  4.16.40     quay.io/openshift-release-dev/ocp-release@sha256:77260d121db0e3d6705d60a57b0da0b2063a3674c889b519f37567b7f7969b5d
  4.16.39     quay.io/openshift-release-dev/ocp-release@sha256:2754cd66072e633063b6bf26446978102f27dd19d4668b20df2c7553ef9ee4cf
  4.15.55     quay.io/openshift-release-dev/ocp-release@sha256:3ff7652a165409a368a7cdc748c95efc17db0761d467cc85fb355cce14643602
  4.15.54     quay.io/openshift-release-dev/ocp-release@sha256:73a9d922d313f952675151d4e1ecf266ec41786cafc2b56e0f7dba890d6affdd
  4.15.53     quay.io/openshift-release-dev/ocp-release@sha256:709a70750bd6b88c32c679fee9459e323e3e0e9d6d3f186f28a46e3010d1cc7d
  4.15.52     quay.io/openshift-release-dev/ocp-release@sha256:bc37f63e71665274a0e71baa8013f21ee67f163480fced1eeab843b3ab25196d
  4.15.51     quay.io/openshift-release-dev/ocp-release@sha256:9fdf7a2305973c26b4075ce1a16b295dbe25092b8ed0ae21506d93cafb6dd9a9
  4.15.50     quay.io/openshift-release-dev/ocp-release@sha256:f0421eafd1ab27c694f6bfc6ec8a8641636030485a3a4c80071ce7fa1a1bf04c

The current cluster version is reported, along with the current release channel and a list of versions that are available.

• Example output with only API deprecations to acknowledge:

Cluster version is 4.15.49

Upgradeable=False

  Reason: AdminAckRequired
  Message: Kubernetes 1.29 and therefore OpenShift 4.16 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/7031404 for details and instructions.

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.16 (available channels: candidate-4.15, candidate-4.16, eus-4.16, fast-4.15, fast-4.16, stable-4.15, stable-4.16)

Recommended updates:

  VERSION     IMAGE
  4.16.45     quay.io/openshift-release-dev/ocp-release@sha256:6d097cefdcc369ba328e25fdfbfcdeaabcecad38be4b3c12a99f2222f384a31c
  4.16.44     quay.io/openshift-release-dev/ocp-release@sha256:f41be65a929742f584bb4299dfa48135b093b3cc86dc76c3ab77265574bdd4fe
  4.16.43     quay.io/openshift-release-dev/ocp-release@sha256:0ac57a43408916ba0b3812d3d3e8beb2a8451cb0cf1f0e23bf96f65896ff426a
  4.16.42     quay.io/openshift-release-dev/ocp-release@sha256:2fa965ede9ab6530be8a2e780b484d487c2b3ea4354e84cec680c22823c255ab
  ...

Note: the state Upgradeable=False is a result of API removals/deprecations, which are not associated with every minor release upgrade - please check this article for more information.

• Example output with issues requiring attention:

Cluster version is 4.15.49

Upgradeable=False

  Reason: DegradedPool
  Message: Cluster operator machine-config should not be upgraded between minor versions: One or more machine config pools are degraded, please see `oc get mcp`
 for further details and resolve before upgrading

ReleaseAccepted=False

  Reason: PreconditionChecks
  Message: Preconditions failed for payload loaded version="4.16.45" image="quay.io/openshift-release-dev/ocp-release@sha256:6d097cefdcc369ba328e25fdfbfcdeaabce
cad38be4b3c12a99f2222f384a31c": Precondition "ClusterVersionUpgradeable" failed because of "DegradedPool": Cluster operator machine-config should not be upgrade
d between minor versions: One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading

Upstream is unset, so the cluster will use an appropriate default.
Channel: stable-4.16 (available channels: candidate-4.15, candidate-4.16, eus-4.16, fast-4.15, fast-4.16, stable-4.15, stable-4.16)

Recommended updates:

  VERSION     IMAGE
  4.16.45     quay.io/openshift-release-dev/ocp-release@sha256:6d097cefdcc369ba328e25fdfbfcdeaabcecad38be4b3c12a99f2222f384a31c
  4.16.44     quay.io/openshift-release-dev/ocp-release@sha256:f41be65a929742f584bb4299dfa48135b093b3cc86dc76c3ab77265574bdd4fe
  4.16.43     quay.io/openshift-release-dev/ocp-release@sha256:0ac57a43408916ba0b3812d3d3e8beb2a8451cb0cf1f0e23bf96f65896ff426a
  4.16.42     quay.io/openshift-release-dev/ocp-release@sha256:2fa965ede9ab6530be8a2e780b484d487c2b3ea4354e84cec680c22823c255ab
  ...

In this example, the state Upgradeable=False is set because the machine-config cluster operator is showing that a machine config pool is degraded. Once this issue is resolved and none of the cluster operators are degraded, then the cluster upgrade command can be issued and the upgrade will commence.

5. Optionally, check for the recommended upgrade versions in the target channel from the current release:

$ export OC_ENABLE_CMD_UPGRADE_RECOMMEND=true
$ oc adm upgrade recommend

The recommend option of oc CLI is currently a Technology Preview feature.

6. Check that there are no restrictive Pod Disruption Budget (PDB) defined in workload namespaces:

$ oc get pdb -A

Review any PDB that is overly restrictive, as it may slow the compute node update process.

Note: it is expected to see some PDB entries for openshift-* namespaces, as these can be safely ignored.

Performing pre-upgrade checks manually

1. Check all the cluster operators for any that may be in DEGRADED=True state:

$ oc get co

2. Check the status of Machine Config Pools:

$ oc get mcp

3. Check that there are no restrictive Pod Disruption Budget (PDB) defined in workload namespaces:

$ oc get pdb -A

4. Any queries regarding this output should be raised with Red Hat Customer Support via a proactive case 7-14 days in advance of the planned upgrade date - see the associated article here for more information.

Operator compatibility

For any additional operators installed from the OperatorHub, verify that the operator versions are compatible with the OpenShift target version by using the Operator checker and consult the OpenShift Operator Life Cycles for more information.