Unable to cancel an in-progress OpenShift 4 cluster upgrade

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Azure Red Hat OpenShift (ARO)
    • 4

Issue

  • When attempting to cancel the upgrade, received an error indicating that the upgrade policy is locked and cannot be halted mid-process:

    Danger alert:Error cancelling update
CLUSTERS-MGMT-400: Upgrade policy '1993039c-XXXX-XXXX-XXXX-0a580a830611' is locked due to upcoming upgrade at '2023-09-06 20:14:09.165 +0000 UTC'
Operation ID: 253c6e5a-XXXX-XXXX-XXXX-1a0996cea482

Resolution

An upgrade can only be canceled if it is still in the download phase, which usually lasts a few seconds (depending on the cluster network bandwidth), with the following command:

$ oc adm upgrade --clear

Note that if the cluster upgrade has started, it cannot be stopped or reverted mid-process, and that is a design feature to maintain the stability and integrity of the cluster. The error message "Upgrade policy 'xxxxxx' is locked due to upcoming upgrade at" indicates that the upgrade policy is locked and cannot be altered at this stage.

If the upgrade cannot be canceled

  1. Monitor the upgrade process: Keep a close eye on the upgrade process using commands like $ oc adm upgrade to track the progress.

  2. Prepare for post-upgrade checks: Plan and prepare for necessary post-upgrade checks or configurations to ensure the smooth operation of your environment once the upgrade is completed.

  3. Communication with Red Hat Support: In case of any concerns or issues, Open a support case with the Red Hat Support team for assistance.

Root Cause

The OpenShift cluster upgrade process is designed to be irreversible once initiated to maintain the stability and security of the cluster. This design prevents potential issues that might arise from halting or reverting an upgrade midway.

From the official doumentation:

Only updating to a newer version is supported. Reverting or rolling back your cluster to a previous version is not supported. If your update fails, contact Red Hat support.

Diagnostic Steps

Use the oc adm upgrade command to monitor the upgrade status and progress.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments