Unable to delete backend Azure disk after accidently first deleting the PV in ARO

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift [ARO]
    • 4.x

Issue

  • The PV was deleted first instead of PVC, which left behind the backend Azure disk.
  • The backend disks in Azure cannot be manually deleted due to Deny Assignment that is configured by design.
  • Can't delete the disks from Azure as the Deny Assignment on ARO Cluster resource group is preventing it.

Resolution

  • It is not recommended to force deleting resources in general. It can result in situations where resources may not exist in OpenShift, but they still exist in the Cloud infrastructure.

  • In order to have the backend Azure disks deleted, please follow the steps below:

    1. Create a new PV by using the attached file 'pv.yaml' as a reference. Make sure you provide the correct Azure Disk details, and also the persistentVolumeReclaimPolicy as 'Delete'.

      $ oc create -f pv.yaml

    2. Check that the PV is created and is in phase 'Available', referencing the right disk.

      $ oc get pv

    3. Create a new PVC by using the the attached file 'pvc.yaml' as a reference.

      $ oc create -f pvc.yaml

    4. Check that the PVC is well created and that the PV is now in phase 'Bound'.

      $ oc get pvc -n <namespace>

    5. Delete the new PVC created.

      $ oc delete pvc <pvc_name> -n <namespace>

    6. Check that the PVC, PV and Disk are deleted altogether.

Root Cause

  • Typically, PVCs should be deleted first, then kube-controller-manager would perform the appropriate steps to delete the related PV following the disk, ensuring that the PVC is not in use before actually deleting it. Deleting the PV before the PVC results in the corresponding Azure disk never being deleted.
  • In ARO, we have a Deny Assignment in place by design to prevent actions taken by users from impacting the health of the cluster (such as deleting or modifying a VM, disk, load balancer, NSG, etc.). This prevents from deleting the disks manually.

Diagnostic Steps

  • Confirm that the PV and PVC are not available for the related Azure disk, which is still online, but detached:
$ oc get pv <pv_name>
$ oc get pvc <pvc_name> -n <namespace>

Attachments

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments