ROSA Cluster Deletion Hangs when AWS Load Balancer Deletion Protection is enabled

Solution Verified - Updated -

Environment

Red Hat OpenShift Service on AWS (ROSA)

Issue

  • ROSA Cluster deletion hangs during the deletion causing some resources not to be deleted.

Sample Errors:

time=\"2023-03-22T21:55:39Z\" level=debug msg=\"OperationNotPermitted: Load balancer 'arn:aws:elasticloadbalancing:us-XXXX-1:XXXXXXXX:loadbalancer/net/XXXXXXXX-XXXX-int/XXXXXXXXXXX' cannot be deleted because deletion protection is enabled\\n\\tstatus code: 400, request id: 950ffb66-a41b-48b8-a934-08bb3c272e18\" arn=\"arn:aws:elasticloadbalancing:us-XXXX-1:XXXXXXXXXX:loadbalancer/net/XXXXXXXXXX-XXXX-int/XXXXXXXXXX\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"InvalidParameterValue: This snapshot is managed by the AWS Backup service and cannot be deleted via EC2 APIs. If you wish to delete this snapshot, please do so via the Backup console.\\n\\tstatus code: 400, request id: 229ab527-3587-49c8-a347-5a827b2689a4\" arn=\"arn:aws:ec2:us-XXXX-1:XXXXXXXX:snapshot/snap-XXXXXXXXXX\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"tagging snapshots for ami-imageXXXXX: InvalidParameterValue: Value ( aws:backup:source-resource ) for parameter key is invalid. Tag keys starting with 'aws:' are reserved for internal use\\n\\tstatus code: 400, request id: 924c824b-bc99-4ab2-8685-1fd1d1891198\" arn=\"arn:aws:ec2:us-XXXX-1:XXXXXXXX:image/ami-imageXXXXX\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"search for matching resources by tag in us-XXXX-1 matching aws.Filter{\\\"kubernetes.io/cluster/XXXXXXXX-XXXX\\\":\\\"owned\\\"}\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"search for matching resources by tag in us-XXXX-1 matching aws.Filter{\\\"openshiftClusterID\\\":\\\"XXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX\\\"}\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"search for IAM roles\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"iterating over a page of 100 IAM roles\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"iterating over a page of 15 IAM roles\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"search for IAM users\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"iterating over a page of 1 IAM users\"\ntime=\"2023-03-22T21:55:39Z\" level=debug msg=\"search for IAM instance profiles\"\n"

Resolution

  • Check with AWS Account Administrator if they can remove the deletion protection of that resource so ROSA
    cluster uninstall can proceed.

Root Cause

  • The ROSA cli leverages Terraform to construct the ROSA cluster, and in doing so, it utilizes certain AWS API
    modules in the background for the installation and removal of the cluster. However, if a certain property
    was modified manually by the administrator or some custom automation outside of the normal ROSA installation, it
    may cause some issues when trying to delete your cluster since Terraform is unaware of this resource's
    changes.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments