Unable to Delete a Project or Namespace in OCP

Resolution

Please try to troubleshoot and delete remaining resources.
Do not force removals unless you know what you are doing.

Troubleshoot and delete remaining resources

This usually happens because something is preventing a resource from being deleted, causing namespace deletion to be stuck. It is necessary to troubleshoot which resources are failing to be deleted and why.

A good troubleshooting approach would be:

• Check the output of command oc api-resources. If it fails, check Projects stuck in Terminating state and unable to run "oc api-resources" on OpenShift.

• Try to list all the items in the namespace with the following command:

  oc api-resources --verbs=list --namespaced -o name | xargs -t -n 1 oc get --show-kind --ignore-not-found -n $PROJECT_NAME
  

• If the previous one fails, please try this one, which might not return a complete list but has less chance to fail:

  oc api-resources --verbs=list --cached --namespaced -o name | xargs -t -n 1 oc get --show-kind --ignore-not-found -n $PROJECT_NAME
  

• Try manually removing every listed resource and, if one fails, troubleshoot why.

• If all the listed resources are removed, try listing resources in the namespace with etcd. If there are still present resources, troubleshoot why

  • In OCP 4, you can do it this way:

    [user@workstation ~]$ POD=`oc get pods -n openshift-etcd -o=jsonpath='{.items[0].metadata.name}'`
    [user@workstation ~]$  oc rsh -n openshift-etcd -c etcdctl $POD 
    sh-4.2# etcdctl get --keys-only --from-key / | grep $PROJECT_NAME
    

  • In OCP 3, you can do it in a master this way:

    etcdctl3 get --keys-only --from-key / | grep $PROJECT_NAME
    

• A good way to start troubleshooting is master controller logs. If the resource is a CRD managed by an operator, troubleshoot that operator.

Important: In case of any issue, please open a support case to get assistance.

In many cases, deleting resources that could not be deleted in the first place ultimately leads to project to be no longer stuck in "Terminating" and be properly removed, but not always, so you may need to force its removal. It may be also possible that the project is waiting on the removal of an object with its own finalizer. If that is the case, and you are 100% sure of what you are doing, you can just remove the finalizer for that object. Both procedures are covered below, but please use them with caution and only if you know what you are doing.

Force individual object removal when it has finalizers - USE WITH CAUTION

Sometimes, a resource (specially a custom resource managed by an operator) may stay "terminating" waiting on a finalizer, although any needed cleanup tasks have been already completed, so it would be necessary to force its removal.

However, a very important warning: forcing the removal of an object without having properly cleaned it up may lead to unstable and unpredictable behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt. The impact would depend on each operator and what object is affected, but can be potentially high.

Only if you know what you are doing and you are 100% sure that any cleanup tasks for the object have been properly completed but still need to force its removal, you do it this way:

$ oc patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

Force namespace removal - USE WITH CAUTION

Sometimes, when a project has been stuck in "Terminating" state, even if all the resources have been properly removed afterwards, namespace may remain stuck in that state forever, so it becomes necessary to force its removal.

However, a very important warning: forcing the removal of a namespace without having properly cleaned it up may lead to unstable and unpredictable cluster behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt.

Only if you know what you are doing and you are 100% sure that you have properly cleaned up any resource from the namespace but still need to force its removal, you can follow these steps to do so:

• Confirm which namespace needs to be removed with oc get namespace
• Create a temporary .json file: oc get namespace <failing namespace> -o json > tmp.json
• Edit the file with your favorite text editor vim tmp.json
  • Remove the kubernetes value from the finalizers field and save the file.

• Your tmp.json file should look similar to this:

  {
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
            "openshift.io/description": "",
            "openshift.io/display-name": "",
            "openshift.io/requester": "system:admin",
            "openshift.io/sa.scc.mcs": "s0:c16,c15",
            "openshift.io/sa.scc.supplemental-groups": "1000270000/10000",
            "openshift.io/sa.scc.uid-range": "1000270000/10000"
        },
        "creationTimestamp": "2020-04-27T08:35:29Z",
        "deletionTimestamp": "2020-04-27T09:07:22Z",
        "name": "test",
        "resourceVersion": "3480943",
        "selfLink": "/api/v1/namespaces/test",
        "uid": "0d2d425c-8862-11ea-bce9-fa163eb0b490"
    },
    "spec": {
        "finalizers": []
    },
    "status": {
        "phase": "Terminating"
    }
  }
  

• Setup a temporary proxy, please keep this terminal open until the namespace is deleted.

  $ oc proxy
  

• In a new terminal window, replace with the name of the failing project/namespace and enter the following:

  $  curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${PROJECT_NAME}/finalize
  

• If you get authorization errors, you can also try running this on a master using certificates without the proxy command above:

  # curl --cacert /etc/origin/master/ca.crt --key /etc/origin/master/admin.key --cert /etc/origin/master/admin.crt -k  -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://127.0.0.1:8443/api/v1/namespaces/<terminating-namespace>/finalize