Unable to Delete a Project or Namespace

Solution Unverified - Updated -

Environment

Red Hat OpenShift Container Platform:

  • 3.11
  • 4.4+

Issue

  • oc delete project dev --force --grace-period=0 does not completely delete a project
  • "I am unable to delete a project"
  • The project is stuck in "Terminating" stage after deletion

Resolution

Troubleshoot and delete remaining resources

This usually happens because something is preventing a resource from being deleted, causing namespace deletion to be stuck. It is necessary to troubleshoot which resources are failing to be deleted and why.

A good troubleshooting approach would be:

  • Try to list all the items in the namespace with: oc api-resources --verbs=list --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n $PROJECT_NAME.
  • If the previous one fails, please try this one, which might not return a complete list but has less chance to fail: oc api-resources --verbs=list --cached --namespaced -o name | xargs -n 1 oc get --show-kind --ignore-not-found -n $PROJECT_NAME
  • Try manually removing every listed resource and, if one fails, troubleshoot why.
  • If all the listed resources are removed, try listing resources in the namespace with etcd. If there are still present resources, troubleshoot why

    • In OCP4, you can do it this way:

      [user@workstation ~]$ POD=`oc get pods -n openshift-etcd -o=jsonpath='{.items[0].metadata.name}'`
      [user@workstation ~]$  oc rsh -n openshift-etcd $POD -- ${ETCDCTL_COMMAND}
      sh-4.2# export ETCDCTL_ENDPOINTS=$(etcdctl member list | awk '{printf "%s%s",sep,$5; sep=","}') # BZ1832923
      sh-4.2# etcdctl get --keys-only --from-key / | grep $PROJECT_NAME
      
    • In OCP3, you can do it in a master this way: etcdctl3 get --keys-only --from-key / | grep $PROJECT_NAME

  • A good way to start troubleshooting is master controller logs. If the resource is a CRD managed by an operator, troubleshoot that operator.

In case of any issue, please open a support case to get assistance.

In many cases, deleting resources that could not be deleted in the first place ultimately leads to project to be no longer stuck in "Terminating" and be properly removed, but not always, so you may need to force its removal. It may be also possible that the project is waiting on the removal of an object with its own finalizer. If that is the case, and you are 100% sure of what you are doing, you can just remove the finalizer for that object. Both procedures are covered below, but please use them with caution and only if you know what you are doing.

Force individual object removal when it has finalizers - USE WITH CAUTION

Sometimes, a resource (specially a custom resource managed by an operator) may stay "terminating" waiting on a finalizer, although any needed cleanup tasks have been already completed, so it would be necessary to force its removal.

However, a very important warning: forcing the removal of an object without having properly cleaned it up may lead to unstable and unpredictable behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt. The impact would depend on each operator and what object is affected, but can be potentially high.

Only if you know what you are doing and you are 100% sure that any cleanup tasks for the object have been properly completed but still need to force its removal, you do it this way:

$ oc patch -n <project-name> <object-kind>/<object-name> --type=merge -p '{"metadata": {"finalizers":null}}'

Force namespace removal - USE WITH CAUTION

Sometimes, when a project has been stuck in "Terminating" state, even if all the resources have been properly removed afterwards, namespace may remain stuck in that state forever, so it becomes necessary to force its removal.

However, a very important warning: forcing the removal of a namespace without having properly cleaned it up may lead to unstable and unpredictable cluster behavior, so you must be 100% sure this is not the case and open a support case if you have even a minimum doubt.

Only if you know what you are doing and you are 100% sure that you have properly cleaned up any resource from the namespace but still need to force its removal, you can follow these steps to do so:

  • Confirm which namespace needs to be removed with oc get namespace
  • Create a temporary .json file: oc get namespace <failing namespace> -o json > tmp.json
  • Edit the file with your favorite text editor vim tmp.json
    • Remove the kubernetes value from the finalizers field and save the file.
  • Your tmp.json file should look similar to this:

    {
      "apiVersion": "v1",
      "kind": "Namespace",
      "metadata": {
          "annotations": {
              "openshift.io/description": "",
              "openshift.io/display-name": "",
              "openshift.io/requester": "system:admin",
              "openshift.io/sa.scc.mcs": "s0:c16,c15",
              "openshift.io/sa.scc.supplemental-groups": "1000270000/10000",
              "openshift.io/sa.scc.uid-range": "1000270000/10000"
          },
          "creationTimestamp": "2020-04-27T08:35:29Z",
          "deletionTimestamp": "2020-04-27T09:07:22Z",
          "name": "test",
          "resourceVersion": "3480943",
          "selfLink": "/api/v1/namespaces/test",
          "uid": "0d2d425c-8862-11ea-bce9-fa163eb0b490"
      },
      "spec": {
          "finalizers": []
      },
      "status": {
          "phase": "Terminating"
      }
    }
    
  • Setup a temporary proxy, please keep this terminal open until the namespace is deleted.

    $ oc proxy
    
  • In a new terminal window, replace with the name of the failing project/namespace and enter the following:

    $  curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/${PROJECT_NAME}/finalize
    
  • If you get authorization errors, you can also try running this on a master using certificates without the proxy command above:

    # curl --cacert /etc/origin/master/ca.crt --key /etc/origin/master/admin.key --cert /etc/origin/master/admin.crt -k  -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://127.0.0.1:8443/api/v1/namespaces/<terminating-namespace>/finalize
    

Root Cause

Every Kubernetes namespace has a kubernetes finalizer that prevents its final deletion when a delete on that namespace is requested. Reason is so that masters can delete the resources in the namespace before deleting the namespace itself.

However, many different reasons can lead to some of this resources to not be properly deleted. A typical example is the failure of an external apiservice (like service catalog).

This solution provides general guidance on how to troubleshoot this kind of situations as well as a procedure to force namespace deletion or the deletion of individual objects that can also be stuck on finalizers. But those procedures must be used as a last resort.

Diagnostic Steps

To double-check whether a project is stuck "Terminating", you can do the following:

First, attempt running:
$ oc delete project <project name> --force --grace-period=0

If this comes back with output of:

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "<project>": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.`

Then run:
$ oc get project <project name> -o yaml

If you see the following at the bottom of the output:

spec:
  finalizers:
  - kubernetes
status:
  phase: Terminating

The 'kubernetes' finalizer is keeping the project from deletion. You need to troubleshoot why and fix as per "resolution" section.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

1 Comments

Getting error:

:namespaces \namespace\ is forbidden: User \"system:anonymous\" cannot update namespaces/finalize in the namespace \"jenkins\": no RBAC policy attached.

But when oc whoami command ran

cluster-admin

and then attempted from system:admin account