must-gather service accounts (ClusterAdmin) and projects are not being deleted in OpenShift 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • must-gather

Issue

  • When running the must-gather tool, a ServiceAccount with the ClusterAdmin role and a project are automatically created. These temporary objects do not get deleted when the must-gather is aborted and (being a ClusterAdmin) could pop up in security scans, audits and penetration tests causing them to fail.
  • The namespace that is created by the oc adm must-gather command with the name of must-gather-xxxxx still exists in the cluster. Is this safe to remove it?

Resolution

Check for ClusterRoleBinding using ServiceAccounts with ClusterAdmin:

oc get clusterrolebindings -o json | jq -r '.items[] | select(.roleRef.name == "cluster-admin")|select(.subjects[0].name=="default") | .subjects[] | select(.kind == "User" or .kind == "Group" or .kind == "ServiceAccount") | .kind + ": " + .name +":" + .namespace'

Or:

$ oc get clusterrolebindings | grep must-gather

If there is any ClusterRoleBinding called must-gather-xxxxx, it is possible to delete it:

$ oc delete clusterrolebinding [must-gather-xxxxx]

Note: DO NOT delete any of the other ClusterRoleBindings or ServiceAccounts, as it can break the cluster. Starting with OpenShift 4.17, via OCPBUGS-31848, when the must-gather namespace is deleted, the ClusterRoleBinding is automatically deleted.

For removing the projects:

$ oc get projects | grep must-gather
$ oc delete project [must-gather-xxxxx]

Root Cause

When using the must-gather tool, a ServiceAccount and project are created to run it. The account has the ClusterAdmin role assigned but does not get deleted when the must-gather is aborted. The must-gather-xxxxx namespace is a temporary namespace that is only created to start the pod that collects all the logs for the must-gather command. Since it has no other purpose, it can be safely deleted.

Diagnostic Steps

  • Check for ClusterRoleBinding using ServiceAccounts with ClusterAdmin:

    $ oc get clusterrolebindings -o json | jq -r '.items[] | select(.roleRef.name == "cluster-admin")|select(.subjects[0].name=="default") | .subjects[] | select(.kind == "User" or .kind == "Group" or .kind == "ServiceAccount") | .kind + ": " + .name +":" + .namespace'

    The output can look like this:

    ServiceAccount: default:openshift-cluster-version   <--- DO NOT delete
ServiceAccount: default:openshift-network-operator  <--- DO NOT delete
ServiceAccount: default:openshift-must-gather-xxxx  <--- Can be deleted
ServiceAccount: default:openshift-must-gather-xxxx  <--- Can be deleted

  • For must-gather projects:

    $ oc get projects | grep must-gather

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments