Restricting cluster-admin permissions in a multi-tenant OpenShift cluster

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (OCP 4)
  • Red Hat OpenShift Service on AWS (ROSA 4)
  • Red Hat Openshift Dedicated 4 (OSD 4)
  • Azure Red Hat Openshift (ARO 4)

Issue

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Cluster-admin users by default have access and permission to do all operations on all resources in the cluster. In a multi-tenant cluster, it is required to block an action performed by cluster admins on certain resources. We can't do it with RBAC, it only allows for adding of permissions, not taking them away.

Resolution

We can leverage an admission controller to block the request to the API. To implement this, we can make use of Kyverno (which deploys an admission controller) and use it for a lot of validation and defaulting of resources in our cluster. Not only can Kyverno validate and block based on a resource's values but it can also validate the operation being performed, and by who.

We can deploy a cluster policy that would block cluster-admin (and anyone else) from performing any specific action on a particular resource.

Steps to implement this solution:

  1. Install kubectl by following the instructions here
  2. Install kyverno by following the instructions here
  3. Deploy a ClusterPolicy resource to deny any action (for ex: delete) request on the required resource.

For example:

Consider a scenario where we need to block every user (including cluster-admin users) from deleting a particular resource. This can be achieved by deploying a cluster policy that would block cluster-admin (and anyone else) from performing delete action on that resource.

Here is the ClusterPolicy yaml to deny delete request for a particular resource:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-resource-deletion-in-namespace
  annotations:
    policies.kyverno.io/title: Block resource deletion
    policies.kyverno.io/subject: DaemonSet
    policies.kyverno.io/description: >-
      Restrict cluster-admins or any other users from deleting resources
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: block-resource-deletion
    match:
      any:
      - resources:       <----- //Kind of resource we want this policy to apply to
          kinds: 
          - Pod 
          - DaemonSet
    preconditions:       <----- //setting a precondition that allows us to do some filtering based on the details of the request
      any:
      - key: "{{ request.object.metadata.name || '' }}"
        operator: Equals
        value: "<Name of the object>"
    validate:
      message: "Here, you can write any message that you want to get displayed when someone performs this action"
      deny:
        conditions:
        - key: "{{request.operation}}"
          operator: In
          value:
          - DELETE
  1. Create a YAML file similar to the one shown above.
  2. Create a cluster policy with the help of this file.
oc create -f filename.yaml

Note: Once the cluster policy gets deployed, any delete request will automatically get denied.

Other use cases:

  • A policy that prevents deletion of any resource that has a do-not-delete: "true" annotation on it to prevent accidental deletion of critical resources (such as persistent volumes or secrets).

  • A policy that prevents fetching the details of secrets in a specific namespace

  • A policy that blocks deletes, updates or patches from everyone except a specific user that can be used to prevent others 'cleaning up' resources.

Root Cause

RBAC provides a strong method of providing permissions to users, groups, or service accounts within a cluster. But what if we need to block an action performed by cluster admins? We can't do it with RBAC, it only allows for adding of permissions, not taking them away.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments