How to recover cluster-admin access to ARO clusters if you lose such permission?

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4.x

Issue

  • Accidentally overwrote the existing ClusterRoleBinding "cluster-admins" on the cluster.

  • I can login to the cluster with my kube:admin user, but not able to execute any operation. The following event is observed:

    Error from server (Forbidden): pods is forbidden: User "kube:admin" cannot list resource "pods" in API group "" in the namespace "default"

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  1. Using the az aro command, retrieve the kubeconfig file:

    $ az aro get-admin-kubeconfig --name <MyCluster> --resource-group <MyResourceGroup>

    Ref: az aro get-admin-kubeconfig

  2. It will generate a kubeconfig on your local machine:

    $ ls -l kubeconfig 
-rw-r--r--. 1 <user> <group> 4764 Jan 31 18:21 kubeconfig

  3. Export the kubeconfig file:

    $ export KUBECONFIG=<local_path>/kubeconfig

  4. From here you are logged in as system-admin, and can perform the recovery steps:

    $ oc whoami
system:admin

    Note: If for some reason you get the event saying "failed to verify certificate" as shown below:

    $ oc whoami
Unable to connect to the server: tls: failed to verify certificate: x509: certificate signed by unknown authority

    You can edit the kubeconfig file and add:

    - cluster:
    insecure-skip-tls-verify: true

Root Cause

  • The existing ClusterRoleBinding "cluster-admins", Subject kind Group, was overwritten on the cluster.

Diagnostic Steps

$ oc get nodes
Error from server (Forbidden): nodes is forbidden: User "kube:admin" cannot list resource "nodes" in API group "" at the cluster scope

$ oc get co
Error from server (Forbidden): clusteroperators.config.openshift.io is forbidden: User "kube:admin" cannot list resource "clusteroperators" in API group "config.openshift.io" at the cluster scope

$ oc get pods
Error from server (Forbidden): pods is forbidden: User "kube:admin" cannot list resource "pods" in API group "" in the namespace "default"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments