Upgrade playbook from 3.10 to 3.11 fails in task openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered

Solution Verified - Updated -

Issue

When running the control plane upgrade playbook, the task "openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered" fails with the error:

Error from server (ServiceUnavailable): the server is currently unable to handle the request

Checking the metrics server in openshift-metrics-server namespace, it is in CrashLoopBackOff state:

# oc get pods -n openshift-metrics-server
NAME                              READY     STATUS             RESTARTS   AGE
metrics-server-544d656869-sjvq9   0/1       CrashLoopBackOff   33         2h

The logs in the metrics-server pod show messages about the pod not being able to read the configmap extension-apiserver-authentication in the kube-system namespace:

W0905 11:47:02.343499       1 authentication.go:220] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
F0905 11:47:02.343537       1 heapster.go:97] Could not create the API server: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:openshift-metrics-server:metrics-server" cannot get configmaps in the namespace "kube-system": User "system:serviceaccount:openshift-metrics-server:metrics-server" cannot get configmaps in project "kube-system": role.rbac.authorization.k8s.io "extension-apiserver-authentication-reader" not found

The above message recommends the creation of a rolebinding, but it already exists:

# oc get rolebinding metrics-server-auth-reader -n kube-system
NAMESPACE      NAME                                         ROLE       ...
kube-system       metrics-server-auth-reader   kube-system/extension-apiserver-authentication-reader   ...

However the role extension-apiserver-authentication-reader in the kube-system namespace is missing:

# oc get role extension-apiserver-authentication-reader -n kube-system 
Error from server (NotFound): roles.authorization.openshift.io "extension-apiserver-authentication-reader" not found

Environment

  • Red Hat Openshift Container Platform
    • 3.10
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content