Upgrade playbook from 3.10 to 3.11 fails in task openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered
Issue
When running the control plane upgrade playbook, the task "openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered" fails with the error:
Error from server (ServiceUnavailable): the server is currently unable to handle the request
Checking the metrics server in openshift-metrics-server namespace, it is in CrashLoopBackOff state:
# oc get pods -n openshift-metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-544d656869-sjvq9 0/1 CrashLoopBackOff 33 2h
The logs in the metrics-server pod show messages about the pod not being able to read the configmap extension-apiserver-authentication in the kube-system namespace:
W0905 11:47:02.343499 1 authentication.go:220] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
F0905 11:47:02.343537 1 heapster.go:97] Could not create the API server: configmaps "extension-apiserver-authentication" is forbidden: User "system:serviceaccount:openshift-metrics-server:metrics-server" cannot get configmaps in the namespace "kube-system": User "system:serviceaccount:openshift-metrics-server:metrics-server" cannot get configmaps in project "kube-system": role.rbac.authorization.k8s.io "extension-apiserver-authentication-reader" not found
The above message recommends the creation of a rolebinding, but it already exists:
# oc get rolebinding metrics-server-auth-reader -n kube-system
NAMESPACE NAME ROLE ...
kube-system metrics-server-auth-reader kube-system/extension-apiserver-authentication-reader ...
However the role extension-apiserver-authentication-reader in the kube-system namespace is missing:
# oc get role extension-apiserver-authentication-reader -n kube-system
Error from server (NotFound): roles.authorization.openshift.io "extension-apiserver-authentication-reader" not found
Environment
- Red Hat Openshift Container Platform
- 3.10
- 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.