kube-rbac-proxy pods crashloop instead of loop waiting for the certificate to be available

Issue

• authentication cluster operator status is flapping

    NAME                           VERSION  AVAILABLE  PROGRESSING  DEGRADED  SINCE
    authentication                 4.6.25   True       False        True      2h16m

With one of the two existent replicas for the router pods logging the following error:

You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log router-c3-547749c789-q48f8))

The following traces can be found in the master nodes:

Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: I0817 03:09:01.370563    1691 status_manager.go:429] Ignoring same status for pod "sdn-zt2hb_openshift-sdn(0ea97571-4835-4180-8f0c-4a701fc467b7)", status: {Phase:Running Conditions:[{Type:Initialized Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2021-05-18 15:25:00 +0000 UTC Reason: Message:} {Type:Ready Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2021-08-09 15:56:21 +0000 UTC Reason: Message:} {Type:ContainersReady Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2021-08-09 15:56:21 +0000 UTC Reason: Message:} {Type:PodScheduled Status:True LastProbeTime:0001-01-01 00:00:00 +0000 UTC LastTransitionTime:2021-05-18 15:25:00 +0000 UTC Reason: Message:}] Message: Reason: NominatedNodeName: HostIP:180.9.142.9 PodIP:180.9.142.9 PodIPs:[{IP:180.9.142.9}] StartTime:2021-05-18 15:25:00 +0000 UTC InitContainerStatuses:[] ContainerStatuses:[{Name:kube-rbac-proxy State:{Waiting:nil Running:&ContainerStateRunning{StartedAt:2021-08-09 15:56:19 +0000 UTC,} Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:&ContainerStateTerminated{ExitCode:1,Signal:0,Reason:Error,Message::443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0706 09:39:04.503668    6825 webhook.go:109] Failed to make webhook authenticator request: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0706 09:39:04.503726    6825 proxy.go:73] Unable to authenticate the request due to an error: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0707 21:15:04.144891    6825 webhook.go:109] Failed to make webhook authenticator request: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0707 21:15:04.144997    6825 proxy.go:73] Unable to authenticate the request due to an error: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0802 22:17:04.145118    6825 webhook.go:109] Failed to make webhook authenticator request: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0802 22:17:04.145221    6825 proxy.go:73] Unable to authenticate the request due to an error: Post "https://10.120.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": context canceled
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: I0809 15:55:03.940876    6825 main.go:356] received interrupt, shutting down
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: E0809 15:55:04.441231    6825 main.go:309] failed to gracefully close secure listener: close tcp [::]:9101: use of closed network connection
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: Traceback (most recent call last):
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:   File "<string>", line 1, in <module>
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:   File "/usr/lib64/python3.6/json/__init__.py", line 299, in load
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:     parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:   File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:     return _default_decoder.decode(s)
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:   File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:   File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Aug 17 03:09:01.370691 ocp01-master-2 hyperkube[1691]: ,StartedAt:2021-08-09 15:56:14 +0000 UTC,FinishedAt:2021-08-09 15:56:17 +0000 UTC,ContainerID:cri-o://028433d034262c98dc4af256155feac886f42ae073930d8a768632dc193a18b0,}} Ready:true RestartCount:1 Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b24a2313c6937cced6f8fb6d2383fbe9c55415b59ca1973076f559f13650d6a ImageID:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b24a2313c6937cced6f8fb6d2383fbe9c55415b59ca1973076f559f13650d6a ContainerID:cri-o://f8d4ec68399b38df473a12347b349a6741a2a01bcfc4c35c6834a7d2c855141a Started:0xc003bb0cfa} {Name:sdn State:{Waiting:nil Running:&ContainerStateRunning{StartedAt:2021-08-09 15:56:13 +0000 UTC,} Terminated:nil} LastTerminationState:{Waiting:nil Running:nil Terminated:nil} Ready:true RestartCount:0 Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e60b952c5fdf59e352921fb9553a31c7ee06f0761769fe6d02ddc96c0b80def ImageID:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e60b952c5fdf59e352921fb9553a31c7ee06f0761769fe6d02ddc96c0b80def ContainerID:cri-o://b2e0caccd9af355dbce598b5bf74dffa9331908300edb9d217257787b4ca392f Started:0xc003bb0cfb}] QOSClass:Burstable EphemeralContainerStatuses:[]}