Some monitoring targets are down and kube-rbac-proxy and kube-rbac-proxy-metric containers show authentication errors

Issue

• The cluster is healthy, but some monitoring targets show up as "down" with an error message like this:

Error: server returned HTTP status 401 Unauthorized
inlcude: grafana/prometheus-k8s/telemeter-client/thanos-sidecar/alertmamager targets

• Logs of kube-rbac-proxy container in prometheus-k8s-0 pod return error messages like the following:

2021-12-15T20:42:14.037307604Z I1215 20:42:14.037071       1 main.go:151] Reading config file: /etc/kube-rbac-proxy/config.yaml
2021-12-15T20:42:14.040466228Z I1215 20:42:14.040394       1 main.go:181] Valid token audiences: 
2021-12-15T20:42:14.040860463Z I1215 20:42:14.040816       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca::/etc/tls/client/client-ca.crt"
2021-12-15T20:42:14.041045983Z I1215 20:42:14.040997       1 main.go:305] Reading certificate files
2021-12-15T20:42:14.041296788Z I1215 20:42:14.041154       1 dynamic_cafile_content.go:167] Starting client-ca::/etc/tls/client/client-ca.crt
2021-12-15T20:42:14.041358071Z I1215 20:42:14.041342       1 reloader.go:98] reloading key /etc/tls/private/tls.key certificate /etc/tls/private/tls.crt
2021-12-15T20:42:14.041722605Z I1215 20:42:14.041682       1 main.go:339] Starting TCP socket on 0.0.0.0:9092
2021-12-15T20:42:14.042563009Z I1215 20:42:14.042495       1 main.go:346] Listening securely on 0.0.0.0:9092
2021-12-16T15:28:14.043412756Z I1216 15:28:14.043113       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca::/etc/tls/client/client-ca.crt"
2021-12-16T15:32:14.042711352Z I1216 15:32:14.042521       1 dynamic_cafile_content.go:129] Loaded a new CA Bundle and Verifier for "client-ca::/etc/tls/client/client-ca.crt"
2021-12-16T20:14:35.200286140Z E1216 20:14:35.200081       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:14:35Z is after 2021-12-16T20:14:30Z
2021-12-16T20:14:44.071594661Z E1216 20:14:44.071463       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:14:44Z is after 2021-12-16T20:14:30Z
2021-12-16T20:15:05.200062356Z E1216 20:15:05.199898       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:15:05Z is after 2021-12-16T20:14:30Z
2021-12-16T20:15:14.072273022Z E1216 20:15:14.070036       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:15:14Z is after 2021-12-16T20:14:30Z

• Logs of container kube-rbac-proxy-metric in pod alertmanager-main-0 show the following error messages too:

2021-12-15T20:41:57.832903155Z I1215 20:41:57.832680       1 main.go:151] Reading config file: /etc/kube-rbac-proxy/config.yaml
2021-12-15T20:41:57.849665600Z I1215 20:41:57.849561       1 main.go:181] Valid token audiences: 
2021-12-15T20:41:57.851298263Z I1215 20:41:57.851142       1 main.go:305] Reading certificate files
2021-12-15T20:41:57.855484254Z I1215 20:41:57.855434       1 dynamic_cafile_content.go:167] Starting client-ca::/etc/tls/client/client-ca.crt
2021-12-15T20:41:57.856139965Z I1215 20:41:57.856001       1 main.go:339] Starting TCP socket on 0.0.0.0:9097
2021-12-15T20:41:57.866276093Z I1215 20:41:57.863687       1 main.go:346] Listening securely on 0.0.0.0:9097
2021-12-16T20:14:30.664561462Z E1216 20:14:30.664417       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:14:30Z is after 2021-12-16T20:14:30Z
2021-12-16T20:14:56.424561585Z E1216 20:14:56.424473       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:14:56Z is after 2021-12-16T20:14:30Z
2021-12-16T20:15:00.664743050Z E1216 20:15:00.664629       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:15:00Z is after 2021-12-16T20:14:30Z
2021-12-16T20:15:26.424947375Z E1216 20:15:26.424880       1 proxy.go:73] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2021-12-16T20:15:26Z is after 2021-12-16T20:14:30Z