The alertmanager pod shows context deadline exceeded
Issue
- The alertmanager container of an alertmanager-xxxx pod in the openshift-monitoring project shows the below error:
2022-06-07T07:58:16.537294001Z E0607 07:58:16.537244 1 webhook.go:111] Failed to make webhook authenticator request: Post "https://172.30.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp 172.30.0.1:443: connect: connection refused
2022-06-07T07:58:16.537380288Z 2022/06/07 07:58:16 oauthproxy.go:793: requestauth: xx.xxx.xx.xxx:51346 Post "https://172.30.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp 172.30.0.1:443: connect: connection refused
Kube-apiserver pod restart time:
- The alertmanager-proxy container of an alertmanager-xxxx pod in the openshift-monitoring project shows the below error:
level=error ts=2022-06-05T12:01:31.248Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Watchdog/webhook[0]: notify retry canceled after 3 attempts: Post \"http://<Application Name>.<NameSpace Name>.svc.cluster.local:7777\": context deadline exceeded"
level=warn ts=2022-06-06T06:01:30.937Z caller=notify.go:674 component=dispatcher receiver=Watchdog integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://<Application Name>.<NameSpace Name>.svc.cluster.local:7777"
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.