The alertmanager pod shows context deadline exceeded

Solution Verified - Updated -

Issue

  • The alertmanager container of an alertmanager-xxxx pod in the openshift-monitoring project shows the below error:
2022-06-07T07:58:16.537294001Z E0607 07:58:16.537244       1 webhook.go:111] Failed to make webhook authenticator request: Post "https://172.30.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp 172.30.0.1:443: connect: connection refused
2022-06-07T07:58:16.537380288Z 2022/06/07 07:58:16 oauthproxy.go:793: requestauth: xx.xxx.xx.xxx:51346 Post "https://172.30.0.1:443/apis/authentication.k8s.io/v1/tokenreviews": dial tcp 172.30.0.1:443: connect: connection refused
Kube-apiserver pod restart time:
  • The alertmanager-proxy container of an alertmanager-xxxx pod in the openshift-monitoring project shows the below error:
level=error ts=2022-06-05T12:01:31.248Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Watchdog/webhook[0]: notify retry canceled after 3 attempts: Post \"http://<Application Name>.<NameSpace Name>.svc.cluster.local:7777\": context deadline exceeded"
level=warn ts=2022-06-06T06:01:30.937Z caller=notify.go:674 component=dispatcher receiver=Watchdog integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://<Application Name>.<NameSpace Name>.svc.cluster.local:7777"

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content