alerts PrometheusNotIngestingSamples / PrometheusNotConnectedToAlertmanagers activated and scrape manager errors in prometheus

Solution Verified - Updated -

Issue

  • AlertManager receiving suddenly multiple alert on production cluster as AlertManager is not connected and not ingesting samples.

  • Errors seen in logs:

2020-10-20T00:50:04.712528301Z level=warn ts=2020-10-20T00:50:04.712Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 57991441 (57993667)"
2020-10-20T00:50:39.714115125Z level=warn ts=2020-10-20T00:50:39.714Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 57993274 (57993834)"

Environment

  • Red Hat OpenShift Container Platform
    • 4.3-4.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content