alerts PrometheusNotIngestingSamples / PrometheusNotConnectedToAlertmanagers activated and scrape manager errors in prometheus
Issue
-
AlertManager receiving suddenly multiple alert on production cluster as AlertManager is not connected and not ingesting samples.
-
Errors seen in logs:
2020-10-20T00:50:04.712528301Z level=warn ts=2020-10-20T00:50:04.712Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 57991441 (57993667)"
2020-10-20T00:50:39.714115125Z level=warn ts=2020-10-20T00:50:39.714Z caller=klog.go:86 component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:263: watch of *v1.Endpoints ended with: too old resource version: 57993274 (57993834)"
Environment
- Red Hat OpenShift Container Platform
- 4.3-4.5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.