Loki Ingester 0/1 fails to Connect to Outdated IPs in Gossip Ring in RHOCP 4
Issue
- Lokistack operator fails to start
logging-loki-ingesterpod due to connection timeout to an outdated IP in the gossip ring. - IP addresses in the gossip ring endpoint list that are no longer in use causing the issue.
- The logs indicate an error connecting to the outdated IP address:
WriteTo failed" addr=<IP>:7946 err="dial tcp <IP>:7946: i/o timeout". - The IP addresses are not present in the podnetwork or in the list of addresses for the
logging-loki-gossip-ring endpoint. - Loki Ingester in
0/1, even, when not having an issue with the Loki storage as described in the Red Hat Knowledge Article "Loki ingesters 0/1 in RHOCP 4" -
Loki Ingester pod throws the error:
msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.x.x.x:9095 past heartbeat timeout"
Environment
- Red OpenShift Container Platform (RHOCP)
- 4
- Red Hat OpenShift Logging (RHOL)
- 5
- 6
- LokiStack
- Loki
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.