Loki Ingester 0/1 fails to Connect to Outdated IPs in Gossip Ring in RHOCP 4

Solution Verified - Updated -

Issue

  • Lokistack operator fails to start logging-loki-ingester pod due to connection timeout to an outdated IP in the gossip ring.
  • IP addresses in the gossip ring endpoint list that are no longer in use causing the issue.
  • The logs indicate an error connecting to the outdated IP address: WriteTo failed" addr=<IP>:7946 err="dial tcp <IP>:7946: i/o timeout".
  • The IP addresses are not present in the podnetwork or in the list of addresses for the logging-loki-gossip-ring endpoint.
  • Loki Ingester in 0/1, even, when not having an issue with the Loki storage as described in the Red Hat Knowledge Article "Loki ingesters 0/1 in RHOCP 4"
  • Loki Ingester pod throws the error:

    msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.x.x.x:9095 past heartbeat timeout"
    

Environment

  • Red OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
    • 6
  • LokiStack
  • Loki

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content