Stale ingesters not leaving the hash ring in LokiStack in RHOCP 4

Solution Verified - Updated -

Issue

  • A logging-loki-ingester or netobserv-loki-ingester pod are unable to initialize due to presence of stale loki-ingester entries in LokiStack hash ring:

    netobserv-loki-ingester-0                        0/1     Running   0          1d
    
    logging-loki-ingester-0                        0/1     Running   0          1d
    
  • The loki-ingester logs shows an error with the ring:

    msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.x.x.x:9095 past heartbeat timeout"
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
  • Network Observability Operator
  • LokiStack

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content