Loki ingesters 0/1 in RHOCP 4
Issue
-
A
logging-loki-ingester
ornetobserv-loki-ingester
pod are unable to initialize due to presence of staleloki-ingester
entries inLokiStack
hash ring:netobserv-loki-ingester-0 0/1 Running 0 1d
logging-loki-ingester-0 0/1 Running 0 1d
-
The
loki-ingester
logs shows an error with the InvalidBucketState/ring:level=error ts=2024-08-14T09:25:53.576834111Z caller=flush.go:143 org_id=infrastructure msg="failed to flush" err="failed to flush chunks: store put chunk: InvalidBucketState: The request is not valid with the current state of the bucket.\n\tstatus code: 409, request id: xxxx-xxxx-xxxx, host id: xxxx-xxxx-xxx, num_chunks: 1, labels: {kubernetes_host=\"example.ocp.com\", log_type=\"infrastructure\"}"
msg="found an existing instance(s) with a problem in the ring, this instance cannot become ready until this problem is resolved. The /ring http endpoint on the distributor (or single binary) provides visibility into the ring." ring=ingester err="instance 10.x.x.x:9095 past heartbeat timeout"
-
Unable see logs in web console log section showing "Too Many Unhealthy Instances In The Ring"
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- Red Hat OpenShift Logging (RHOL)
- 5
- Network Observability Operator
- LokiStack
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.