Loki ingester pod fails to start due to corrupted segment in RHOCP 4

Solution Verified - Updated -

Issue

  • Messages below appear in ingester logs :

    msg="error running loki" err="corruption in segment /tmp/loki/tsdb-index/wal/<segment> at 8222: unexpected checksum 1c130b28, expected fd226a23\nerror recovering from TSDB WAL\ngithub.com/grafana/loki/pkg/storage/stores/tsdb.recoverHead\n\t/remote-source/loki/app/pkg/storage/stores/tsdb/head_manager.go:594\ngithub.com/grafana/loki/pkg/storage/stores/tsdb.(*tsdbManager).BuildFromWALs\n
  • Loki ingester pod is in a CrashLoopBackOff status.
  • Loki ingester logs are showing corruption segment WAL errors.

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat Openshift Logging (RHOL)
    • 5.9
  • Loki

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content