How to remove Alert PrometheusTSDBCompactionsFailing due to the disk for Prometheus becoming full

Solution Verified - Updated -

Issue

  • Alert PrometheusTSDBCompactionsFailing
    1. What is the alert about?
    2. Will this impact our services?
  • Before this, we had the same alert with error message in logs (no space left in device) and we decide to extend /prometheus disk.
  • After extend the disk we still got this alert but with different message in logs:
(err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/wal/000xxxxx at 12xxxxxx: unexpected full record")
  • Relates to Prometheus using permanent storage not ephemeral storage

Environment

  • Red Hat OpenShift Container Platform
    • 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content