Prometheus: Opening storage failed open block: tombstones header: invalid size

Solution Verified - Updated -

Issue

  • Prometheus pod is in CrashLoopBackOff state with the error below:

    $ oc logs <prometheus pod> -c prometheus
    ...
    level=info ts=2020-01-20T11:31:51.966720313Z caller=web.go:415 component=web msg="Start listening for connections" address=127.0.0.1:9090
    level=info ts=2020-01-20T11:31:51.966169539Z caller=main.go:533 msg="Starting TSDB ..."
    level=info ts=2020-01-20T11:31:51.967487728Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1577880000000 maxt=1577944800000 ulid=01DXJQWTJAHCNPEZYJN3QH92FJ
    ...
    level=info ts=2020-01-20T11:31:51.972484595Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1579219200000 maxt=1579226400000 ulid=01DYRQ2X1AM517EJM4ST1CN8Q1
    ...
    level=info ts=2020-01-20T11:31:53.396927938Z caller=main.go:402 msg="Stopping scrape discovery manager..."
    level=info ts=2020-01-20T11:31:53.396990862Z caller=main.go:416 msg="Stopping notify discovery manager..."
    ...
    level=error ts=2020-01-20T11:31:53.397686813Z caller=main.go:596 err="Opening storage failed open block /prometheus/01DYRQ2X1AM517EJM4ST1CN8Q1: tombstones header: invalid size"
    

Environment

  • Red Hat OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content