Prometheus: Opening storage failed open block: tombstones header: invalid size

Solution Verified - Updated -

Issue

  • Prometheus pod is in CrashLoopBack state with the error below:

    $ oc logs <prometheus pod> -c prometheus
    ...
    level=info ts=2020-01-20T11:31:51.966720313Z caller=web.go:415 component=web msg="Start listening for connections" address=127.0.0.1:9090
    level=info ts=2020-01-20T11:31:51.966169539Z caller=main.go:533 msg="Starting TSDB ..."
    level=info ts=2020-01-20T11:31:51.967487728Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1577880000000 maxt=1577944800000 ulid=01DXJQWTJAHCNPEZYJN3QH92FJ
    ...
    level=info ts=2020-01-20T11:31:51.972484595Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1579219200000 maxt=1579226400000 ulid=01DYRQ2X1AM517EJM4ST1CN8Q1
    ...
    level=info ts=2020-01-20T11:31:53.396927938Z caller=main.go:402 msg="Stopping scrape discovery manager..."
    level=info ts=2020-01-20T11:31:53.396990862Z caller=main.go:416 msg="Stopping notify discovery manager..."
    ...
    level=error ts=2020-01-20T11:31:53.397686813Z caller=main.go:596 err="Opening storage failed open block /prometheus/01DYRQ2X1AM517EJM4ST1CN8Q1: tombstones header: invalid size"
    

Environment

  • OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In