Prometheus: Opening storage failed open block: tombstones header: invalid size
Issue
-
Prometheus pod is in CrashLoopBackOff state with the error below:
$ oc logs <prometheus pod> -c prometheus ... level=info ts=2020-01-20T11:31:51.966720313Z caller=web.go:415 component=web msg="Start listening for connections" address=127.0.0.1:9090 level=info ts=2020-01-20T11:31:51.966169539Z caller=main.go:533 msg="Starting TSDB ..." level=info ts=2020-01-20T11:31:51.967487728Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1577880000000 maxt=1577944800000 ulid=01DXJQWTJAHCNPEZYJN3QH92FJ ... level=info ts=2020-01-20T11:31:51.972484595Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1579219200000 maxt=1579226400000 ulid=01DYRQ2X1AM517EJM4ST1CN8Q1 ... level=info ts=2020-01-20T11:31:53.396927938Z caller=main.go:402 msg="Stopping scrape discovery manager..." level=info ts=2020-01-20T11:31:53.396990862Z caller=main.go:416 msg="Stopping notify discovery manager..." ... level=error ts=2020-01-20T11:31:53.397686813Z caller=main.go:596 err="Opening storage failed open block /prometheus/01DYRQ2X1AM517EJM4ST1CN8Q1: tombstones header: invalid size"
Environment
- Red Hat OpenShift Container Platform 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.