Prometheus: Opening storage failed: lock DB directory: resource temporarily unavailable.

Solution Verified - Updated -

Issue

  • Prometheus pods are in CrashLoopBackoff state with following error logs.
2022-09-13T18:32:37.033014592Z ts=2022-09-13T18:32:37.032Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/prometheus/lock
2022-09-13T18:32:37.033068734Z ts=2022-09-13T18:32:37.033Z caller=main.go:816 level=info msg="Stopping scrape discovery manager..."
2022-09-13T18:32:37.033086982Z ts=2022-09-13T18:32:37.033Z caller=main.go:830 level=info msg="Stopping notify discovery manager..."
2022-09-13T18:32:37.033241345Z ts=2022-09-13T18:32:37.033Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
2022-09-13T18:32:37.033297123Z ts=2022-09-13T18:32:37.033Z caller=main.go:1072 level=info msg="Notifier manager stopped"
2022-09-13T18:32:37.033432293Z ts=2022-09-13T18:32:37.033Z caller=main.go:1081 level=error err="opening storage failed: lock DB directory: resource temporarily unavailable"
  • Why Prometheus pods are failing to start, Although enough back-end storage space is available?

Environment

  • Red Hat Openshift Container Platform
    • 4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content