Prometheus pods not able to mount NFS volumes after upgrading the cluster to 4.6.8
Issue
- SCC of prometheus/alertmanager pods has been changed from
restricted
tononroot
post upgrade. - Prometheus-k8s pods unable to access the NFS volume, getting into crashloopbackoff state.
- Below error messages are seen in the logs of failing prometheus container:
level=info ts=2020-12-19T16:25:02.433Z caller=main.go:361 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2020-12-19T16:25:02.434Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffcdab51153, 0xb, 0x14, 0x2ef0a80, 0xc0012f0780, 0x2ef0a80)
/go/src/github.com/prometheus/prometheus/promql/query_logger.go:117 +0x4cf
main.main()
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:388 +0x536c
Environment
- Red Hat OpenShift Container Platform
- 4.6.8+
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.