iscsi storage provider failed to apply right permission for non-root user
Issue
- After upgrade to OpenShift 4.6.8+, Prometheus pods failed to start and getting
CrashLoopbackoff
- Can see following message in the log of prometheus container inside prometheus-k8s-* pods
oc logs -n openshift-monitoring prometheus-k8s-0 -c prometheus
...
...
level=error ts=2021-11-19T16:25:02.434Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log
goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffcdab51153, 0xb, 0x14, 0x2ef0a80, 0xc0012f0780, 0x2ef0a80)
/go/src/github.com/prometheus/prometheus/promql/query_logger.go:117 +0x4cf
main.main()
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:388 +0x536c
- Delete PVC and release PV, won't work. The new created prometheus pod still have permission error on mount point
/prometheus
Environment
- Red Hat OpenShift Container Platform
- 4.6.8+
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.