The user-defined Prometheus pods cannot mount hostpath

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4.10
  • Prometheus Operator
    • 2.32.1

Issue

After installing prometheus via enabling monitoring and using hostpath directory as persistent volume, as below message which printed by prometheus-user-workload pods you can see:

[root@bastion prometheus-hostpath]# oc logs -f prometheus-user-workload-0 -c prometheus
ts=2023-03-25T07:58:45.104Z caller=main.go:532 level=info msg="Starting Prometheus" version="(version=2.32.1, branch=rhaos-4.10-rhel-8, revision=ea9b063e0311db4b89c2c792dcfcff442799d6a3)"
ts=2023-03-25T07:58:45.104Z caller=main.go:537 level=info build_context="(go=go1.17.5, user=root@60620cb00446, date=20220531-23:08:37)"
ts=2023-03-25T07:58:45.105Z caller=main.go:538 level=info host_details="(Linux 4.18.0-305.49.1.el8_4.x86_64 #1 SMP Wed May 11 09:47:48 EDT 2022 x86_64 prometheus-user-workload-0 (none))"
ts=2023-03-25T07:58:45.105Z caller=main.go:539 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-03-25T07:58:45.105Z caller=main.go:540 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-03-25T07:58:45.105Z caller=query_logger.go:86 level=error component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:/cho
github.com/prometheus/prometheus/promql.NewActiveQueryTracker({0x7fff79200589, 0xb}, 0x14, {0x354b680, 0xc0005df590})
    /go/src/github.com/prometheus/prometheus/promql/query_logger.go:116 +0x3d7
main.main()
    /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:584 +0x5e53

In the host which the pods have been scheduled, you can find the logs about SELinux privileges in the audit.log

type=AVC msg=audit(1679884214.181:6395): avc:  denied  { write } for  pid=350042 comm="prometheus" name="prometheus-db" dev="sda4" ino=35651814 scontext=system_u:system_r:container_t:s0:c5,c21 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0

Resolution

    1. Change direstory's permission to 777.
    1. Set userid(uid) and groupid(gid) to hostpath directory.
    1. Change SELinux context to hostpath directory.
[root@worker03 mnt]# mkdir /mnt/prometheus-data
[root@worker03 mnt]# chmod 777 -R /mnt/prometheus-data
[root@worker03 mnt]# chown -R 65534:65534 /mnt/prometheus-data
[root@worker03 mnt]# chcon -t container_file_t /mnt/prometheus-data
[root@worker03 mnt]# ls -lZd /mnt/prometheus-data/
drwxrwxrwx. 2 nfsnobody nfsnobody system_u:object_r:container_file_t:s0 6 Mar 27 06:07 /mnt/prometheus-data/

Root Cause

Proper SELinux context container_file_t should be set on hostpath directory.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments