Pod crashes when modifying file permissions when backed by Azure Files storage

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
  • Azure Red Hat OpenShift (ARO)

Issue

  • Pod encounters error when executing commands that modify file permissions when backed by Azure Files storage provider.

Resolution

  • If the pod workload supports ReadWriteOnce access mode, then consider using Azure Disk.
  • If ReadWriteMany access mode is required, consider using Azure Files NFS v4.1 which supports POSIX-compliant file locking.

Root Cause

Azure Files is a CIFS implementation of the SMB protocol and is not POSIX compliant, and does not support changing file permissions.

Diagnostic Steps

  • Deploy a Postgres pod backed by Azure Files storage. The pod crashes and logs show errors related to chmod operations:
chmod: changing permissions of '/var/lib/pgsql/data/userdata': Operation not permitted
  • As another example, deploy a Grafana pod backed by Azure Files storage. The pod is unable to change file permissions and logs show the resulting error:
logger=sqlstore t=2024-07-09T20:27:20.515482447Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxrwxrwx expected=-rw-r-----
2Error: ✗ migration failed (id = create migration_log table): database is locked

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments