Multipath uid/gid/mode settings within multipath.conf not always being applied
Environment
-
Red Hat Enterprise Linux (RHEL) 5
-
device-mapper-multipath PRIOR to version
0.4.7-30.el5_4.4.
- The same issue exists on RHEL4, but a different fix will be needed and is NOT covered by this article, see Bugzilla 531131.
Issue
- Multipath uid/gid/mode settings not working correctly
- Multipath UID/GID/MODE settings not always taking effect
- Multipath uid/gid/mode settings occasionally at not correctly applied
- multipath.conf uid/gid settings ignored
- Slow to boot, udev limits
Resolution
-
The issue has been resolved for RHEL5 via Bugzilla 518575 and released with errata RHBA-2009:1645.
-
The resolution is to change the multipath rules as shown below.
File: /etc/udev/rules.d/40-multipath.rules
# multipath wants the devmaps presented as meaninglful device names # so name them after their devmap name SUBSYSTEM!="block", GOTO="end_mpath" KERNEL!="dm-[0-9]*", GOTO="end_mpath" PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath" PROGRAM!="/sbin/dmsetup info -c --noheadings -j %M -m %m", GOTO="end_mpath" RESULT!="*:*:*:*:*:*:*:mpath-*", GOTO="kpartx_check" PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", RUN+="/sbin/kpartx -a -p p /dev/mapper/%c" OPTIONS="last_rule" LABEL="kpartx_check" RESULT!="*:*:*:*:*:*:*:part*-mpath-*", GOTO="end_mpath" PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c" OPTIONS="last_rule"
LABEL="end_mpath"
Root Cause
Root cause is a race condition between udev and mpath, specifically when dmsetup ls --exec is run as part of the 40-multipath.rules udev sequence. The new rules avoid running dmsetup in this way.
Diagnostic Steps
The steps to reproduce this problem are typically:
1) create multipath{} definition in multipath.conf that uses the uid/gid/mode settings
2) create multipath devices
3) observe the ownership/permissions on the multipath devices in /dev/mapper don't match what is set in multipath.conf
or
1) Modify multipath.conf and add uid, gid and mode settings for the relevent multipath devices.
2) Flush multipath maps if they exist already
3) Create multipath maps
-----------------------------------------------------------------------------------------------
1. Edit multipath.conf, adding uid/gid info, for example:
multipath {
wwid 149455400000000000000000001000000ea6300000f000000
alias sinbadtehsailor
uid 200
gid 200
mode 0660
}
2. Clean up messages before starting test run
# /usr/sbin/logrotate --force /etc/logrotate.conf
3. Run systemtap scripts to show ownership changes on devices
# rm -f chown_time.tapout ;stap chown_time.stp -o chown_time.tapout
4. run the following test to flush/rebuild maps repeatedly
# while true; do multipath -F;sleep 1;multipath;sleep 1;ll /dev/mapper/sinbadtehsailor | grep disktest || break; done ;
Review logged output and notice:
name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\ << good
6551934 25632 5289 (multipath) 11 0
name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\ << good
8692582 25724 5289 (multipath) 7 0
name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\ << good
10824586 25843 5289 (multipath) 9 0
name: chown path: /dev/mapper/sinbadtehsailor owner: 0 group: 6\ << bad!
11882820 25867 25858 (dmsetup) 9 0
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments