Multipath uid/gid/mode settings within multipath.conf not always being applied

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5

  • device-mapper-multipath PRIOR to version 0.4.7-30.el5_4.4.

  • The same issue exists on RHEL4, but a different fix will be needed and is NOT covered by this article, see Bugzilla 531131.

Issue

  • Multipath uid/gid/mode settings not working correctly
  • Multipath UID/GID/MODE settings not always taking effect
  • Multipath uid/gid/mode settings occasionally at not correctly applied
  • multipath.conf uid/gid settings ignored
  • Slow to boot, udev limits

Resolution

  • The issue has been resolved for RHEL5 via Bugzilla 518575 and released with errata RHBA-2009:1645.

  • The resolution is to change the multipath rules as shown below.

File: /etc/udev/rules.d/40-multipath.rules

# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
PROGRAM!="/sbin/dmsetup info -c --noheadings -j %M -m %m", GOTO="end_mpath"
RESULT!="*:*:*:*:*:*:*:mpath-*", GOTO="kpartx_check"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c", RUN+="/sbin/kpartx -a -p p /dev/mapper/%c"
OPTIONS="last_rule"
LABEL="kpartx_check"
RESULT!="*:*:*:*:*:*:*:part*-mpath-*", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c"
OPTIONS="last_rule"
LABEL="end_mpath"

Root Cause

Root cause is a race condition between udev and mpath, specifically when dmsetup ls --exec is run as part of the 40-multipath.rules udev sequence. The new rules avoid running dmsetup in this way.

Diagnostic Steps

The steps to reproduce this problem are typically:

1) create multipath{} definition in multipath.conf that uses the uid/gid/mode settings
2) create multipath devices
3) observe the ownership/permissions on the multipath devices in /dev/mapper don't match what is set in multipath.conf

or

1) Modify multipath.conf and add uid, gid and mode settings for the relevent multipath devices.
2) Flush multipath maps if they exist already
3) Create multipath maps

-----------------------------------------------------------------------------------------------

1. Edit multipath.conf, adding uid/gid info, for example:

      multipath {
               wwid    149455400000000000000000001000000ea6300000f000000
               alias   sinbadtehsailor
               uid     200
               gid     200
               mode    0660

       }

2. Clean up messages before starting test run

    # /usr/sbin/logrotate --force /etc/logrotate.conf

3. Run systemtap scripts to show ownership changes on devices

    # rm -f chown_time.tapout ;stap chown_time.stp -o chown_time.tapout

4. run the following test to flush/rebuild maps repeatedly

    # while true; do multipath -F;sleep 1;multipath;sleep 1;ll /dev/mapper/sinbadtehsailor | grep disktest || break; done ;

Review logged output and notice:

name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\          << good
        6551934 25632 5289 (multipath) 11 0
name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\          << good
        8692582 25724 5289 (multipath) 7 0
name: chown path: /dev/mapper/sinbadtehsailor owner: 200 group: 200\          << good
        10824586 25843 5289 (multipath) 9 0

name: chown path: /dev/mapper/sinbadtehsailor owner: 0 group: 6\              << bad!
        11882820 25867 25858 (dmsetup) 9 0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments