Are any additional steps required when configuring multipath storage on a system where /var is a separate file system?

Solution Verified - Updated -

Issue

  • LVM commands are hanging on certain devices
  • Data corruption is occurring on SAN luns
  • The multipath devices are remapping themselves while not being rebooted.  This occurs while system is up and running .
  • Have seen that the LUN's presented from the SAN have become read-only with the following error message:

        kernel: EXT3-fs error (device dm-11): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
        kernel: Aborting journal on device dm-11.
        kernel: ext3_abort called.
        kernel: EXT3-fs error (device dm-11): ext3_journal_start_sb: Detected aborted journal
        kernel: Remounting filesystem read-only
    

note: there is more than 1 cause of the above error

  • One problem is with dm-multipath which appears to be mapping to the wrong LUN presented from our SAN.
  • Multiple mpaths (mpath10 and mpath11) that both appear to be mapped to lun #10 and this is causing one of the mpaths (mpath 11) shows as empty.
  • System was showing 100% iowait, was rebooted and now is having issues activating logical volume.
  • Multipath appears to have renumbered the devices and is lost/confused about LVs on those LUNs
  • Multipath appears to have mapped the same devices into multiple maps, or is reporting the wrong mpath names for certain devices
  • Multipath is seeing duplicate physical volumes after a reboot, corrupting the filesystem.  After a reboot, the following error appears from multipath:

        # multipath -v2 
        remove: mpath53 (dup of mpath55) 
        mpath53: map in use 
        remove: mpath54 (dup of mpath56)
        mpath54: map in use 
    
  • Friendly names not working properly, missing /dev/mapper/mpath* names during boot which prevents volume groups being properly recognized and mounted.  The following is seen during boot:

Cannot make directory [/var/lib] : Read-only file system

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • device-mapper-multipath
    • Configured to use the user_friendly_names configuration option
    • Multipath bindings database file is located on /var filesystem (default location is /var/lib/multipath/bindings)
  • The /var filesystem is mounted onto the root filesystem at boot time (as opposed to being a directory on the root filesystem that doesn't require a mount to be available).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In