Are any additional steps required when configuring multipath storage on a system where /var is a separate file system?

Solution Verified - Updated -

Issue

  • LVM commands are hanging on certain devices
  • Data corruption is occurring on SAN luns
  • The multipath devices are remapping themselves while not being rebooted.  This occurs while system is up and running .
  • Have seen that the LUN's presented from the SAN have become read-only with the following error message:

        kernel: EXT3-fs error (device dm-11): ext3_readdir: bad entry in directory #2: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
        kernel: Aborting journal on device dm-11.
        kernel: ext3_abort called.
        kernel: EXT3-fs error (device dm-11): ext3_journal_start_sb: Detected aborted journal
        kernel: Remounting filesystem read-only
    

note: there is more than 1 cause of the above error

  • One problem is with dm-multipath which appears to be mapping to the wrong LUN presented from our SAN.
  • Multiple mpaths (mpath10 and mpath11) that both appear to be mapped to lun #10 and this is causing one of the mpaths (mpath 11) shows as empty.
  • System was showing 100% iowait, was rebooted and now is having issues activating logical volume.
  • Multipath appears to have renumbered the devices and is lost/confused about LVs on those LUNs
  • Multipath appears to have mapped the same devices into multiple maps, or is reporting the wrong mpath names for certain devices
  • Multipath is seeing duplicate physical volumes after a reboot, corrupting the filesystem.  After a reboot, the following error appears from multipath:

        # multipath -v2 
        remove: mpath53 (dup of mpath55) 
        mpath53: map in use 
        remove: mpath54 (dup of mpath56)
        mpath54: map in use 
    
  • Friendly names not working properly, missing /dev/mapper/mpath* names during boot which prevents volume groups being properly recognized and mounted.  The following is seen during boot:

Cannot make directory [/var/lib] : Read-only file system

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • device-mapper-multipath
    • Configured to use the user_friendly_names configuration option
    • Multipath bindings database file is located on /var filesystem (default location is /var/lib/multipath/bindings)
  • The /var filesystem is mounted onto the root filesystem at boot time (as opposed to being a directory on the root filesystem that doesn't require a mount to be available).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content