Ghosts of Block-devs Past?

Latest response

Recently, a fairly junior technician was added to our staff. As part of the tech's onboarding, they were given the task of reviewing current standard operating procedures. This was done both to ensure that the SoPs were clear enough for someone very junior to use and to familiarize the tech with day-to-day tasksings.

 

This week, while working through SoPs related to SAN-based storage for physical Linux systems, the tech either ran into or created a really odd configuration error. Specifically, when attempting to run through the procecures to take some CLARiiON based LUN and get the pulled under control of dm-multipath and LVM, two mpathX devices (sort of) ended up in dm-multipath's database and both seemed to refer to the same LUN.

 

When the tech called me over to help - was getting "filesystem mounted?" errors when trying to run `pvcreate` the first bit of odditiy I noticed was that there was an "mpath1" device in multipath -ll's output but no "mpath0" device. The fdisk utility's output was showing, in addition to the /dev/sdX devices for the CLARiiON, a dm-7p1 device and a dm-8 device.

 

In tryint to work through the "filesystem mounted?" errors, I'd had the tech clear and re-do the partition tables (fdisk had shown layout errors on the /dev/sdX devices and was trying to render them consistent/good). However, after resetting the partitions, we noticed that the mis-partitioning came back and that the pooched partitioning matched what was on the dm-7p1 fdisk entries rather than the dm-8 entries.

 

Ultimately, ended up having the tech do a `multipath -f <DEVIDSTRING>` to flush out dm-multipath's entries. This got rid of the ghost and allowed the LUN to be correctly partitioned and got rid of the "filesystem mounted?" errors that `pvcreate` was spitting out. I just can't figure out how the system got into such an odd state in the first place. Anyone ever encounter something like this?

TJ
Log in to join the conversation

Responses

AV Red Hat Pro 539 points
31 October 2012 12:55 AM

To create a partition which one is used /dev/dm-n, /dev/mpath/ or /dev/mapper/ ?

RJ Active Contributor 212 points
31 October 2012 1:37 AM

A good summary can be found in section 2.1 of the the RHEL5 flavor of the DM Multipath documentation (here) which says:

 

When new devices are brought under the control of DM-Multipath, the new devices may be seen in three different places under the /dev directory: /dev/mapper/mpathn, /dev/mpath/mpathn, and /dev/dm-n.

  • The devices in /dev/mapper are created early in the boot process. Use these devices to access the multipathed devices, for example when creating logical volumes.
  • The devices in /dev/mpath are provided as a convenience so that all multipathed devices can be seen in one directory. These devices are created by the udev device manager and may not be available on startup when the system needs to access them. Do not use these devices for creating logical volumes or filesystems.
  • Any devices of the form /dev/dm- n are for internal use only and should never be used
Ryan Sawhill's picture Red Hat Guru 3719 points
21 December 2012 4:16 PM

Indeed -- I've seen a few cases where RHEL5 systems were using /dev/mpath/ devices and suffering from boot problems.

Note that this was remedied in RHEL6. Here's the same section from the RHEL6 guide.

(Note: I believe my colleague Arand was trying to ask the question of the OP Thomas Jones.)

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.