Chapter 1. Overview of device mapper multipathing

With Device mapper multipathing (DM Multipath), you can configure multiple I/O paths between server nodes and storage arrays into a single device. These I/O paths are physical Storage Area Network (SAN) connections that can include separate cables, switches, and controllers. Multipathing aggregates the I/O paths and creates a new device that consists of the aggregated paths.

DM Multipath provides:

Redundancy
DM Multipath can provide failover in an active/passive configuration. In an active/passive configuration, only a subset of the paths is used at any time for I/O. If any element of an I/O path such as the cable, switch, or controller fails, DM Multipath switches to an alternate path.
Note

The number of paths is dependent on the setup. Usually, DM Multipath setups have 2, 4, or 8 paths to the storage, but this is a common setup and other numbers are possible for the paths.

Improved Performance
DM Multipath can be configured in an active/active mode, where I/O is spread over the paths in a round-robin fashion. In some configurations, DM Multipath can detect loading on the I/O paths and dynamically rebalance the load.

1.1. Active/Passive multipath configuration with one RAID device

In this configuration, there are two Host Bus Adapters (HBAs) on the server, two SAN switches, and two RAID controllers. Following are the possible failure in this configuration:

  • HBA failure
  • Fibre Channel cable failure
  • SAN switch failure
  • Array controller port failure

With DM Multipath configured, a failure at any of these points causes DM Multipath to switch to the alternate I/O path. The following image describes the configuration with two I/O paths from the server to a RAID device. Here, there is one I/O path that goes through hba1, SAN1, and cntrlr1 and a second I/O path that goes through hba2, SAN2, and cntrlr2.

Figure 1.1. Active/Passive multipath configuration with one RAID device

Active/Passive multipath configuration with one RAID device

1.2. Active/Passive multipath configuration with two RAID devices

In this configuration, there are two HBAs on the server, two SAN switches, and two RAID devices with two RAID controllers each. With DM Multipath configured, a failure at any of the points of the I/O path to either of the RAID devices causes DM Multipath to switch to the alternate I/O path for that device. The following image describes the configuration with two I/O paths to each RAID device. Here, there are two I/O paths to each RAID device.

Figure 1.2. Active/Passive multipath configuration with two RAID device

Active/Passive multipath configuration with two RAID device

1.3. Active/Active multipath configuration with one RAID device

In this configuration, there are two HBAs on the server, two SAN switches, and two RAID controllers. The following image describes the configuration with two I/O paths from the server to a storage device. Here, I/O can be spread among these two paths.

Figure 1.3. Active/Active multipath configuration with one RAID device

Active/Active multipath configuration with one RAID device

1.4. DM Multipath components

The following table describes the DM Multipath components.

Table 1.1. Components of DM Multipath

Component

Description

dm_multipath kernel module

Reroutes I/O and supports failover for paths and path groups.

mpathconf utility

Configures and enables device mapper multipathing.

multipath command

Lists and configures the multipath devices. It is also executed by udev whenever a block device is added, to determine if the device should be part of a multipath device or not.

multipathd daemon

Automatically creates and removes multipath devices and monitors paths; as paths fail and come back, it may update the multipath device. Allows interactive changes to multipath devices. Reload the service if there are any changes to the /etc/multipath.conf file.

kpartx command

Creates device mapper devices for the partitions on a device. This command is automatically executed by udev when multipath devices are created to create partition devices on top of them. The kpartx command is provided in its own package, but the device-mapper-multipath package depends on it.

mpathpersist

Sets up SCSI-3 persistent reservations on multipath devices. This command works similarly to the way sg_persist works for SCSI devices that are not multipathed, but it handles setting persistent reservations on all paths of a multipath device. It coordinates with multipathd to ensure that the reservations are set up correctly on paths that are added later. To use this functionality, the reservation_key attribute must be defined in the /etc/multipath.conf file. Otherwise the multipathd daemon will not check for persistent reservations for newly discovered paths or reinstated paths.

1.5. The multipath command

The multipath command is used to detect and combine multiple paths to devices. It provides a variety of options you can use to administer your multipathed devices.

The following table describes some options of the multipath command that you may find useful.

Table 1.2. Useful multipath command options

OptionDescription

-l

Display the current multipath configuration gathered from sysfs and the device mapper.

-ll

Display the current multipath configuration gathered from sysfs, the device mapper, and all other available components on the system.

-f device

Remove the named multipath device.

-F

Remove all unused multipath devices.

-w device

Remove the wwid of the specified device from the wwids file.

-W

Reset the wwids file to include only the current multipath devices.

1.6. Multipath command output

When you create, modify, or list a multipath device, you get a display of the current device setup. The format is as follows.

  • For each multipath device: 
action_if_any: alias (wwid_if_different_from_alias) dm_device_name_if_known vendor,product size=size features='features' hwhandler='hardware_handler' wp=write_permission_if_known
  • For each path group: 
-+- policy='scheduling_policy' prio=prio_if_known status=path_group_status_if_known
  • For each path: 
 `- host:channel:id:lun devnode major:minor dm_status_if_known path_status online_status

For example, the output of a multipath command might appear as follows: 

3600d0230000000000e13955cc3757800 dm-1 WINSYS,SF2372
size=269G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 6:0:0:0 sdb 8:16  active ready  running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 7:0:0:0 sdf 8:80  active ready  running

If the path is up and ready for I/O, the status of the path is ready or ghost. If the path is down, the status is faulty or shaky. The path status is updated periodically by the multipathd daemon based on the polling interval defined in the /etc/multipath.conf file.

Additional possible path status values are as follows.

  • i/o pending: The checker is actively checking this path, and the state will be updated shortly.
  • i/o timeout: This is the same as faulty. It lets the user know that the checker did not return either success or failure before the timeout period.
  • removed: The path has been removed from the system, and will shortly be removed from the multipath device. It is treated the same as faulty.
  • wild: multipathd was unable to run the path checker, because of an internal error or configuration issue. This is roughly the same as faulty, except multipath will skip many actions on the path.
  • unchecked: The path checker has not run on this path, either because it has just been discovered, it does not have an assigned path checker, or the path checker encountered an error. This is treated the same as wild.
  • delayed: The path checker returns that the path is up, but multipath is delaying the reinstatement of the path because the path has recently failed multiple times and multipath has been configured to delay paths in this case.

In terms of the kernel, the dm status is similar to the path status The active dm state covers the ready and ghost path states. The pending path state has no equivalent dm state. All other path states map to the failed dm state. The dm state will retain its current status until the path checker has completed.

The possible values for online_status are running and offline. The offline status means that this SCSI device has been disabled.

Note

When you create or modify a multipath device, multipath prints the device configuration. However, some of the features, for example, write permissions, and other feature information might be unknown. There might be a difference between the output and the features that you selected during creation or modification. This is normal behaviour. List the device after creation to view the correct state.

1.7. Displaying multipath configuration

You can use the -l and multipath command to display the current multipath configuration. The -l option displays multipath topology gathered from information in sysfs and the device mapper. The -ll option displays the information the -l option displays in addition to all other available components of the system.

When displaying the multipath configuration, you can specify a verbosity level with the -v option of the multipath command. Specifying -v0 yields no output. Specifying -v1 outputs the created or updated multipath names only, which you can then feed to other tools such as kpartx. Specifying -v2 prints all detected paths, multipaths, and device maps. For even more detailed information, you can also specify -v3, -v4, or -v5.

The following example shows the output of a multipath -l command. 

# multipath -l
3600d0230000000000e13955cc3757800 dm-1 WINSYS,SF2372
size=269G features='0' hwhandler='0' wp=rw
|-- policy='round-robin 0' prio=1 status=active
| `- 6:0:0:0 sdb 8:16  active ready  running
`-- policy='round-robin 0' prio=1 status=enabled
  `- 7:0:0:0 sdf 8:80  active ready  running

The following example shows the output of a multipath -ll command. 

# multipath -ll
3600d0230000000000e13955cc3757801 dm-10 WINSYS,SF2372
size=269G features='0' hwhandler='0' wp=rw
|-- policy='round-robin 0' prio=1 status=enabled
| `- 19:0:0:1 sdc 8:32  active ready  running
`-- policy='round-robin 0' prio=1 status=enabled
  `- 18:0:0:1 sdh 8:112 active ready  running
3600d0230000000000e13955cc3757803 dm-2 WINSYS,SF2372
size=125G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 19:0:0:3 sde 8:64  active ready  running
  `- 18:0:0:3 sdj 8:144 active ready  running

1.8. Additional resources

  • multipath(8) and multipathd(8) man pages
  • /etc/multipath.conf file