Why does Multipath fail to create a map with error "mpathx: domap (0) failure for create/reload map"?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
  • Multipath

Issue

  • Multipath doesn't create a map for some specific devices.
  • Other devices maps are being created correctly.

Resolution

  • Check whether any application is holding those scsi devices.
  • If this is happening on boot devices, make sure that the initramfs contains the correct multipath configuration so that the maps are created at boot time, before anything else takes control over them.
  • If this is happening at boot time but not only on the boot device, make sure that your LVM filter is correctly set up to ignore paths but scan the maps. Also in this case, once the LVM filter and the multipath.conf file are configured correctly, recreating the initial ramdisk image is important.

Root Cause

The multipath error messages are from multipath being unable to claim the underlying scsi devices. It is not able to claim device paths because some of the paths are open and in use. multipath will not bind to devices which are already in use.

Diagnostic Steps

  • In the verbose multipath output ( # multipath -v4 -ll) it should be possible to see a flow similar to this:
Dec 11 12:50:32 | mpathhy: verified path sdk dev_t 8:160
Dec 11 12:50:32 | mpathhy: verified path sds dev_t 65:32
Dec 11 12:50:32 | mpathhy: verified path sdaa dev_t 65:160
Dec 11 12:50:32 | mpathhy: verified path sdbjm dev_t 69:1664
Dec 11 12:50:32 | mpathhy: verified path sdbki dev_t 70:1760
Dec 11 12:50:32 | mpathhy: verified path sdbkq dev_t 71:1632
Dec 11 12:50:32 | mpathhy: verified path sdbky dev_t 71:1760
Dec 11 12:50:32 | mpathhy: verified path sdebz dev_t 128:3344
Dec 11 12:50:32 | mpathhy: pgfailback = -2 (controller setting)
Dec 11 12:50:32 | mpathhy: pgpolicy = group_by_prio (controller setting)
Dec 11 12:50:32 | mpathhy: selector = round-robin 0 (internal default)
Dec 11 12:50:32 | mpathhy: features = 1 queue_if_no_path (controller setting)
Dec 11 12:50:32 | mpathhy: hwhandler = 1 alua (controller setting)
Dec 11 12:50:32 | mpathhy: rr_weight = 1 (internal default)
Dec 11 12:50:32 | mpathhy: minio = 1 rq (config file default)
Dec 11 12:50:32 | mpathhy: no_path_retry = NONE (internal default)
Dec 11 12:50:32 | pg_timeout = NONE (internal default)
Dec 11 12:50:32 | mpathhy: retain_attached_hw_handler = 1 (config file default)
Dec 11 12:50:32 | mpathhy: set ACT_CREATE (map does not exist)
Dec 11 12:50:32 | libdevmapper: ioctl/libdm-iface.c(1742): device-mapper: create ioctl on mpathhy failed: Device or resource busy
Dec 11 12:50:32 | libdevmapper: ioctl/libdm-iface.c(1742): device-mapper: create ioctl on mpathhy failed: Device or resource busy
Dec 11 12:50:32 | mpathhy: domap (0) failure for create/reload map
Dec 11 12:50:32 | mpathhy: ignoring map
Dec 11 12:50:32 | mpathhy: remove multipath map
Dec 11 12:50:32 | sdk: orphaned
Dec 11 12:50:32 | sds: orphaned
Dec 11 12:50:32 | sdaa: orphaned
Dec 11 12:50:32 | sdbjm: orphaned
Dec 11 12:50:32 | sdbki: orphaned
Dec 11 12:50:32 | sdbkq: orphaned
Dec 11 12:50:32 | sdbky: orphaned
Dec 11 12:50:32 | sdebz: orphaned
  • Multipath map doesn't created for follwoing scsi devics. sdc, sdu , sdl , sdad and sda
    • sda is local disk and all other disks are coming from FUJITSU,ETERNUS_DX400
     3:0:1:1   sdad  65:208   -1  undef ready  FUJITSU,ETERNUS_DX400 running
     2:0:0:1   sdc   8:32     -1  undef ready  FUJITSU,ETERNUS_DX400 running
     3:0:0:1   sdu   65:64    -1  undef ready  FUJITSU,ETERNUS_DX400 running
     2:0:1:1   sdl   8:176    -1  undef ready  FUJITSU,ETERNUS_DX400 running

     0:2:0:0   sda   8:0      -1  undef ready  LSI,MRSASRoMB-4i      running
  • Check sosreport and see whether any any process is holding these devices.
$ egrep -w "sdu|sdc|sdl|sdad" proc/partitions
   8       32    2097152 sdc
   8      176    2097152 sdl
  65       64    2097152 sdu
  65      208    2097152 sdad

$ egrep -w "sdu|sdc|sdl|sdad" lsof

$  egrep -w "8,32|65,64|8,176|65,208" lsof
  • Noticed oracleasm process is running on this system. It may happen if oracle is holding this device.
  • Please check whether oracle is holding this device or not?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments