Critical Problem for NETAPP users with 6.4 release of device-mapper-multipath

Posted on

Updating device-mapper-multipath to 6.4 w/out updating kernel on your NETAPP system and it either won't boot or all your multipath devices break

The problem affects all users of NETAPP direct-attached (not network-attached) storage, i.e. iSCSI and FC but not NFS/CIFS etc.

The mpath target feature "retain_attached_hardware_handler" was added to the kernel and userspace multipath-tools for RHEL-6.4:

The kernel change was merged in 2.6.32-299.el6

The updated package will attempt to set this feature on NETAPP devices (the only built-in controller config to use this) but does not check the target or kernel version. This causes an error during table load and prevents multipath maps from being created on older kernels:

multipath: Unrecognised multipath feature request
  ioctl: error adding target to table
  dracut Warning: LVM vg/lv not found

How to Reproduce the Issue

  1. Configure a multipath device to use retain_attached_hardware_handler
  2. Boot a kernel < 299.el6
  3. 3. Attempt to configure multipath maps

Workarounds

There are three ways to work around this:

  • Update to a 6.4 kernel that supports this device-mapper feature i.e. latest available 2.6.32-358.2.1.el6
  • Downgrade the device-mapper-multipath package if the update to a RHEL-6.4 kernel is not possible
  • Specify a custom NETAPP configuration in /etc/multipath.conf that removes the retain_attached_hardware_handler feature for these devices, e.g.:
    devices {
      device {
        vendor "NETAPP"
        product "LUN.*"
        path_grouping_policy group_by_prio
        getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        path_selector "round-robin 0"
        path_checker tur
        features "3 queue_if_no_path pg_init_retries 50"
        hardware_handler "0"
        prio ontap
        failback immediate
        rr_weight uniform
        rr_min_io 128
        rr_min_io_rq 1
        flush_on_last_del yes
        fast_io_fail_tmo 5
        dev_loss_tmo infinity
        retain_attached_hw_handler no
        detect_prio yes
      }
    }