I/O to multipath map hangs even after all paths are failed and configured to not queue in RHEL 6

Solution In Progress - Updated -

Issue

  • Multipath map is configured with no_path_retry fail, but I/O hangs even after all paths have failed
  • Cluster node was not evicted by qdiskd after all paths failed, even though the map is configured to not queue. qdiskd was reported as blocked by the kernel:
Jul 9 16:56:42,node1,[daemon.notice],multipathd:, 36006016061602d008296e08eb281e211: remaining active paths: 0
[...]
Jul 9 17:00:12,node1,[kern.err],kernel:,INFO: task qdiskd:24750 blocked for more than 120 seconds.
Jul 9 17:00:12,node1,[kern.err],kernel:,"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 9 17:00:12,node1,[kern.info],kernel:,qdiskd D 0000000000000004 0 24750 1 0x00000000
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804706dfb18 0000000000000082 0000000000000000 ffffffffa00041fc
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804706dfae8 00000000e4bd82fc ffff8804706dfb08 ffff880875e5ad80
Jul 9 17:00:12,node1,[kern.warning],kernel:, ffff8804726ad0f8 ffff8804706dffd8 000000000000f4e8 ffff8804726ad0f8
Jul 9 17:00:12,node1,[kern.warning],kernel:,Call Trace:
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? dm_table_unplug_all+0x5c/0x100 [dm_mod]
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] io_schedule+0x73/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] __blockdev_direct_IO_newtrunc+0x6fe/0xb90
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] __blockdev_direct_IO+0x5e/0xd0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? blkdev_get_blocks+0x0/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] blkdev_direct_IO+0x57/0x60
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? blkdev_get_blocks+0x0/0xc0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] generic_file_aio_read+0x6bb/0x700
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? futex_requeue+0x310/0x890
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] do_sync_read+0xfa/0x140
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? autoremove_wake_function+0x0/0x40
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? unmap_region+0x110/0x130
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] ? security_file_permission+0x16/0x20
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] vfs_read+0xb5/0x1a0
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] sys_read+0x51/0x90
Jul 9 17:00:12,node1,[kern.warning],kernel:, [] system_call_fastpath+0x16/0x1b

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • device-mapper-multipath
    • multipath map in question is configured to not queue (no_path_retry != queue)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.