Kernel panic due to Hard LOCKUP in the function blk_throtl_drain()

Solution Verified - Updated -

Issue

  • Kernel panic due to Hard LOCKUP or the system hang with the following warning message. Issue has only been reported with third-party software, either VxDMP or PowerPath installed.
  • VxDMP:
WARNING: at block/blk-throttle.c:1222 blk_throtl_drain+0xff/0x180() (Tainted: P   ------------- )
Hardware name: PowerEdge R720
Modules linked in: vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) autofs4 dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm vxcafs(P)(U) vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs microcode iTCO_wdt iTCO_vendor_support dcdbas power_meter acpi_ipmi ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core shpchp joydev ses enclosure sg ixgbe dca mdio tg3 ext4 jbd2 mbcache sr_mod cdrom usb_storage sd_mod crc_t10dif mpt2sas scsi_transport_sas raid_class ahci qla2xxx scsi_transport_fc scsi_tgt mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Pid: 11034, comm: vxconfigd Tainted: P   --------------    2.6.32-754.6.3.el6.x86_64 #1
Call Trace:
 [<ffffffff810805c1>] ? warn_slowpath_common+0x91/0xe0
 [<ffffffff8108062a>] ? warn_slowpath_null+0x1a/0x20
 [<ffffffff812978af>] ? blk_throtl_drain+0xff/0x180
 [<ffffffff81286cd1>] ? __blk_drain_queue+0x91/0x140
 [<ffffffff81286fab>] ? blk_cleanup_queue+0xeb/0x1d0
 [<ffffffffa09c7c38>] ? vxvm_put_gendisk+0x68/0xf0 [vxio]
 [<ffffffffa09c7ce2>] ? volsys_unset_device+0x22/0x40 [vxio]
 [<ffffffffa0a0ef82>] ? vol_rmgroup_devices+0x82/0x100 [vxio]
 [<ffffffffa0a10eaa>] ? voldg_delete+0x1a/0x130 [vxio]
 [<ffffffffa0ba2390>] ? vol_rv_dgdelete_prepare_group+0x30/0x60 [vxio]
 [<ffffffffa0a1233b>] ? vol_delete_group+0x23b/0x290 [vxio]
 [<ffffffffa0ad3bf3>] ? volconfig_ioctl+0x603/0x6b0 [vxio]
 [<ffffffff8124326f>] ? security_capable+0x2f/0x40
 [<ffffffff8108e77a>] ? capable+0x2a/0x60
 [<ffffffffa0adcdb2>] ? volsioctl_real+0x412/0x550 [vxio]
 [<ffffffff81054e64>] ? __do_page_fault+0x1f4/0x500
 [<ffffffffa030a13c>] ? vols_ioctl+0x5c/0x80 [vxspec]
 [<ffffffffa030a37d>] ? vols_compat_ioctl+0x3d/0x60 [vxspec]
 [<ffffffff811fb1d8>] ? compat_sys_ioctl+0xf8/0x520
 [<ffffffff81565be7>] ? sysenter_dispatch+0x20/0x4b
---[ end trace d7e320c384a74d7d ]---
Kernel panic - not syncing: Hard LOCKUP
  • PowerPath:
kernel: WARNING: at block/blk-throttle.c:1222 blk_throtl_drain+0xff/0x180() (Tainted: P           -- ------------   )
kernel: Hardware name: ProLiant DL380p Gen8
kernel: Modules linked in: oracleacfs(P)(U) oracleadvm(P)(U) oracleoks(P)(U) autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand freq_table pcc_cpufreq bonding vfat fat emcpdm(P)(U) emcpmpx(P)(U) emcpgpx(P)(U) emcp(P)(U) microcode iTCO_wdt iTCO_vendor_support joydev hpilo hpwdt be2net serio_raw lpc_ich mfd_core bnx2x ptp pps_core libcrc32c mdio sg power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext3 jbd mbcache sd_mod hpsa(U) scsi_transport_sas lpfc scsi_transport_fc scsi_tgt crc_t10dif pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
kernel: Pid: 7622, comm: emcpadm Tainted: P           -- ------------    2.6.32-754.6.3.el6.x86_64 #1
kernel: Call Trace:
kernel: [<ffffffff810805c1>] ? warn_slowpath_common+0x91/0xe0
kernel: [<ffffffff8108062a>] ? warn_slowpath_null+0x1a/0x20
kernel: [<ffffffff812978af>] ? blk_throtl_drain+0xff/0x180
kernel: [<ffffffff81286cd1>] ? __blk_drain_queue+0x91/0x140
kernel: [<ffffffff81286fab>] ? blk_cleanup_queue+0xeb/0x1d0
kernel: [<ffffffffa0381494>] ? emcpCleanupPDev+0x84/0x120 [emcp]
kernel: [<ffffffffa038bfc7>] ? EmsLnxUnmap+0x127/0x1a0 [emcp]
kernel: [<ffffffffa0390cbf>] ? EmcpUnmapTopGuts+0x5f/0x100 [emcp]
kernel: [<ffffffffa0385a5b>] ? EmsSuspendIoAtTop+0xfb/0x150 [emcp]
kernel: [<ffffffffa0390ea9>] ? EmcpUnmapGuts+0x59/0xb0 [emcp]
kernel: [<ffffffffa03911d7>] ? EmcpUnmapDeviceId+0x67/0xc0 [emcp]
kernel: [<ffffffffa039126a>] ? EmcpIocmdUnmapDeviceId+0x3a/0x50 [emcp]
kernel: [<ffffffffa039419b>] ? PowerIocmdHandler+0x1bb/0x260 [emcp]
kernel: [<ffffffffa03943ae>] ? EmcpIocmd+0x16e/0x1e0 [emcp]
kernel: [<ffffffffa037cf8a>] ? PowerPlatformMalloc+0x6a/0x150 [emcp]
kernel: [<ffffffffa03945c4>] ? emcp_iocmd+0x1a4/0x200 [emcp]
kernel: [<ffffffffa0394946>] ? emcp_pseudo_ctl_ioctl+0x326/0x490 [emcp]
kernel: [<ffffffff811b71c2>] ? vfs_ioctl+0x92/0xc0
kernel: [<ffffffff811b7314>] ? do_vfs_ioctl+0x84/0x590
kernel: [<ffffffff811b78a1>] ? sys_ioctl+0x81/0xa0
kernel: [<ffffffff81564272>] ? system_call_after_swapgs+0xa2/0x152
kernel: [<ffffffff8156427e>] ? system_call_after_swapgs+0xae/0x152
kernel: [<ffffffff81564272>] ? system_call_after_swapgs+0xa2/0x152
kernel: [<ffffffff81564357>] ? system_call_fastpath+0x35/0x3a
kernel: [<ffffffff8156427e>] ? system_call_after_swapgs+0xae/0x152
kernel: ---[ end trace 7b23eba818a6efe0 ]---
Kernel panic - not syncing: Hard LOCKUP

Environment

  • Red Hat Enterprise Linux 6.10
  • kernel-2.6.32-754.6.3.el6
  • Either:
    • Veritas VxDMP
    • EMC PowerPath
    • Veeam Agent for Linux

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content