[RHEL 9][VxDMP]Panic in blk_mq_tag_busy() inlined from __blk_mq_alloc_requests() called from Veritas VxDMP
Issue
System crashes with a kernel stack trace:
PID: 368856 TASK: ff42868b5ce04700 CPU: 10 COMMAND: "dmpdaemon"
#0 [ff7b07ad64ff33e0] machine_kexec at ffffffff9d6780b7
#1 [ff7b07ad64ff3438] __crash_kexec at ffffffff9d7ef8ea
#2 [ff7b07ad64ff34f8] crash_kexec at ffffffff9d7f0b78
#3 [ff7b07ad64ff3500] oops_end at ffffffff9d62f98b
#4 [ff7b07ad64ff3520] page_fault_oops at ffffffff9d68a75b
#5 [ff7b07ad64ff3578] exc_page_fault at ffffffff9e286aa2
#6 [ff7b07ad64ff35a0] asm_exc_page_fault at ffffffff9e400c12
[exception RIP: __blk_mq_alloc_requests+0x238]
RIP: ffffffff9dbb0ae8 RSP: ff7b07ad64ff3658 RFLAGS: 00010246
RAX: ff42868affb40000 RBX: 0000000000004000 RCX: 0000000000000000
RDX: 0000000000000001 RSI: ff42867bcc4b4598 RDI: 0000000000000000
RBP: ff7b07ad64ff36b0 R8: ff42867bcc4b4598 R9: ff7b07ad64ff3784
R10: 0000000000000000 R11: 0000000000000100 R12: 0000000000000000
R13: ff42867bcc4b4598 R14: ff7b07ad64ff36b8 R15: 0000000000000001
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ff7b07ad64ff36a8] blk_mq_alloc_request at ffffffff9dbb0d4b
#8 [ff7b07ad64ff3720] dmp_get_request at ffffffffc3697da8 [vxdmp]
#9 [ff7b07ad64ff3728] dmp_setup_clone_req at ffffffffc369858f [vxdmp]
#10 [ff7b07ad64ff3760] gendmpstrategy at ffffffffc36a5ec9 [vxdmp]
#11 [ff7b07ad64ff37d0] vxdmp_queue_rq at ffffffffc3695167 [vxdmp]
#12 [ff7b07ad64ff37e8] __blk_mq_issue_directly at ffffffff9dbb232a
#13 [ff7b07ad64ff3828] blk_mq_try_issue_directly at ffffffff9dbb5792
#14 [ff7b07ad64ff3860] blk_mq_submit_bio at ffffffff9dbb6582
#15 [ff7b07ad64ff38d8] submit_bio_noacct_nocheck at ffffffff9dba5ce4
#16 [ff7b07ad64ff3938] submit_bh_wbc at ffffffff9da81c95
#17 [ff7b07ad64ff3958] block_read_full_folio at ffffffff9da84135
#18 [ff7b07ad64ff3a08] filemap_read_folio at ffffffff9d92e5f0
#19 [ff7b07ad64ff3aa8] do_read_cache_folio at ffffffff9d930662
#20 [ff7b07ad64ff3b60] read_cache_page at ffffffff9d930954
#21 [ff7b07ad64ff3b78] read_part_sector at ffffffff9dbc188a
#22 [ff7b07ad64ff3b98] read_lba at ffffffff9dbc421f
#23 [ff7b07ad64ff3be8] find_valid_gpt.constprop.0 at ffffffff9dbc4845
#24 [ff7b07ad64ff3c78] efi_partition at ffffffff9dbc4d40
#25 [ff7b07ad64ff3d40] check_partition at ffffffff9dbc1150
#26 [ff7b07ad64ff3d68] bdev_disk_changed at ffffffff9dbc13b8
#27 [ff7b07ad64ff3dd0] blkdev_get_whole at ffffffff9db9b6aa
#28 [ff7b07ad64ff3df0] blkdev_get_by_dev at ffffffff9db9c483
#29 [ff7b07ad64ff3e38] disk_scan_partitions at ffffffff9dbbd835
#30 [ff7b07ad64ff3e60] device_add_disk at ffffffff9dbbf06d
#31 [ff7b07ad64ff3e98] dmp_register_disk at ffffffffc3699170 [vxdmp]
#32 [ff7b07ad64ff3ed0] dmp_register_blk_device at ffffffffc3699312 [vxdmp]
#33 [ff7b07ad64ff3ee8] dmp_daemons_loop at ffffffffc36e3424 [vxdmp]
#34 [ff7b07ad64ff3f18] kthread at ffffffff9d735aad
#35 [ff7b07ad64ff3f50] ret_from_fork at ffffffff9d602c69
The dmpdaemon is registering a new disk and is reading a partition from it.
The console messages reflect this activity just a fraction of a second before hitting the BUG:
[23177.323857] VxVM vxdmp V-5-0-1990 driver version 8.0.2.1700 installed
[23177.323860]
[23182.295915] VxVM vxio V-5-0-1990 driver version 8.0.2.1700 installed
[23182.308154] VxVM vxspec V-5-0-1990 driver version 8.0.2.1700 installed
[23186.992217] VxVM vxdmp V-5-0-34 [Info] added disk array DISKS, datype = Disk
[23186.992220]
[23187.002512] VxVM vxdmp V-5-0-34 [Info] added disk array CN0WW56VFCP0049A036Y, datype = Dell_NVMe
[23187.002515]
[23187.015763] VxVM vxdmp V-5-3-0 dmp_decode_add_disk da_flags: 20402
[23187.022932]
[23187.025150] VxVM vxdmp V-5-3-0 dmp_decode_add_disk da_flags: a0202
Environment
- Red Hat Enterprise Linux 9
- Veritas VxDMP dynamic multipathing
- vxdmp module version "8.0.2.1700"
- NVMe disk / storage managed by VxDMP
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.