[RHEL 9][VxDMP]Panic in blk_mq_tag_busy() inlined from __blk_mq_alloc_requests() called from Veritas VxDMP

Solution Verified - Updated -

Issue

System crashes with a kernel stack trace:

PID: 368856   TASK: ff42868b5ce04700  CPU: 10   COMMAND: "dmpdaemon"
 #0 [ff7b07ad64ff33e0] machine_kexec at ffffffff9d6780b7
 #1 [ff7b07ad64ff3438] __crash_kexec at ffffffff9d7ef8ea
 #2 [ff7b07ad64ff34f8] crash_kexec at ffffffff9d7f0b78
 #3 [ff7b07ad64ff3500] oops_end at ffffffff9d62f98b
 #4 [ff7b07ad64ff3520] page_fault_oops at ffffffff9d68a75b
 #5 [ff7b07ad64ff3578] exc_page_fault at ffffffff9e286aa2
 #6 [ff7b07ad64ff35a0] asm_exc_page_fault at ffffffff9e400c12
    [exception RIP: __blk_mq_alloc_requests+0x238]
    RIP: ffffffff9dbb0ae8  RSP: ff7b07ad64ff3658  RFLAGS: 00010246
    RAX: ff42868affb40000  RBX: 0000000000004000  RCX: 0000000000000000
    RDX: 0000000000000001  RSI: ff42867bcc4b4598  RDI: 0000000000000000
    RBP: ff7b07ad64ff36b0   R8: ff42867bcc4b4598   R9: ff7b07ad64ff3784
    R10: 0000000000000000  R11: 0000000000000100  R12: 0000000000000000
    R13: ff42867bcc4b4598  R14: ff7b07ad64ff36b8  R15: 0000000000000001
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ff7b07ad64ff36a8] blk_mq_alloc_request at ffffffff9dbb0d4b
 #8 [ff7b07ad64ff3720] dmp_get_request at ffffffffc3697da8 [vxdmp]
 #9 [ff7b07ad64ff3728] dmp_setup_clone_req at ffffffffc369858f [vxdmp]
#10 [ff7b07ad64ff3760] gendmpstrategy at ffffffffc36a5ec9 [vxdmp]
#11 [ff7b07ad64ff37d0] vxdmp_queue_rq at ffffffffc3695167 [vxdmp]
#12 [ff7b07ad64ff37e8] __blk_mq_issue_directly at ffffffff9dbb232a
#13 [ff7b07ad64ff3828] blk_mq_try_issue_directly at ffffffff9dbb5792
#14 [ff7b07ad64ff3860] blk_mq_submit_bio at ffffffff9dbb6582
#15 [ff7b07ad64ff38d8] submit_bio_noacct_nocheck at ffffffff9dba5ce4
#16 [ff7b07ad64ff3938] submit_bh_wbc at ffffffff9da81c95
#17 [ff7b07ad64ff3958] block_read_full_folio at ffffffff9da84135
#18 [ff7b07ad64ff3a08] filemap_read_folio at ffffffff9d92e5f0
#19 [ff7b07ad64ff3aa8] do_read_cache_folio at ffffffff9d930662
#20 [ff7b07ad64ff3b60] read_cache_page at ffffffff9d930954
#21 [ff7b07ad64ff3b78] read_part_sector at ffffffff9dbc188a
#22 [ff7b07ad64ff3b98] read_lba at ffffffff9dbc421f
#23 [ff7b07ad64ff3be8] find_valid_gpt.constprop.0 at ffffffff9dbc4845
#24 [ff7b07ad64ff3c78] efi_partition at ffffffff9dbc4d40
#25 [ff7b07ad64ff3d40] check_partition at ffffffff9dbc1150
#26 [ff7b07ad64ff3d68] bdev_disk_changed at ffffffff9dbc13b8
#27 [ff7b07ad64ff3dd0] blkdev_get_whole at ffffffff9db9b6aa
#28 [ff7b07ad64ff3df0] blkdev_get_by_dev at ffffffff9db9c483
#29 [ff7b07ad64ff3e38] disk_scan_partitions at ffffffff9dbbd835
#30 [ff7b07ad64ff3e60] device_add_disk at ffffffff9dbbf06d
#31 [ff7b07ad64ff3e98] dmp_register_disk at ffffffffc3699170 [vxdmp]
#32 [ff7b07ad64ff3ed0] dmp_register_blk_device at ffffffffc3699312 [vxdmp]
#33 [ff7b07ad64ff3ee8] dmp_daemons_loop at ffffffffc36e3424 [vxdmp]
#34 [ff7b07ad64ff3f18] kthread at ffffffff9d735aad
#35 [ff7b07ad64ff3f50] ret_from_fork at ffffffff9d602c69

The dmpdaemon is registering a new disk and is reading a partition from it.

The console messages reflect this activity just a fraction of a second before hitting the BUG:

[23177.323857] VxVM vxdmp V-5-0-1990 driver version 8.0.2.1700 installed
[23177.323860] 
[23182.295915] VxVM vxio V-5-0-1990 driver version 8.0.2.1700 installed
[23182.308154] VxVM vxspec V-5-0-1990 driver version 8.0.2.1700 installed
[23186.992217] VxVM vxdmp V-5-0-34 [Info] added disk array DISKS, datype = Disk
[23186.992220] 
[23187.002512] VxVM vxdmp V-5-0-34 [Info] added disk array CN0WW56VFCP0049A036Y, datype = Dell_NVMe
[23187.002515] 
[23187.015763] VxVM vxdmp V-5-3-0 dmp_decode_add_disk da_flags: 20402
[23187.022932] 
[23187.025150] VxVM vxdmp V-5-3-0 dmp_decode_add_disk da_flags: a0202

Environment

  • Red Hat Enterprise Linux 9
  • Veritas VxDMP dynamic multipathing
  • vxdmp module version "8.0.2.1700"
  • NVMe disk / storage managed by VxDMP

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content