System crashes in fc_timed_out() or fc_eh_timed_out() with NULL pointer dereference while using storvsc driver
Issue
-
System crashes in
fc_timed_out()
with NULL pointer dereference at 0000000000000028:[354005.702276] sd 1:0:0:21: alua: port group 02 state A preferred supports tolUsNA [354005.702822] sd 1:0:0:22: [storvsc] Sense Key : Unit Attention [current] [354005.702831] sd 1:0:0:22: [storvsc] Add. Sense: Power on, reset, or bus device reset occurred [354005.703131] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028 [354005.703206] IP: [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc] [354005.703287] PGD 0 [354005.703409] Oops: 0000 [#1] SMP [354005.703445] Modules linked in: udp_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter joydev pcc_cpufreq dm_service_time vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_multipath hv_utils sg ptp pps_core binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c [354005.704058] sd_mod crc_t10dif crct10dif_generic hv_storvsc scsi_transport_fc serio_raw hv_netvsc hyperv_fb scsi_tgt hid_hyperv hyperv_keyboard crct10dif_pclmul crct10dif_common crc32c_intel hv_vmbus dm_mirror dm_region_hash dm_log dm_mod [354005.704058] CPU: 0 PID: 8924 Comm: kworker/0:1H Kdump: loaded Not tainted 3.10.0-1062.1.2.el7.x86_64 #1 [354005.704058] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012 [354005.704058] Workqueue: kblockd blk_timeout_work [354005.704058] task: ffff8b97eea6c1c0 ti: ffff8b97f359c000 task.ti: ffff8b97f359c000 [354005.704058] RIP: 0010:[<ffffffffc01c919c>] [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc] [354005.704058] RSP: 0018:ffff8b97f359fd88 EFLAGS: 00010003 [354005.704058] RAX: 0000000000000000 RBX: ffff8b97b02d3d80 RCX: ffff8baf434da930 .... [354005.704058] RDX: ffff8baf434da990 RSI: 0000000000002007 RDI: ffff8baf2de38d00 [354005.704058] RBP: ffff8b97f359fda8 R08: ffff8b97b02d3ed0 R09: d19fbf98529fe818 [354005.704058] R10: d19fbf98529fe818 R11: 7fffffffffffffff R12: ffff8baf2de38d00 [354005.704058] R13: ffff8baf434da800 R14: ffff8b97f2ffe438 R15: ffff8b97b02d3ed0 [354005.704058] FS: 0000000000000000(0000) GS:ffff8bab47600000(0000) knlGS:0000000000000000 [354005.704058] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [354005.704058] CR2: 0000000000000028 CR3: 0000000466f50000 CR4: 00000000001606f0 [354005.704058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [354005.704058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [354005.704058] Call Trace: [354005.704058] [<ffffffffa8ce1341>] ? scsi_times_out+0x61/0x240 [354005.704058] [<ffffffffa8b5706d>] blk_rq_timed_out+0x1d/0x70 [354005.704058] [<ffffffffa8b571b8>] blk_timeout_work+0xf8/0x150 [354005.704058] [<ffffffffa88bd0ff>] process_one_work+0x17f/0x440 [354005.704058] [<ffffffffa88be216>] worker_thread+0x126/0x3c0 [354005.704058] [<ffffffffa88be0f0>] ? manage_workers.isra.26+0x2a0/0x2a0 [354005.704058] [<ffffffffa88c50d1>] kthread+0xd1/0xe0 [354005.704058] [<ffffffffa88c5000>] ? insert_kthread_work+0x40/0x40 [354005.704058] [<ffffffffa8f8cd37>] ret_from_fork_nospec_begin+0x21/0x21 [354005.977370] scsi host1: scsi_prep_async_scan called twice [354005.979520] scsi 1:0:1:0: Direct-Access DGC VRAID 4401 PQ: 0 ANSI: 6 [354005.704058] [<ffffffffa88c5000>] ? insert_kthread_work+0x40/0x40 [354005.704058] Code: 1f 44 00 00 48 8b 07 55 48 89 e5 48 8b 80 48 01 00 00 5d 48 8b 10 31 c0 48 81 ba 80 02 00 00 80 92 1c c0 48 8d 4a a0 48 0f 44 c1 <83> 78 28 04 0f 94 c0 0f b6 c0 01 c0 c3 0f 1f 80 00 00 00 00 0f [354005.704058] RIP [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc] [354005.704058] RSP <ffff8b97f359fd88> [354005.704058] CR2: 0000000000000028
-
Also seen crashing in
fc_eh_timed_out()
:[6271488.438948] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8 [6271488.438998] PGD 80000002369ce067 P4D 80000002369ce067 PUD 236962067 PMD 0 [6271488.439038] Oops: 0000 [#1] SMP PTI [6271488.439059] CPU: 14 PID: 538 Comm: kworker/14:1H Kdump: loaded Tainted: P OE --------- - - 4.18.0-372.19.1.el8_6.x86_64 #1 [6271488.439117] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012 [6271488.439169] Workqueue: kblockd blk_mq_timeout_work [6271488.439199] RIP: 0010:fc_eh_timed_out+0x22/0x50 [scsi_transport_fc] .... [6271488.439697] Call Trace: [6271488.439713] scsi_times_out+0x6b/0x240 [6271488.439739] ? __switch_to_asm+0x41/0x70 [6271488.439763] blk_mq_check_expired+0x10a/0x130 [6271488.439789] bt_iter+0x79/0x80 [6271488.439809] blk_mq_queue_tag_busy_iter+0x19f/0x2f0 [6271488.439836] ? blk_mq_put_rq_ref+0x40/0x40 [6271488.439858] ? __switch_to_asm+0x41/0x70 [6271488.439880] ? __switch_to_asm+0x35/0x70 [6271488.439901] ? blk_mq_put_rq_ref+0x40/0x40 [6271488.439924] ? entry_SYSCALL_64_after_hwframe+0xb8/0xca [6271488.439952] blk_mq_timeout_work+0x54/0x120 [6271488.439975] process_one_work+0x1a7/0x360 [6271488.439999] ? create_worker+0x1a0/0x1a0 [6271488.440021] worker_thread+0x30/0x390 [6271488.440042] ? create_worker+0x1a0/0x1a0
Environment
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 9
- VM configured in Microsoft Hyper-V
- Devices connected through MS Hyper-V virtual SCSI adapters
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.