System crashes in fc_timed_out() or fc_eh_timed_out() with NULL pointer dereference while using storvsc driver

Solution Verified - Updated -

Issue

  • System crashes in fc_timed_out() with NULL pointer dereference at 0000000000000028:

    [354005.702276] sd 1:0:0:21: alua: port group 02 state A preferred supports tolUsNA
    [354005.702822] sd 1:0:0:22: [storvsc] Sense Key : Unit Attention [current] 
    [354005.702831] sd 1:0:0:22: [storvsc] Add. Sense: Power on, reset, or bus device reset occurred
    [354005.703131] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
    [354005.703206] IP: [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc]
    [354005.703287] PGD 0 
    [354005.703409] Oops: 0000 [#1] SMP 
    [354005.703445] Modules linked in: udp_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter joydev pcc_cpufreq dm_service_time vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_multipath hv_utils sg ptp pps_core binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c
    [354005.704058]  sd_mod crc_t10dif crct10dif_generic hv_storvsc scsi_transport_fc serio_raw hv_netvsc hyperv_fb scsi_tgt hid_hyperv hyperv_keyboard crct10dif_pclmul crct10dif_common crc32c_intel hv_vmbus dm_mirror dm_region_hash dm_log dm_mod
    [354005.704058] CPU: 0 PID: 8924 Comm: kworker/0:1H Kdump: loaded Not tainted 3.10.0-1062.1.2.el7.x86_64 #1
    [354005.704058] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
    [354005.704058] Workqueue: kblockd blk_timeout_work
    [354005.704058] task: ffff8b97eea6c1c0 ti: ffff8b97f359c000 task.ti: ffff8b97f359c000
    [354005.704058] RIP: 0010:[<ffffffffc01c919c>]  [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc]
    [354005.704058] RSP: 0018:ffff8b97f359fd88  EFLAGS: 00010003
    [354005.704058] RAX: 0000000000000000 RBX: ffff8b97b02d3d80 RCX: ffff8baf434da930
    ....
    [354005.704058] RDX: ffff8baf434da990 RSI: 0000000000002007 RDI: ffff8baf2de38d00
    [354005.704058] RBP: ffff8b97f359fda8 R08: ffff8b97b02d3ed0 R09: d19fbf98529fe818
    [354005.704058] R10: d19fbf98529fe818 R11: 7fffffffffffffff R12: ffff8baf2de38d00
    [354005.704058] R13: ffff8baf434da800 R14: ffff8b97f2ffe438 R15: ffff8b97b02d3ed0
    [354005.704058] FS:  0000000000000000(0000) GS:ffff8bab47600000(0000) knlGS:0000000000000000
    [354005.704058] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [354005.704058] CR2: 0000000000000028 CR3: 0000000466f50000 CR4: 00000000001606f0
    [354005.704058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [354005.704058] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
    [354005.704058] Call Trace:
    [354005.704058]  [<ffffffffa8ce1341>] ? scsi_times_out+0x61/0x240
    [354005.704058]  [<ffffffffa8b5706d>] blk_rq_timed_out+0x1d/0x70
    [354005.704058]  [<ffffffffa8b571b8>] blk_timeout_work+0xf8/0x150
    [354005.704058]  [<ffffffffa88bd0ff>] process_one_work+0x17f/0x440
    [354005.704058]  [<ffffffffa88be216>] worker_thread+0x126/0x3c0
    [354005.704058]  [<ffffffffa88be0f0>] ? manage_workers.isra.26+0x2a0/0x2a0
    [354005.704058]  [<ffffffffa88c50d1>] kthread+0xd1/0xe0
    [354005.704058]  [<ffffffffa88c5000>] ? insert_kthread_work+0x40/0x40
    [354005.704058]  [<ffffffffa8f8cd37>] ret_from_fork_nospec_begin+0x21/0x21
    [354005.977370] scsi host1: scsi_prep_async_scan called twice
    [354005.979520] scsi 1:0:1:0: Direct-Access     DGC      VRAID            4401 PQ: 0 ANSI: 6
    [354005.704058]  [<ffffffffa88c5000>] ? insert_kthread_work+0x40/0x40
    [354005.704058] Code: 1f 44 00 00 48 8b 07 55 48 89 e5 48 8b 80 48 01 00 00 5d 48 8b 10 31 c0 48 81 ba 80 02 00 00 80 92 1c c0 48 8d 4a a0 48 0f 44 c1 <83> 78 28 04 0f 94 c0 0f b6 c0 01 c0 c3 0f 1f 80 00 00 00 00 0f 
    [354005.704058] RIP  [<ffffffffc01c919c>] fc_timed_out+0x2c/0x40 [scsi_transport_fc]
    [354005.704058]  RSP <ffff8b97f359fd88>
    [354005.704058] CR2: 0000000000000028
    
  • Also seen crashing in fc_eh_timed_out():

    [6271488.438948] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
    [6271488.438998] PGD 80000002369ce067 P4D 80000002369ce067 PUD 236962067 PMD 0 
    [6271488.439038] Oops: 0000 [#1] SMP PTI
    [6271488.439059] CPU: 14 PID: 538 Comm: kworker/14:1H Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-372.19.1.el8_6.x86_64 #1
    [6271488.439117] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
    [6271488.439169] Workqueue: kblockd blk_mq_timeout_work
    [6271488.439199] RIP: 0010:fc_eh_timed_out+0x22/0x50 [scsi_transport_fc]
    ....
    [6271488.439697] Call Trace:
    [6271488.439713]  scsi_times_out+0x6b/0x240
    [6271488.439739]  ? __switch_to_asm+0x41/0x70
    [6271488.439763]  blk_mq_check_expired+0x10a/0x130
    [6271488.439789]  bt_iter+0x79/0x80
    [6271488.439809]  blk_mq_queue_tag_busy_iter+0x19f/0x2f0
    [6271488.439836]  ? blk_mq_put_rq_ref+0x40/0x40
    [6271488.439858]  ? __switch_to_asm+0x41/0x70
    [6271488.439880]  ? __switch_to_asm+0x35/0x70
    [6271488.439901]  ? blk_mq_put_rq_ref+0x40/0x40
    [6271488.439924]  ? entry_SYSCALL_64_after_hwframe+0xb8/0xca
    [6271488.439952]  blk_mq_timeout_work+0x54/0x120
    [6271488.439975]  process_one_work+0x1a7/0x360
    [6271488.439999]  ? create_worker+0x1a0/0x1a0
    [6271488.440021]  worker_thread+0x30/0x390
    [6271488.440042]  ? create_worker+0x1a0/0x1a0
    

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • VM configured in Microsoft Hyper-V
  • Devices connected through MS Hyper-V virtual SCSI adapters

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content