NULL pointer dereference often happens in rcu_segcblist_enqueue() with a "delete_controller" command being performed on a nvme controller

Solution Verified - Updated -

Issue

  • null-deref crash often happens in rcu_segcblist_enqueue() with a "delete_controller" command being performed on a nvme controller.
[454339.774039] nvme nvme1: Property Set error: -4, offset 0x14
[454339.780850] BUG: kernel NULL pointer dereference, address: 0000000000000000
[454339.791467] #PF: supervisor write access in kernel mode
[454339.798812] #PF: error_code(0x0002) - not-present page
[454339.807549] PGD 8000000343e47067 P4D 8000000343e47067 PUD 403c1d067 PMD 0 
[454339.817900] Oops: 0002 [#1] PREEMPT SMP PTI
[454339.825574] CPU: 27 PID: 3749920 Comm: sdt-4.5.0.260 Kdump: loaded Tainted: P        W IOE    --------  ---  5.14.0-284.18.1.el9_2.x86_64 #1
[454339.844957] Hardware name: Dell Inc. VxFlex-R740xd/01KPX8, BIOS 1.4.5 03/30/2018
[454339.855833] RIP: 0010:rcu_segcblist_enqueue+0x22/0x30
[454339.863853] Code: c3 cc cc cc cc 0f 1f 00 0f 1f 44 00 00 f0 48 83 47 48 01 48 8b 47 68 48 83 c0 01 48 89 47 68 48 c7 06 00 00 00 00 48 8b 47 20 <48> 89 30 48 89 77 20 c3 cc cc cc cc 66 90 0f 1f 44 00 00 48 8b 57
[454339.889893] RSP: 0018:ffff9c51280bfc58 EFLAGS: 00010002
[454339.898669] RAX: 0000000000000000 RBX: ffff8d8801e2a638 RCX: 0000000000000002
[454339.908524] RDX: 0000000000000001 RSI: ffff9c51280bfcb0 RDI: ffff8d8210000048
[454339.919308] RBP: ffff8d8210000000 R08: ffff8d9e0fb6b638 R09: ffff8d9e0fb6b638
[454339.930097] R10: 00000000000001b0 R11: 0000000000007410 R12: 0000000000000000
[454339.939884] R13: ffff9c51280bfcb0 R14: 0000000000000286 R15: ffff8d8210000040
[454339.950619] FS:  00007f1895d2b640(0000) GS:ffff8d9e0fb40000(0000) knlGS:0000000000000000
[454339.962279] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[454339.970788] CR2: 0000000000000000 CR3: 000000036829a001 CR4: 00000000007706e0
[454339.981508] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[454339.992243] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[454340.002674] PKRU: 55555554
[454340.008726] Call Trace:
[454340.014632]  <TASK>
[454340.019560]  srcu_gp_start_if_needed+0xe2/0x1e0
[454340.027377]  __synchronize_srcu.part.0+0x75/0xf0
[454340.034917]  ? __bpf_trace_rcu_stall_warning+0x10/0x10
[454340.043238]  nvme_tcp_delete_ctrl+0x34/0x70 [nvme_tcp]
[454340.051158]  nvme_do_delete_ctrl+0x63/0x6f [nvme_core]
[454340.059483]  nvme_sysfs_delete.cold+0x8/0xd [nvme_core]
[454340.067399]  kernfs_fop_write_iter+0x121/0x1b0
[454340.074926]  new_sync_write+0xfc/0x190
[454340.081729]  vfs_write+0x1ef/0x280
[454340.088065]  ksys_write+0x5f/0xe0
[454340.094228]  do_syscall_64+0x59/0x90
[454340.100151]  ? handle_mm_fault+0xc5/0x2a0
[454340.106935]  ? do_user_addr_fault+0x1d8/0x690
[454340.113903]  ? exc_page_fault+0x62/0x150
[454340.120442]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[454340.128146] RIP: 0033:0x7f18b433ebcf
[454340.134375] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d0 f5 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d0 f5 ff 48
[454340.158235] RSP: 002b:00007f1895d2a000 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[454340.168485] RAX: ffffffffffffffda RBX: 00007f1895d2b640 RCX: 00007f18b433ebcf
[454340.177878] RDX: 0000000000000001 RSI: 0000000000d61874 RDI: 0000000000000035
[454340.187681] RBP: 00007f1895d2a060 R08: 0000000000000000 R09: 00000000ffffffff
[454340.197454] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f1895d2b640
[454340.207222] R13: 0000000000000000 R14: 00007f18b429f530 R15: 0000000000000001
[454340.216973]  </TASK>
[454340.221732] Modules linked in: [...]
[454340.222032]  [last unloaded: ...]
[454340.317156] CR2: 0000000000000000

Environment

  • Red Hat Enterprise Linux 9.2.z - kernel-5.14.0-284.18.1.el9_2.x86_64
  • Dell VxFlex-R740xd
  • NVMe-TCP

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content