RHEL 8: Kernel panic at rb_insert_color with xprt_request_enqueue_receive

Issue

• Kernel panic with below logs:

[979931.201234] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[979931.201240] PGD 0 P4D 0 
[979931.201243] Oops: 0000 [#1] SMP PTI
[979931.201246] CPU: 7 PID: 318669 Comm: kworker/u16:2 Kdump: loaded Not tainted 4.18.0-372.13.1.el8_6.x86_64 #1
[979931.201248] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
[979931.201251] Workqueue: rpciod rpc_async_schedule [sunrpc]
[979931.201282] RIP: 0010:rb_insert_color+0x18/0x150
[979931.201287] Code: 00 00 00 00 e9 b7 ff ff ff 90 90 90 90 90 90 90 90 90 48 8b 07 48 85 c0 0f 84 37 01 00 00 48 8b 10 f6 c2 01 0f 85 33 01 00 00 <48> 8b 4a 08 49 89 d0 48 39 c1 74 4b 48 85 c9 74 05 f6 01 01 74 7c
[979931.201289] RSP: 0018:ffffb931a26abd90 EFLAGS: 00010246
[979931.201291] RAX: ffff9501a03ae6c8 RBX: ffff9501dcb7ce00 RCX: ffff9501a03ae6d0
[979931.201293] RDX: 0000000000000000 RSI: ffff94feaacca6c8 RDI: ffff9501dcb7cec8
[979931.201294] RBP: ffff9500743a1e90 R08: ffff9501a03ae6d8 R09: 0002d78bfff60060
[979931.201295] R10: e5f4610000000000 R11: 0002d78bfff60060 R12: ffff94feaacca000
[979931.201297] R13: ffff94feaacca678 R14: ffffffffc061d740 R15: ffffffffc060fc40
[979931.201298] FS:  0000000000000000(0000) GS:ffff95126bdc0000(0000) knlGS:0000000000000000
[979931.201300] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[979931.201301] CR2: 0000000000000008 CR3: 0000000169c10001 CR4: 00000000003706e0
[979931.201334] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[979931.201336] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[979931.201337] Call Trace:
[979931.201359]  xprt_request_enqueue_receive+0x128/0x170 [sunrpc]
[979931.201376]  call_encode+0x190/0x320 [sunrpc]
[979931.201391]  ? rpc_malloc+0x4d/0x90 [sunrpc]
[979931.201410]  ? call_decode+0x200/0x200 [sunrpc]
[979931.201423]  __rpc_execute+0x77/0x450 [sunrpc]
[979931.201441]  ? try_to_wake_up+0x61/0x510
[979931.201446]  rpc_async_schedule+0x29/0x40 [sunrpc]
[979931.201462]  process_one_work+0x1a7/0x360
[979931.201465]  ? create_worker+0x1a0/0x1a0
[979931.201466]  worker_thread+0x30/0x390
[979931.201468]  ? create_worker+0x1a0/0x1a0
[979931.201469]  kthread+0x10a/0x120
[979931.201473]  ? set_kthread_struct+0x40/0x40
[979931.201477]  ret_from_fork+0x35/0x40
[979931.201480] Modules linked in: mptcp_diag xsk_diag vsock_diag tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag nfs_layout_nfsv41_files rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm vmw_balloon crct10dif_pclmul vmw_vmci crc32_pclmul ghash_clmulni_intel rapl pcspkr joydev i2c_piix4 binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic sd_mod t10_pi vmwgfx sg ttm drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ahci drm libahci ata_piix libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse
[979931.201525] CR2: 0000000000000008

• Another pattern with logs

[435035.144118] general protection fault: 0000 [#1] SMP PTI
[435035.145348] CPU: 7 PID: 0 Comm: swapper/7 Kdump: loaded Not tainted 4.18.0-305.19.1.el8_4.x86_64 #1
[435035.146334] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.11599183.B64.1901030933 01/03/2019
[435035.148435] RIP: 0010:run_timer_softirq+0x126/0x410
[435035.149505] Code: 4c 8d 6c c4 08 49 8b 45 00 48 85 c0 74 db 4d 8b 75 00 4c 89 73 08 0f 1f 44 00 00 49 8b 06 49 8b 56 08 48 89 02 48 85 c0 74 04 <48> 89 50 08 49 c7 46 08 00 00 00 00 49 8b 76 18 48 89 df 48 b8 00
[435035.151896] RSP: 0018:ffffbc6d03398ef0 EFLAGS: 00010086
[435035.152939] RAX: dead000000000200 RBX: ffffa0c5f3ddaac0 RCX: 0000000119e982c0
[435035.154073] RDX: ffffbc6d03398ef8 RSI: 0000000000000005 RDI: ffffa0c5f3ddaae8
[435035.155274] RBP: 0000000000000000 R08: ffffa0c5f3ddb5c8 R09: ffffbc6d03398f00
[435035.156399] R10: ffffa0c5f3ddab30 R11: 0000000000000000 R12: 0000000119e98000
[435035.157447] R13: ffffbc6d03398ef8 R14: ffffa0c3ed121600 R15: 0000000000000202
[435035.158495] FS:  0000000000000000(0000) GS:ffffa0c5f3dc0000(0000) knlGS:0000000000000000
[435035.159502] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[435035.160510] CR2: 00007f6241ba60c0 CR3: 0000000574e10002 CR4: 00000000001706e0
[435035.161686] Call Trace:
[435035.162748]  <IRQ>
[435035.163783]  ? __hrtimer_run_queues+0x110/0x280
[435035.164977]  ? recalibrate_cpu_khz+0x10/0x10
[435035.166145]  ? ktime_get+0x36/0xa0
[435035.167205]  __do_softirq+0xd7/0x2d6
[435035.168306]  irq_exit+0xf7/0x100
[435035.169409]  smp_apic_timer_interrupt+0x74/0x130
[435035.170488]  apic_timer_interrupt+0xf/0x20
[435035.171749]  </IRQ>
[435035.172977] RIP: 0010:native_safe_halt+0xe/0x10
[435035.174206] Code: ff ff 7f c3 65 48 8b 04 25 40 5c 01 00 f0 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 90 e9 07 00 00 00 0f 00 2d e6 bf 4b 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d d6 bf 4b 00 f4 c3 90 90 0f 1f 44 00
[435035.177319] RSP: 0018:ffffbc6d031e3e38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13