RHEL8: Soft lockup and/or kernel panic in scan_swap_map_slots()

Issue

• Kernel panic in scan_swap_map_slots() with the following call traces:

[206906.757183] BUG: unable to handle kernel paging request at ffffbfe688eb5001
[206906.762089] PGD 107d46067 P4D 107d46067 PUD 107d47067 PMD fbd3b5067 PTE 0
[206906.766847] Oops: 0000 [#1] SMP PTI
[206906.770148] CPU: 0 PID: 109 Comm: kswapd0 Kdump: loaded Not tainted 4.18.0-305.el8.x86_64 #1
[206906.777144] Hardware name: Amazon EC2 r5.2xlarge/, BIOS 1.0 10/16/2017
[206906.781790] RIP: 0010:scan_swap_map_slots+0x2c7/0x860
[206906.785757] Code: 64 a0 00 00 48 8b 3d 40 40 dd 01 48 8b 44 24 20 48 83 c0 01 48 89 44 24 20 48 39 f0 0f 87 73 01 00 00 49 8b 94 24 40 a0 00 00 <0f> b6 0c 02 84 c9 0f 84 8b 01 00 00 48 8b 15 16 40 dd 01 48 01 d2
[206906.798865] RSP: 0018:ffffbfe686f139c0 EFLAGS: 00010293
[206906.802978] RAX: 000000000003c001 RBX: 0000000000000000 RCX: ffff9f91fb267400
[206906.809399] RDX: ffffbfe688e79000 RSI: 000000000003ffff RDI: 000000000003ffff
[206906.815840] RBP: 0000000000000100 R08: ffff9f91fb267400 R09: 000000000000001d
[206906.822248] R10: ffffffff9d01fb00 R11: 0000000000000001 R12: ffff9f83495c0000
[206906.828717] R13: 0000000000000000 R14: ffff9f83495c0000 R15: ffff9f83495c0050
[206906.835138] FS:  0000000000000000(0000) GS:ffff9f920dc00000(0000) knlGS:0000000000000000
[206906.841995] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[206906.846293] CR2: ffffbfe688eb5001 CR3: 0000000ab9410005 CR4: 00000000007706f0
[206906.852744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[206906.859188] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[206906.865660] PKRU: 55555554
[206906.868598] Call Trace:
[206906.871448]  get_swap_pages+0x22e/0x380
[206906.874924]  get_swap_page+0xf3/0x210
[206906.878295]  add_to_swap+0x14/0x70
[206906.881560]  shrink_page_list+0x833/0xbd0
[206906.885102]  shrink_inactive_list+0x19f/0x3e0
[206906.888794]  shrink_lruvec+0x3a9/0x640
[206906.892208]  shrink_node+0x184/0x6a0
[206906.895546]  balance_pgdat+0x2d7/0x550
[206907.010556]  ? __switch_to_asm+0x41/0x70
[206907.014070]  kswapd+0x1f9/0x3c0
[206907.017241]  ? finish_wait+0x80/0x80
[206907.020581]  ? balance_pgdat+0x550/0x550
[206907.024101]  kthread+0x116/0x130
[206907.027320]  ? kthread_flush_work_fn+0x10/0x10
[206907.031044]  ret_from_fork+0x35/0x40
[206907.034387] Modules linked in: ib_core rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm ppdev rapl parport_pc pcspkr i2c_piix4 parport binfmt_misc ip_tables xfs libcrc32c crct10dif_pclmul crc32_pclmul crc32c_intel nvme serio_raw ghash_clmulni_intel nvme_core ena t10_pi sunrpc dm_mirror dm_region_hash dm_log dm_mod
[206907.059304] CR2: ffffbfe688eb5001

• A soft lockup in scan_swap_map_slots() with the following call traces:

kernel: watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [kswapd0:107]
kernel: Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm pp
dev rapl pcspkr parport_pc i2c_piix4 parport binfmt_misc ip_tables xfs libcrc32c crct10dif_pclmul crc32_pclmul crc32c_intel nvme ghash_clmulni_intel serio_raw nvme_core ena t10_pi sunrpc dm_
mirror dm_region_hash dm_log dm_mod
kernel: CPU: 6 PID: 107 Comm: kswapd0 Kdump: loaded Not tainted 4.18.0-305.el8.x86_64 #1
kernel: Hardware name: Amazon EC2 r5.2xlarge/, BIOS 1.0 10/16/2017
kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x20/0x1d0
kernel: Code: ff ff 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 0f 1f 44 00 00 ba 01 00 00 00 8b 07 85 c0 75 09 f0 0f b1 17 85 c0 75 f2 c3 f3 90 <eb> ed 81 fe 00 01 00 
00 74 40 81 e6 00 ff ff ff 75 6d f0 0f ba 2f
kernel: RSP: 0018:ffffafc646ecb9b0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
kernel: RAX: 00000000000000fd RBX: 0000000000000000 RCX: 000000000003ffff
kernel: RDX: 0000000000000001 RSI: 00000000000000fd RDI: ffff9a80bb362000
kernel: RBP: 0000000000000100 R08: ffff9a80c4254e00 R09: 000000000000001d
kernel: R10: ffffffff9401fb00 R11: 0000000000000001 R12: ffff9a80cbc00000
kernel: R13: 0000000000000000 R14: 03ffffffffffffff R15: ffff9a80bb362000
kernel: FS:  0000000000000000(0000) GS:ffff9a80cdd80000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007fad5e878000 CR3: 0000000eb1a10002 CR4: 00000000007706e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: _raw_spin_lock+0x1c/0x20
kernel: scan_swap_map_slots+0x17e/0x860
kernel: get_swap_pages+0x22e/0x380
kernel: get_swap_page+0xf3/0x210
kernel: add_to_swap+0x14/0x70
kernel: shrink_page_list+0x833/0xbd0
kernel: shrink_inactive_list+0x19f/0x3e0
kernel: shrink_lruvec+0x3a9/0x640
kernel: shrink_node+0x184/0x6a0
kernel: balance_pgdat+0x2d7/0x550
kernel: ? __switch_to_asm+0x41/0x70
kernel: kswapd+0x1f9/0x3c0
kernel: ? finish_wait+0x80/0x80
kernel: ? balance_pgdat+0x550/0x550
kernel: kthread+0x116/0x130
kernel: ? kthread_flush_work_fn+0x10/0x10
kernel: ret_from_fork+0x35/0x40