Massive iteration in find_get_entry() causes huge number of soft lockups and rcu_sched CPU stalls
Issue
- Massive iteration in find_get_entry() causes huge number of soft lockups and rcu_sched CPU stalls
[49754.048640] rcu: INFO: rcu_sched self-detected stall on CPU
[49754.048644] rcu: 5-....: (59997 ticks this GP) idle=bfe/1/0x4000000000000002 softirq=3748856/3748856 fqs=14869
[49754.048650] (t=60000 jiffies g=11257665 q=319355)
[49754.048653] NMI backtrace for cpu 5
[49754.048654] CPU: 5 PID: 2821175 Comm: TaniumCX Kdump: loaded Tainted: P B OEL --------- - - 4.18.0-477.15.1.el8_8.x86_64 #1
...
[49754.048726] RIP: 0010:xas_load+0x21/0x80
[49754.048729] Code: b1 41 00 66 0f 1f 44 00 00 49 89 f8 e8 28 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 05 e9 8f b1 41 00 <48> 8d 70 fe 0f b6 48 fe 41 38 48 10 77 ed 49 8b 50 08 48 d3 ea 83
[49754.048731] RSP: 0018:ffffc012eb5ffc88 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[49754.048734] RAX: ffff9c894ab5d91a RBX: ffffe925e6a9ac00 RCX: 0000000000000006
[49754.048736] RDX: 0000000000000002 RSI: ffff9c894ab5e910 RDI: ffffc012eb5ffc90
[49754.048737] RBP: 000000000000028c R08: ffffc012eb5ffc90 R09: cf8948ffee6a73e8
[49754.048738] R10: 1047894c08578948 R11: 9000008529e13d83 R12: 0000000000000000
[49754.048740] R13: ffff9c894ab71398 R14: 000000000000028c R15: ffffffff8c68d320
[49754.048743] ? xas_load+0x8/0x80
[49754.048745] find_get_entry+0x5f/0x1c0
[49754.048749] pagecache_get_page+0x30/0x310
[49754.048751] generic_file_buffered_read+0x187/0xba0
[49754.048754] ? __mod_lruvec_page_state+0x5e/0x80
[49754.048758] ? handle_pte_fault+0x484/0x880
[49754.048760] ? _cond_resched+0x15/0x30
[49754.048764] ? down_read+0xe/0xa0
[49754.048768] xfs_file_buffered_read+0x3e/0xb0 [xfs]
[49754.048863] xfs_file_read_iter+0x6e/0xe0 [xfs]
[49754.048916] new_sync_read+0x10f/0x160
[49754.048920] vfs_read+0x91/0x150
[49754.048923] ksys_read+0x4f/0xb0
[49754.048925] do_syscall_64+0x5b/0x1b0
[49754.048929] entry_SYSCALL_64_after_hwframe+0x61/0xc6
...
[49779.501420] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [TaniumCX:2821175]
...
[49779.502102] CPU: 5 PID: 2821175 Comm: TaniumCX Kdump: loaded Tainted: P B OEL --------- - - 4.18.0-477.15.1.el8_8.x86_64 #1
[49779.502105] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[49779.502106] RIP: 0010:find_get_entry+0x77/0x1c0
[49779.502112] Code: 00 48 c7 44 24 18 03 00 00 00 48 89 e7 e8 f1 a2 74 00 48 89 c3 48 3d 06 04 00 00 74 e4 48 3d 02 04 00 00 74 dc 48 85 c0 74 63 <a8> 01 75 5f 8b 40 34 85 c0 74 cc 8d 48 01 48 8d 53 34 f0 0f b1 4b
[49779.502115] RSP: 0018:ffffc012eb5ffc90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[49779.502117] RAX: ffffe925e6a9ac00 RBX: ffffe925e6a9ac00 RCX: 0000000000000000
[49779.502119] RDX: 000000000000000c RSI: ffff9c894ab5e910 RDI: ffffc012eb5ffc90
[49779.502121] RBP: 000000000000028c R08: ffffc012eb5ffc90 R09: cf8948ffee6a73e8
[49779.502122] R10: 1047894c08578948 R11: 9000008529e13d83 R12: 0000000000000000
[49779.502123] R13: ffff9c894ab71398 R14: 000000000000028c R15: ffffffff8c68d320
[49779.502125] FS: 00007f87937fe700(0000) GS:ffff9c946fb40000(0000) knlGS:0000000000000000
[49779.502127] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[49779.502128] CR2: 00007f8784016000 CR3: 0000000b0b128003 CR4: 00000000007706e0
[49779.502174] PKRU: 55555554
[49779.502175] Call Trace:
[49779.502179] pagecache_get_page+0x30/0x310
[49779.502182] generic_file_buffered_read+0x187/0xba0
[49779.502185] ? __mod_lruvec_page_state+0x5e/0x80
[49779.502189] ? handle_pte_fault+0x484/0x880
[49779.502192] ? _cond_resched+0x15/0x30
[49779.502197] ? down_read+0xe/0xa0
[49779.502201] xfs_file_buffered_read+0x3e/0xb0 [xfs]
[49779.502295] xfs_file_read_iter+0x6e/0xe0 [xfs]
[49779.502349] new_sync_read+0x10f/0x160
[49779.502352] vfs_read+0x91/0x150
[49779.502355] ksys_read+0x4f/0xb0
[49779.502357] do_syscall_64+0x5b/0x1b0
[49779.502362] entry_SYSCALL_64_after_hwframe+0x61/0xc6
...
Environment
- Red Hat Enterprise Linux 8.8.z - kernel-4.18.0-477.36.1.el8_8
- VMware vSphere ESXi
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.