Massive iteration in find_get_entry() causes huge number of soft lockups and rcu_sched CPU stalls

Solution Unverified - Updated -

Issue

  • Massive iteration in find_get_entry() causes huge number of soft lockups and rcu_sched CPU stalls
[49754.048640] rcu: INFO: rcu_sched self-detected stall on CPU
[49754.048644] rcu:     5-....: (59997 ticks this GP) idle=bfe/1/0x4000000000000002 softirq=3748856/3748856 fqs=14869 
[49754.048650]  (t=60000 jiffies g=11257665 q=319355)
[49754.048653] NMI backtrace for cpu 5
[49754.048654] CPU: 5 PID: 2821175 Comm: TaniumCX Kdump: loaded Tainted: P    B      OEL   --------- -  - 4.18.0-477.15.1.el8_8.x86_64 #1
    ...
[49754.048726] RIP: 0010:xas_load+0x21/0x80
[49754.048729] Code: b1 41 00 66 0f 1f 44 00 00 49 89 f8 e8 28 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 05 e9 8f b1 41 00 <48> 8d 70 fe 0f b6 48 fe 41 38 48 10 77 ed 49 8b 50 08 48 d3 ea 83
[49754.048731] RSP: 0018:ffffc012eb5ffc88 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
[49754.048734] RAX: ffff9c894ab5d91a RBX: ffffe925e6a9ac00 RCX: 0000000000000006
[49754.048736] RDX: 0000000000000002 RSI: ffff9c894ab5e910 RDI: ffffc012eb5ffc90
[49754.048737] RBP: 000000000000028c R08: ffffc012eb5ffc90 R09: cf8948ffee6a73e8
[49754.048738] R10: 1047894c08578948 R11: 9000008529e13d83 R12: 0000000000000000
[49754.048740] R13: ffff9c894ab71398 R14: 000000000000028c R15: ffffffff8c68d320
[49754.048743]  ? xas_load+0x8/0x80
[49754.048745]  find_get_entry+0x5f/0x1c0
[49754.048749]  pagecache_get_page+0x30/0x310
[49754.048751]  generic_file_buffered_read+0x187/0xba0
[49754.048754]  ? __mod_lruvec_page_state+0x5e/0x80
[49754.048758]  ? handle_pte_fault+0x484/0x880
[49754.048760]  ? _cond_resched+0x15/0x30
[49754.048764]  ? down_read+0xe/0xa0
[49754.048768]  xfs_file_buffered_read+0x3e/0xb0 [xfs]
[49754.048863]  xfs_file_read_iter+0x6e/0xe0 [xfs]
[49754.048916]  new_sync_read+0x10f/0x160
[49754.048920]  vfs_read+0x91/0x150
[49754.048923]  ksys_read+0x4f/0xb0
[49754.048925]  do_syscall_64+0x5b/0x1b0
[49754.048929]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
    ...
[49779.501420] watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [TaniumCX:2821175]
    ...
[49779.502102] CPU: 5 PID: 2821175 Comm: TaniumCX Kdump: loaded Tainted: P    B      OEL   --------- -  - 4.18.0-477.15.1.el8_8.x86_64 #1
[49779.502105] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[49779.502106] RIP: 0010:find_get_entry+0x77/0x1c0
[49779.502112] Code: 00 48 c7 44 24 18 03 00 00 00 48 89 e7 e8 f1 a2 74 00 48 89 c3 48 3d 06 04 00 00 74 e4 48 3d 02 04 00 00 74 dc 48 85 c0 74 63 <a8> 01 75 5f 8b 40 34 85 c0 74 cc 8d 48 01 48 8d 53 34 f0 0f b1 4b
[49779.502115] RSP: 0018:ffffc012eb5ffc90 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[49779.502117] RAX: ffffe925e6a9ac00 RBX: ffffe925e6a9ac00 RCX: 0000000000000000
[49779.502119] RDX: 000000000000000c RSI: ffff9c894ab5e910 RDI: ffffc012eb5ffc90
[49779.502121] RBP: 000000000000028c R08: ffffc012eb5ffc90 R09: cf8948ffee6a73e8
[49779.502122] R10: 1047894c08578948 R11: 9000008529e13d83 R12: 0000000000000000
[49779.502123] R13: ffff9c894ab71398 R14: 000000000000028c R15: ffffffff8c68d320
[49779.502125] FS:  00007f87937fe700(0000) GS:ffff9c946fb40000(0000) knlGS:0000000000000000
[49779.502127] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[49779.502128] CR2: 00007f8784016000 CR3: 0000000b0b128003 CR4: 00000000007706e0
[49779.502174] PKRU: 55555554
[49779.502175] Call Trace:
[49779.502179]  pagecache_get_page+0x30/0x310
[49779.502182]  generic_file_buffered_read+0x187/0xba0
[49779.502185]  ? __mod_lruvec_page_state+0x5e/0x80
[49779.502189]  ? handle_pte_fault+0x484/0x880
[49779.502192]  ? _cond_resched+0x15/0x30
[49779.502197]  ? down_read+0xe/0xa0
[49779.502201]  xfs_file_buffered_read+0x3e/0xb0 [xfs]
[49779.502295]  xfs_file_read_iter+0x6e/0xe0 [xfs]
[49779.502349]  new_sync_read+0x10f/0x160
[49779.502352]  vfs_read+0x91/0x150
[49779.502355]  ksys_read+0x4f/0xb0
[49779.502357]  do_syscall_64+0x5b/0x1b0
[49779.502362]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
    ...

Environment

  • Red Hat Enterprise Linux 8.8.z - kernel-4.18.0-477.36.1.el8_8
  • VMware vSphere ESXi

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content