System hang due to busy spinlock in `AfdgSleepEvent` from the module `oracleafd`
Issue
- System hangs with following messages in kernel ring buffer:
[342285.442217] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 8, t=125343500 jiffies, g=10582083, c=10582082, q=0)
[342285.442910] sending NMI to all CPUs:
[342285.444034] NMI backtrace for cpu 0
[342285.444036] CPU: 0 PID: 13628 Comm: ora_arc3_frd11 Tainted: P OEL ------------ 3.10.0-327.62.1.el7.x86_64 #1
[342285.444036] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[342285.444037] task: ffff8808c8096780 ti: ffff8808bae74000 task.ti: ffff8808bae74000
[342285.444038] RIP: 0010:[<ffffffff8163e8c7>] [<ffffffff8163e8c7>] _raw_spin_lock_irqsave+0x47/0x60
[342285.444039] RSP: 0018:ffff8808bae77af8 EFLAGS: 00000093
[342285.444040] RAX: 0000000000000d7e RBX: ffff8808bae77b58 RCX: 0000000000000000
[342285.444040] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8801005a10a0
[342285.444041] RBP: ffff8808bae77af8 R08: 0000000000000246 R09: 0000000002000000
[342285.444042] R10: 0000000000000001 R11: ffff8808bae77c88 R12: ffff8801005a10a0
[342285.444042] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[342285.444043] FS: 00007f8993a2f2c0(0000) GS:ffff880dc7600000(0000) knlGS:0000000000000000
[342285.444044] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[342285.444044] CR2: 000000000bf89620 CR3: 00000008c39a4000 CR4: 00000000001407f0
[342285.444045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[342285.444046] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[342285.444047] Stack:
[342285.444047] ffff8808bae77b30 ffffffff810a6877 0000000000000001 00000000ffffffff
[342285.444048] ffff8801005a1098 ffff8801005a10a0 ffff8808c8096780 ffff8808bae77b90
[342285.444049] ffffffffa04b2bc2 0000000000000000 ffff8808c8096780 ffffffff810a6b80
[342285.444049] Call Trace:
[342285.444050] [<ffffffff810a6877>] prepare_to_wait+0x27/0x90
[342285.444051] [<ffffffffa04b2bc2>] AfdgSleepEvent+0xb2/0x160 [oracleafd]
[342285.444052] [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[342285.444052] [<ffffffffa04afcfd>] afdq_batch_reap+0x1dd/0x410 [oracleafd]
[342285.444053] [<ffffffffa04a0a3b>] afdc_io+0x87b/0x1670 [oracleafd]
[342285.444054] [<ffffffffa04a2e39>] afdc_execute_ioctl+0x8d9/0x1970 [oracleafd]
[342285.444054] [<ffffffff8129e23d>] ? context_struct_compute_av+0x34d/0x470
[342285.444055] [<ffffffff812f9632>] ? radix_tree_lookup_slot+0x22/0x50
[342285.444056] [<ffffffff8116923e>] ? __find_get_page+0x1e/0xa0
[342285.444056] [<ffffffffa049c0b8>] ? AFD_LOG_INT+0x58/0x80 [oracleafd]
[342285.444057] [<ffffffffa04b4e5c>] afd_ioctl+0x17c/0x2d0 [oracleafd]
[342285.444058] [<ffffffff812d9553>] blkdev_ioctl+0x193/0x760
[342285.444058] [<ffffffff8121a481>] block_ioctl+0x41/0x50
[342285.444059] [<ffffffff811f2a65>] do_vfs_ioctl+0x2e5/0x4c0
[342285.444060] [<ffffffff8128c44e>] ? file_has_perm+0xae/0xc0
[342285.444060] [<ffffffff811f2ce1>] SyS_ioctl+0xa1/0xc0
[342285.444061] [<ffffffff81647549>] system_call_fastpath+0x16/0x1b
Another example of backtrace:
[2085196.985457] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=60002 jiffies, g=119782677, c=119782676, q=0)
[2085196.985469] sending NMI to all CPUs:
[2085196.986663] NMI backtrace for cpu 0
[2085196.986665] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P OE ------------ 3.10.0-327.53.1.el7.x86_64 #1
...
[2085196.986718] NMI backtrace for cpu 2
[2085196.986719] CPU: 2 PID: 146 Comm: ksoftirqd/2 Tainted: P OE ------------ 3.10.0-327.53.1.el7.x86_64 #1
[2085196.986720] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[2085196.986721] task: ffff880ac1f2f300 ti: ffff880ac1f64000 task.ti: ffff880ac1f64000
[2085196.986721] RIP: 0010:[<ffffffff8163e587>] [<ffffffff8163e587>] _raw_spin_lock_irqsave+0x47/0x60
[2085196.986722] RSP: 0018:ffff880ac1f67d58 EFLAGS: 00000093
[2085196.986723] RAX: 0000000000006f93 RBX: ffff8802d250c498 RCX: 0000000000000000
[2085196.986724] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8802d250c498
[2085196.986724] RBP: ffff880ac1f67d58 R08: 0000000000000246 R09: ffff880ad36a0430
[2085196.986725] R10: ffffffff81417377 R11: ffffea0004077c00 R12: 20c49ba5e353f7cf
[2085196.986726] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
[2085196.986727] FS: 0000000000000000(0000) GS:ffff880ad3680000(0000) knlGS:0000000000000000
[2085196.986727] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[2085196.986728] CR2: 000000000c774c44 CR3: 000000034e207000 CR4: 00000000001407e0
[2085196.986729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2085196.986729] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2085196.986730] Stack:
[2085196.986731] ffff880ac1f67d90 ffffffff810b0df3 ffff8802d250c490 20c49ba5e353f7cf
[2085196.986732] 0000000000000000 ffffffff81943080 0000000000000001 ffff880ac1f67da8
[2085196.986733] ffffffffa0462214 ffff8802d250c460 ffff880ac1f67dc8 ffffffffa046423b
[2085196.986733] Call Trace:
[2085196.986734] [<ffffffff810b0df3>] __wake_up+0x23/0x50
[2085196.986735] [<ffffffffa0462214>] AfdgWakeupAll+0x24/0x30 [oracleafd]
[2085196.986735] [<ffffffffa046423b>] afd_done_tasklet+0xab/0x110 [oracleafd]
[2085196.986736] [<ffffffff810848bd>] tasklet_hi_action+0x7d/0x140
[2085196.986737] [<ffffffff81084baf>] __do_softirq+0xef/0x280
[2085196.986738] [<ffffffff81084d70>] run_ksoftirqd+0x30/0x50
[2085196.986738] [<ffffffff810ae5bf>] smpboot_thread_fn+0xff/0x1a0
[2085196.986739] [<ffffffff8163c229>] ? schedule+0x29/0x70
[2085196.986740] [<ffffffff810ae4c0>] ? lg_double_unlock+0x90/0x90
[2085196.986741] [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[2085196.986741] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[2085196.986742] [<ffffffff81647158>] ret_from_fork+0x58/0x90
[2085196.986743] [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
Environment
-
Red Hat Enterprise Linux 7
-
Unsigned (U)
oracleafd
kernel module
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.