System hang due to busy spinlock in `AfdgSleepEvent` from the module `oracleafd`

Solution Verified - Updated -

Issue

  • System hangs with following messages in kernel ring buffer:
[342285.442217] INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 8, t=125343500 jiffies, g=10582083, c=10582082, q=0)
[342285.442910] sending NMI to all CPUs:
[342285.444034] NMI backtrace for cpu 0
[342285.444036] CPU: 0 PID: 13628 Comm: ora_arc3_frd11 Tainted: P           OEL ------------   3.10.0-327.62.1.el7.x86_64 #1
[342285.444036] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[342285.444037] task: ffff8808c8096780 ti: ffff8808bae74000 task.ti: ffff8808bae74000
[342285.444038] RIP: 0010:[<ffffffff8163e8c7>]  [<ffffffff8163e8c7>] _raw_spin_lock_irqsave+0x47/0x60
[342285.444039] RSP: 0018:ffff8808bae77af8  EFLAGS: 00000093
[342285.444040] RAX: 0000000000000d7e RBX: ffff8808bae77b58 RCX: 0000000000000000
[342285.444040] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8801005a10a0
[342285.444041] RBP: ffff8808bae77af8 R08: 0000000000000246 R09: 0000000002000000
[342285.444042] R10: 0000000000000001 R11: ffff8808bae77c88 R12: ffff8801005a10a0
[342285.444042] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[342285.444043] FS:  00007f8993a2f2c0(0000) GS:ffff880dc7600000(0000) knlGS:0000000000000000
[342285.444044] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[342285.444044] CR2: 000000000bf89620 CR3: 00000008c39a4000 CR4: 00000000001407f0
[342285.444045] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[342285.444046] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[342285.444047] Stack:
[342285.444047]  ffff8808bae77b30 ffffffff810a6877 0000000000000001 00000000ffffffff
[342285.444048]  ffff8801005a1098 ffff8801005a10a0 ffff8808c8096780 ffff8808bae77b90
[342285.444049]  ffffffffa04b2bc2 0000000000000000 ffff8808c8096780 ffffffff810a6b80
[342285.444049] Call Trace:
[342285.444050]  [<ffffffff810a6877>] prepare_to_wait+0x27/0x90
[342285.444051]  [<ffffffffa04b2bc2>] AfdgSleepEvent+0xb2/0x160 [oracleafd]
[342285.444052]  [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
[342285.444052]  [<ffffffffa04afcfd>] afdq_batch_reap+0x1dd/0x410 [oracleafd]
[342285.444053]  [<ffffffffa04a0a3b>] afdc_io+0x87b/0x1670 [oracleafd]
[342285.444054]  [<ffffffffa04a2e39>] afdc_execute_ioctl+0x8d9/0x1970 [oracleafd]
[342285.444054]  [<ffffffff8129e23d>] ? context_struct_compute_av+0x34d/0x470
[342285.444055]  [<ffffffff812f9632>] ? radix_tree_lookup_slot+0x22/0x50
[342285.444056]  [<ffffffff8116923e>] ? __find_get_page+0x1e/0xa0
[342285.444056]  [<ffffffffa049c0b8>] ? AFD_LOG_INT+0x58/0x80 [oracleafd]
[342285.444057]  [<ffffffffa04b4e5c>] afd_ioctl+0x17c/0x2d0 [oracleafd]
[342285.444058]  [<ffffffff812d9553>] blkdev_ioctl+0x193/0x760
[342285.444058]  [<ffffffff8121a481>] block_ioctl+0x41/0x50
[342285.444059]  [<ffffffff811f2a65>] do_vfs_ioctl+0x2e5/0x4c0
[342285.444060]  [<ffffffff8128c44e>] ? file_has_perm+0xae/0xc0
[342285.444060]  [<ffffffff811f2ce1>] SyS_ioctl+0xa1/0xc0
[342285.444061]  [<ffffffff81647549>] system_call_fastpath+0x16/0x1b

Another example of backtrace:

[2085196.985457] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=60002 jiffies, g=119782677, c=119782676, q=0)
[2085196.985469] sending NMI to all CPUs:
[2085196.986663] NMI backtrace for cpu 0
[2085196.986665] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           OE  ------------   3.10.0-327.53.1.el7.x86_64 #1
...
[2085196.986718] NMI backtrace for cpu 2
[2085196.986719] CPU: 2 PID: 146 Comm: ksoftirqd/2 Tainted: P           OE  ------------   3.10.0-327.53.1.el7.x86_64 #1
[2085196.986720] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
[2085196.986721] task: ffff880ac1f2f300 ti: ffff880ac1f64000 task.ti: ffff880ac1f64000
[2085196.986721] RIP: 0010:[<ffffffff8163e587>]  [<ffffffff8163e587>] _raw_spin_lock_irqsave+0x47/0x60
[2085196.986722] RSP: 0018:ffff880ac1f67d58  EFLAGS: 00000093
[2085196.986723] RAX: 0000000000006f93 RBX: ffff8802d250c498 RCX: 0000000000000000
[2085196.986724] RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff8802d250c498
[2085196.986724] RBP: ffff880ac1f67d58 R08: 0000000000000246 R09: ffff880ad36a0430
[2085196.986725] R10: ffffffff81417377 R11: ffffea0004077c00 R12: 20c49ba5e353f7cf
[2085196.986726] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
[2085196.986727] FS:  0000000000000000(0000) GS:ffff880ad3680000(0000) knlGS:0000000000000000
[2085196.986727] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[2085196.986728] CR2: 000000000c774c44 CR3: 000000034e207000 CR4: 00000000001407e0
[2085196.986729] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2085196.986729] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2085196.986730] Stack:
[2085196.986731]  ffff880ac1f67d90 ffffffff810b0df3 ffff8802d250c490 20c49ba5e353f7cf
[2085196.986732]  0000000000000000 ffffffff81943080 0000000000000001 ffff880ac1f67da8
[2085196.986733]  ffffffffa0462214 ffff8802d250c460 ffff880ac1f67dc8 ffffffffa046423b
[2085196.986733] Call Trace:
[2085196.986734]  [<ffffffff810b0df3>] __wake_up+0x23/0x50
[2085196.986735]  [<ffffffffa0462214>] AfdgWakeupAll+0x24/0x30 [oracleafd]
[2085196.986735]  [<ffffffffa046423b>] afd_done_tasklet+0xab/0x110 [oracleafd]
[2085196.986736]  [<ffffffff810848bd>] tasklet_hi_action+0x7d/0x140
[2085196.986737]  [<ffffffff81084baf>] __do_softirq+0xef/0x280
[2085196.986738]  [<ffffffff81084d70>] run_ksoftirqd+0x30/0x50
[2085196.986738]  [<ffffffff810ae5bf>] smpboot_thread_fn+0xff/0x1a0
[2085196.986739]  [<ffffffff8163c229>] ? schedule+0x29/0x70
[2085196.986740]  [<ffffffff810ae4c0>] ? lg_double_unlock+0x90/0x90
[2085196.986741]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[2085196.986741]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[2085196.986742]  [<ffffffff81647158>] ret_from_fork+0x58/0x90
[2085196.986743]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140

Environment

  • Red Hat Enterprise Linux 7

  • Unsigned (U) oracleafd kernel module

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content