RHEL7: Kernel Panic in fscache_object_work_func or in fscache_put_operation with "fs/fscache/operation.c:494!".
Issue
The server crashes in fscache_object_work_func with Bad RIP value when under load with the following backtrace:
[94207.863822] Call Trace:
[94207.863971] [<ffffffffa03e4695>] ? fscache_object_work_func+0xa5/0x200 [fscache]
[94207.864262] [<ffffffff8109d5db>] process_one_work+0x17b/0x470
[94207.864411] [<ffffffff8109e4ac>] worker_thread+0x21c/0x400
[94207.864560] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
[94207.864711] [<ffffffff810a5acf>] kthread+0xcf/0xe0
[94207.864860] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
[94207.865012] [<ffffffff816460d8>] ret_from_fork+0x58/0x90
[94207.865160] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
[94207.865307] Code: Bad RIP value.
[94207.865457] RIP [<0000000000000002>] 0x1
[94207.865607] RSP <ffff8809b477bdf8>
[94207.865750] CR2: 0000000000000002
[94207.866339] --[ end trace 9498ae93c476f8c5 ]--
Another server crashed in fscache_put_operation with the following backtrace:
[4352864.053179] ------------[ cut here ]------------
[4352864.053327] kernel BUG at fs/fscache/operation.c:494!
[4352864.053486] invalid opcode: 0000 [#1] SMP
[4352864.056175] CPU: 19 PID: 1562 Comm: kworker/u48:0 Not tainted 3.10.0-514.12.1.el7.x86_64 #1
[4352864.056464] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 02/17/2017
[4352864.056768] Workqueue: fscache_operation fscache_op_work_func [fscache]
[4352864.056932] task: ffff88000adc8000 ti: ffff880131ea4000 task.ti: ffff880131ea4000
[4352864.057242] RIP: 0010:[<ffffffffa03f5269>] [<ffffffffa03f5269>] fscache_put_operation+0x1e9/0x220 [fscache]
[4352864.057536] RSP: 0018:ffff880131ea7de8 EFLAGS: 00010246
[4352864.057685] RAX: 0000000000000019 RBX: ffff8800636b8840 RCX: 0000000000000006
[4352864.057985] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff88181f4cf830
[4352864.058309] RBP: ffff880131ea7e00 R08: 0000000000000092 R09: 000000000000058f
[4352864.058643] R10: 00000000000000a0 R11: 0000000000000050 R12: ffff880823703a00
[4352864.059116] R13: ffff88015ba57c00 R14: ffff88181960de00 R15: 0000000000000600
[4352864.059415] FS: 0000000000000000(0000) GS:ffff88181f4c0000(0000) knlGS:0000000000000000
[4352864.059702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4352864.060583] CR2: 00000000004cf202 CR3: 00000000019ba000 CR4: 00000000001407e0
[4352864.060867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4352864.061174] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4352864.061461] Stack:
[4352864.061603] ffff8800636b8840 ffff880823703a00 ffff88015ba57c00 ffff880131ea7e18
[4352864.061911] ffffffffa03f54ca ffff8800636b8840 ffff880131ea7e60 ffffffff810a810b
[4352864.062231] 000000005ba57c20 0000000000000000 ffff88015ba57c20 ffff880823703a30
[4352864.062529] Call Trace:
[4352864.062679] [<ffffffffa03f54ca>] fscache_op_work_func+0x2a/0x50 [fscache]
[4352864.062843] [<ffffffff810a810b>] process_one_work+0x17b/0x470
[4352864.062992] [<ffffffff810a8f46>] worker_thread+0x126/0x410
[4352864.063162] [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
[4352864.063313] [<ffffffff810b06ff>] kthread+0xcf/0xe0
[4352864.063462] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[4352864.063614] [<ffffffff81696958>] ret_from_fork+0x58/0x90
[4352864.063763] [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[4352864.064069] Code: 31 c0 e8 14 a7 28 e1 48 c7 c7 80 a8 3f a0 31 c0 e8 06 a7 28 e1 8b 73 40 ba 05 00 00 00 48 c7 c7 78 9d 3f a0 31 c0 e8 f0 a6 28 e1 <0f> 0b 48 c7 c7 72 a8 3f a0 31 c0 e8 e0 a6 28 e1 48 c7 c7 80 a8
[4352864.064756] RIP [<ffffffffa03f5269>] fscache_put_operation+0x1e9/0x220 [fscache]
[4352864.065060] RSP <ffff880131ea7de8>
Environment
- Red Hat Enterprise Linux (RHEL) 7
- kernel prior to 3.10.0-514.16.1.el7
- fscache
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.