RHEL7: Kernel Panic in fscache_object_work_func or in fscache_put_operation with "fs/fscache/operation.c:494!".

Solution Verified - Updated -

Issue

The server crashes in fscache_object_work_func with Bad RIP value when under load with the following backtrace:

[94207.863822] Call Trace: 
[94207.863971] [<ffffffffa03e4695>] ? fscache_object_work_func+0xa5/0x200 [fscache] 
[94207.864262] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 
[94207.864411] [<ffffffff8109e4ac>] worker_thread+0x21c/0x400 
[94207.864560] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 
[94207.864711] [<ffffffff810a5acf>] kthread+0xcf/0xe0 
[94207.864860] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 
[94207.865012] [<ffffffff816460d8>] ret_from_fork+0x58/0x90 
[94207.865160] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 
[94207.865307] Code: Bad RIP value. 
[94207.865457] RIP [<0000000000000002>] 0x1 
[94207.865607] RSP <ffff8809b477bdf8> 
[94207.865750] CR2: 0000000000000002 
[94207.866339] --[ end trace 9498ae93c476f8c5 ]-- 

Another server crashed in fscache_put_operation with the following backtrace:

[4352864.053179] ------------[ cut here ]------------
[4352864.053327] kernel BUG at fs/fscache/operation.c:494!
[4352864.053486] invalid opcode: 0000 [#1] SMP 
[4352864.056175] CPU: 19 PID: 1562 Comm: kworker/u48:0 Not tainted 3.10.0-514.12.1.el7.x86_64 #1
[4352864.056464] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 02/17/2017
[4352864.056768] Workqueue: fscache_operation fscache_op_work_func [fscache]
[4352864.056932] task: ffff88000adc8000 ti: ffff880131ea4000 task.ti: ffff880131ea4000
[4352864.057242] RIP: 0010:[<ffffffffa03f5269>]  [<ffffffffa03f5269>] fscache_put_operation+0x1e9/0x220 [fscache]
[4352864.057536] RSP: 0018:ffff880131ea7de8  EFLAGS: 00010246
[4352864.057685] RAX: 0000000000000019 RBX: ffff8800636b8840 RCX: 0000000000000006
[4352864.057985] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff88181f4cf830
[4352864.058309] RBP: ffff880131ea7e00 R08: 0000000000000092 R09: 000000000000058f
[4352864.058643] R10: 00000000000000a0 R11: 0000000000000050 R12: ffff880823703a00
[4352864.059116] R13: ffff88015ba57c00 R14: ffff88181960de00 R15: 0000000000000600
[4352864.059415] FS:  0000000000000000(0000) GS:ffff88181f4c0000(0000) knlGS:0000000000000000
[4352864.059702] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4352864.060583] CR2: 00000000004cf202 CR3: 00000000019ba000 CR4: 00000000001407e0
[4352864.060867] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4352864.061174] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4352864.061461] Stack:
[4352864.061603]  ffff8800636b8840 ffff880823703a00 ffff88015ba57c00 ffff880131ea7e18
[4352864.061911]  ffffffffa03f54ca ffff8800636b8840 ffff880131ea7e60 ffffffff810a810b
[4352864.062231]  000000005ba57c20 0000000000000000 ffff88015ba57c20 ffff880823703a30
[4352864.062529] Call Trace:
[4352864.062679]  [<ffffffffa03f54ca>] fscache_op_work_func+0x2a/0x50 [fscache]
[4352864.062843]  [<ffffffff810a810b>] process_one_work+0x17b/0x470
[4352864.062992]  [<ffffffff810a8f46>] worker_thread+0x126/0x410
[4352864.063162]  [<ffffffff810a8e20>] ? rescuer_thread+0x460/0x460
[4352864.063313]  [<ffffffff810b06ff>] kthread+0xcf/0xe0
[4352864.063462]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[4352864.063614]  [<ffffffff81696958>] ret_from_fork+0x58/0x90
[4352864.063763]  [<ffffffff810b0630>] ? kthread_create_on_node+0x140/0x140
[4352864.064069] Code: 31 c0 e8 14 a7 28 e1 48 c7 c7 80 a8 3f a0 31 c0 e8 06 a7 28 e1 8b 73 40 ba 05 00 00 00 48 c7 c7 78 9d 3f a0 31 c0 e8 f0 a6 28 e1 <0f> 0b 48 c7 c7 72 a8 3f a0 31 c0 e8 e0 a6 28 e1 48 c7 c7 80 a8 
[4352864.064756] RIP  [<ffffffffa03f5269>] fscache_put_operation+0x1e9/0x220 [fscache]
[4352864.065060]  RSP <ffff880131ea7de8>

Environment

  • Red Hat Enterprise Linux (RHEL) 7
    • kernel prior to 3.10.0-514.16.1.el7
  • fscache

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.