System crashed in context of svc_recv( ) function in nfsd in Red Hat Enterprise Linux 7

Solution In Progress - Updated -

Issue

  • Crashed in context of nfsd with RIP svc_recv+0x30e
crash> bt 
PID: 1235   TASK: ffff8815e5d39fa0  CPU: 7   COMMAND: "nfsd"
 #0 [ffff8815e55d3bc0] machine_kexec at ffffffff8105c58b
 #1 [ffff8815e55d3c20] __crash_kexec at ffffffff81106742
 #2 [ffff8815e55d3cf0] crash_kexec at ffffffff81106830
 #3 [ffff8815e55d3d08] oops_end at ffffffff816b0aa8
 #4 [ffff8815e55d3d30] die at ffffffff8102e87b
 #5 [ffff8815e55d3d60] do_general_protection at ffffffff816b042e
 #6 [ffff8815e55d3d90] general_protection at ffffffff816af898
    [exception RIP: svc_recv+782]
    RIP: ffffffffc031c26e  RSP: ffff8815e55d3e40  RFLAGS: 00010203
    RAX: ffff8815e55d3fd8  RBX: 0032003400340032  RCX: ffff8815e55d3fd8
    RDX: ffff8815e55d3fd8  RSI: 0000000000000200  RDI: ffffffffc031c7ef
    RBP: ffff8815e55d3e98   R8: ffff8815e55d0000   R9: 0000000000000013
    R10: 0000000000000000  R11: 7fffffffffffffff  R12: ffff8815e5254030
    R13: ffff8815d2a8fb80  R14: 0000000000000000  R15: ffff8815e5254a60
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018

 #7 [ffff8815e55d3ea0] nfsd at ffffffffc037eeed [nfsd]
 #8 [ffff8815e55d3ec8] kthread at ffffffff810b252f
 #9 [ffff8815e55d3f50] ret_from_fork at ffffffff816b8798
crash> 

Slab corruption by cifsd


[12980025.523693] Call Trace: [12980025.523748] [<ffffffff816a5ea1>] dump_stack+0x19/0x1b [12980025.523778] [<ffffffff811dd67c>] print_trailer+0x14c/0x210 [12980025.523783] [<ffffffff811dd87f>] check_bytes_and_report+0xcf/0x110 [12980025.523788] [<ffffffff811de503>] check_object+0x193/0x250 [12980025.523793] [<ffffffff816a3011>] free_debug_processing+0xcc/0x259 [12980025.523810] [<ffffffff81187177>] ? mempool_free_slab+0x17/0x20 [12980025.523918] [<ffffffffc0528626>] ? smb2_open_op_close+0xd6/0x250 [cifs] [12980025.523934] [<ffffffffc0528626>] ? smb2_open_op_close+0xd6/0x250 [cifs] [12980025.523938] [<ffffffff811e06a0>] __slab_free+0x250/0x2f0 [12980025.523953] [<ffffffffc0528626>] ? smb2_open_op_close+0xd6/0x250 [cifs] [12980025.523957] [<ffffffff811e0e03>] kfree+0x103/0x140 << [12980025.523972] [<ffffffffc0528626>] smb2_open_op_close+0xd6/0x250 [cifs] [12980025.523977] [<ffffffff8118930e>] ? __get_free_pages+0xe/0x40 [12980025.523982] [<ffffffff811de9fe>] ? kmalloc_order_trace+0x2e/0xa0 [12980025.523997] [<ffffffffc05288a5>] smb2_query_path_info+0x85/0x190 [cifs] [12980025.524010] [<ffffffffc0507102>] cifs_get_inode_info+0x3a2/0x990 [cifs] [12980025.524022] [<ffffffffc04f9bc6>] ? build_path_from_dentry+0xe6/0x410 [cifs] [12980025.524026] [<ffffffff811e2538>] ? __kmalloc+0x1b8/0x230 [12980025.524038] [<ffffffffc04f9bc6>] ? build_path_from_dentry+0xe6/0x410 [cifs] [12980025.524050] [<ffffffffc04f9c1f>] ? build_path_from_dentry+0x13f/0x410 [cifs] [12980025.524062] [<ffffffffc04fb113>] cifs_lookup+0x1a3/0x530 [cifs] < [12980025.524082] [<ffffffff8120d32d>] lookup_real+0x1d/0x50 [12980025.524086] [<ffffffff8120dc02>] __lookup_hash+0x42/0x60 [12980025.524091] [<ffffffff816a3432>] lookup_slow+0x42/0xa7 [12980025.524096] [<ffffffff812111ab>] path_lookupat+0x77b/0x7b0 [12980025.524100] [<ffffffff8120d2b7>] ? path_get+0x27/0x30 [12980025.524105] [<ffffffff8121120b>] filename_lookup+0x2b/0xc0 [12980025.524110] [<ffffffff81214d87>] user_path_at_empty+0x67/0xc0 [12980025.524114] [<ffffffff81214df1>] user_path_at+0x11/0x20 [12980025.524120] [<ffffffff81208223>] vfs_fstatat+0x63/0xc0 [12980025.524127] [<ffffffff8120878e>] SYSC_newstat+0x2e/0x60 [12980025.524150] [<ffffffff816b895a>] ? system_call_after_swapgs+0x187/0x214 [12980025.524157] [<ffffffff816b8953>] ? system_call_after_swapgs+0x180/0x214 [12980025.524162] [<ffffffff816b894c>] ? system_call_after_swapgs+0x179/0x214 [12980025.524171] [<ffffffff81208a6e>] SyS_newstat+0xe/0x10 [12980025.524176] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b [12980025.524180] [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214 [12980025.524185] FIX kmalloc-8192: Restoring 0xffff880449e62000-0xffff880449e62007=0xcc [12980039.012815] ============================================================================= [12980039.015043] BUG kmalloc-8192 (Tainted: P B W OE ------------ ): Poison overwritten [12980039.015904] ----------------------------------------------------------------------------- [12980039.017454] INFO: 0xffff880449e62018-0xffff880449e62059. First byte 0x68 instead of 0x6b [12980039.018269] INFO: Slab 0xffffea0011279800 objects=3 used=3 fp=0x (null) flags=0x2fffff00004080 << [12980039.019091] INFO: Object 0xffff880449e62018 @offset=8216 fp=0xffff880449e60000 [12980039.020739] Bytes b4 ffff880449e62008: 00 00 00 00 00 00 00 00 2d 00 52 00 69 00 63 00 ........-.R.i.c. [12980039.021605] Object ffff880449e62018: 68 00 65 00 6c 00 69 00 65 00 75 00 2c 00 32 00 h.e.l.i.e.u.,.2. [12980039.022531] Object ffff880449e62028: 34 00 35 00 37 00 2c 00 43 00 44 00 2c 00 31 00 4.5.7.,.C.D.,.1. [12980039.023416] Object ffff880449e62038: 32 00 34 00 34 00 32 00 30 00 2c 00 4d 00 52 00 2.4.4.2.0.,.M.R. [12980039.024301] Object ffff880449e62048: 43 00 2c 00 2c 00 2c 00 32 00 34 00 0a 00 44 00 C.,.,.,.2.4...D. [12980039.025208] Object ffff880449e62058: 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ..kkkkkkkkkkkkkk [12980039.026146] Object ffff880449e62068: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [12980039.027035] Object ffff880449e62078: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk .. < downsized output>

Environment

  • RHEL 7.4
    • 3.10.0-693.11.6.el7.x86_64
  • cifs and nfs nested mounts

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content