RHEL7.1: kernel crashes with nfsd in __d_lookup called from nfsd_readdir - BUG: unable to handle kernel paging request at 0000000001000018

Solution In Progress - Updated -

Issue

  • kernel crashes with the following oops
[2905015.894860] BUG: unable to handle kernel paging request at 0000000001000018
[2905015.895023] IP: [<ffffffff811dfa08>] __d_lookup+0x68/0x160
[2905015.895141] PGD 0 
[2905015.895188] Oops: 0000 [#1] SMP 
[2905015.895251] Modules linked in: xfs libcrc32c iTCO_wdt iTCO_vendor_support lpc_ich mfd_core coretemp kvm serio_raw pcspkr i2c_i801 i5000_edac edac_core ioatdma dca i5k_amb acpi_cpufreq shpchp ipmi_si ipmi_msghandler nfsd auth_rpcgss nfs_acl lockd sunrpc btrfs xor zlib_deflate raid6_pq sd_mod crc_t10dif crct10dif_common sr_mod cdrom ata_generic pata_acpi radeon i2c_algo_bit ata_piix drm_kms_helper libata ttm drm 3w_9xxx i2c_core e1000e ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
[2905015.895822] CPU: 3 PID: 2567 Comm: nfsd Not tainted 3.10.0-229.4.2.el7.x86_64 #1
[2905015.895822] Hardware name: Intel S5000PAL/S5000PAL0, BIOS S5000.86B.12.00.0098.062320091136 06/23/2009
[2905015.895822] task: ffff88008c00ad80 ti: ffff88014c614000 task.ti: ffff88014c614000
[2905015.895822] RIP: 0010:[<ffffffff811dfa08>]  [<ffffffff811dfa08>] __d_lookup+0x68/0x160
[2905015.895822] RSP: 0018:ffff88014c617b58  EFLAGS: 00010206
[2905015.895822] RAX: ffffc90000290d78 RBX: 0000000001000000 RCX: 0000000000000013
[2905015.895822] RDX: ffffc90000002000 RSI: ffff88014c617c40 RDI: ffff880125daac00
[2905015.895822] RBP: ffff88014c617b98 R08: 0000000000000000 R09: ffff880106c6584c
[2905015.895822] R10: ffff880102376000 R11: ffff88010237647c R12: ffff880125daac00
[2905015.895822] R13: ffff88014c617c40 R14: 0000000076cda393 R15: 0000000000000000
[2905015.895822] FS:  0000000000000000(0000) GS:ffff88015fcc0000(0000) knlGS:0000000000000000
[2905015.895822] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[2905015.895822] CR2: 0000000001000018 CR3: 000000008c32b000 CR4: 00000000000407e0
[2905015.895822] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2905015.895822] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2905015.895822] Stack:
[2905015.895822]  ffff88014c617be8 ffff880124e77140 ffffffff0000001b 00000000000e2e4a
[2905015.895822]  ffff88014c617c40 ffff880125daac00 ffff88014c617c17 0000000000000000
[2905015.895822]  ffff88014c617bc0 ffffffff811dfb2a 0000000000000000 ffff880125daac00
[2905015.895822] Call Trace:
[2905015.895822]  [<ffffffff811dfb2a>] d_lookup+0x2a/0x50
[2905015.895822]  [<ffffffff811d0da0>] lookup_dcache+0x30/0xb0
[2905015.895822]  [<ffffffff811d0e4d>] __lookup_hash+0x2d/0x60
[2905015.895822]  [<ffffffff811d1d7e>] lookup_one_len+0xee/0x140
[2905015.895822]  [<ffffffffa0481a52>] encode_entryplus_baggage+0xb2/0x1f0 [nfsd]
[2905015.895822]  [<ffffffffa0481eb2>] encode_entry.isra.12+0x322/0x370 [nfsd]
[2905015.895822]  [<ffffffffa0483390>] ? nfs3svc_encode_entry+0x20/0x20 [nfsd]
[2905015.895822]  [<ffffffffa04833a4>] nfs3svc_encode_entry_plus+0x14/0x20 [nfsd]
[2905015.895822]  [<ffffffffa0478527>] nfsd_readdir+0x187/0x270 [nfsd]
[2905015.895822]  [<ffffffffa047faed>] nfsd3_proc_readdirplus+0x12d/0x280 [nfsd]
[2905015.895822]  [<ffffffffa0471e1b>] nfsd_dispatch+0xbb/0x200 [nfsd]
[2905015.895822]  [<ffffffffa0564b33>] svc_process_common+0x453/0x6f0 [sunrpc]
[2905015.895822]  [<ffffffffa0564ed3>] svc_process+0x103/0x170 [sunrpc]
[2905015.895822]  [<ffffffffa04717a7>] nfsd+0xe7/0x150 [nfsd]
[2905015.895822]  [<ffffffffa04716c0>] ? nfsd_destroy+0x80/0x80 [nfsd]
[2905015.895822]  [<ffffffff8109726f>] kthread+0xcf/0xe0
[2905015.895822]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[2905015.895822]  [<ffffffff816140bc>] ret_from_fork+0x7c/0xb0
[2905015.895822]  [<ffffffff810971a0>] ? kthread_create_on_node+0x140/0x140
[2905015.895822] Code: 89 c2 d3 ea 01 d0 23 05 ab 8e 84 00 48 8b 15 98 8e 84 00 48 8d 04 c2 48 8b 18 48 83 e3 fe 75 0b eb 31 90 48 8b 1b 48 85 db 74 28 <44> 39 73 18 75 f2 4c 8d 7b 50 4c 89 ff e8 96 ba 42 00 4c 39 63 
[2905015.895822] RIP  [<ffffffff811dfa08>] __d_lookup+0x68/0x160
[2905015.895822]  RSP <ffff88014c617b58>
[2905015.895822] CR2: 0000000001000018

Another crash reported on kernel 3.10.0-514.16.1.el7.x86_64

crash> bt
PID: 11931  TASK: ffff8800aca0edd0  CPU: 0   COMMAND: "sh"
 #0 [ffff8800024078d0] machine_kexec at ffffffff81059bdb
 #1 [ffff880002407930] __crash_kexec at ffffffff811057c2
<.... >
 #9 [ffff880002407b88] do_async_page_fault at ffffffff81691fcb
#10 [ffff880002407ba0] async_page_fault at ffffffff8168eab8
    [exception RIP: __d_lookup+0x68]
    RIP: ffffffff812183f8  RSP: ffff880002407c50  RFLAGS: 00010206
    RAX: ffffc90000421198  RBX: 0000000002c00a00  RCX: 0000000000000014
    RDX: ffffc90000000000  RSI: ffff880002407e60  RDI: ffff880035ba00c0
    RBP: ffff880002407c90   R8: ffff880002407de4   R9: ffff8800689e0000
    R10: fefefefefefefeff  R11: 2f2f2f2f2f2f2f2f  R12: ffff880035ba00c0
    R13: ffff880002407e60  R14: 00000000aca14f59  R15: 0000000000000301
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#11 [ffff880002407c98] d_lookup at ffffffff8121851a
#12 [ffff880002407cc0] lookup_dcache at ffffffff81209310
#13 [ffff880002407d00] do_last at ffffffff8120d249
#14 [ffff880002407db0] path_openat at ffffffff8120e2f2
#15 [ffff880002407e48] do_filp_open at ffffffff8121046b
#16 [ffff880002407f18] do_sys_open at ffffffff811fd973
#17 [ffff880002407f70] sys_open at ffffffff811fda8e
#18 [ffff880002407f80] system_call_fastpath at ffffffff81697089

Environment

  • Red Hat Enterprise Linux 7.1 (NFS server)
    • seen on kernel-3.10.0-229.4.2.el7
    • also reported on 3.10.0-514.16.1.el7.x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content