Kernel crash at nfs_readpage_async+0x43 or nfs_updatepage+0x1b9 in Red Hat Enterprise Linux 7.5

Solution Verified - Updated -

Issue

  • Crash backtrace
crash> bt 
PID: 94660  TASK: ffffa0167fed5ee0  CPU: 1   COMMAND: "splunkd"
 #0 [ffffa00ddb6b3860] machine_kexec at ffffffffab0629da
 #1 [ffffa00ddb6b38c0] __crash_kexec at ffffffffab116692
 #2 [ffffa00ddb6b3990] crash_kexec at ffffffffab116780
 #3 [ffffa00ddb6b39a8] oops_end at ffffffffab71d728
 #4 [ffffa00ddb6b39d0] no_context at ffffffffab70c6cd
 #5 [ffffa00ddb6b3a20] __bad_area_nosemaphore at ffffffffab70c764
 #6 [ffffa00ddb6b3a70] bad_area at ffffffffab70ca74
 #7 [ffffa00ddb6b3a98] __do_page_fault at ffffffffab72088f
 #8 [ffffa00ddb6b3b00] do_page_fault at ffffffffab7208d5
 #9 [ffffa00ddb6b3b30] page_fault at ffffffffab71c758
    [exception RIP: nfs_readpage_async+67]
    RIP: ffffffffc07c2f13  RSP: ffffa00ddb6b3be0  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffffd79fc60ab280  RCX: 0000000000000000
    RDX: ffffd79fc60ab280  RSI: ffffa01606822ab8  RDI: ffffa010faf29780
    RBP: ffffa00ddb6b3cb8   R8: ffffa010faf29100   R9: 000000018020001b
    R10: 00000000faf29801  R11: ffffd79fdaebca40  R12: ffffa01606822ab8
    R13: 0000000000000000  R14: ffffa010faf29780  R15: ffffa0167fed5ee0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffffa00ddb6b3cc0] nfs_readpage at ffffffffc07c3206 [nfs]
#11 [ffffa00ddb6b3cf0] generic_file_aio_read at ffffffffab198960
#12 [ffffa00ddb6b3dc8] nfs_file_read at ffffffffc07b6371 [nfs]
#13 [ffffa00ddb6b3e00] do_sync_read at ffffffffab21e5c3
#14 [ffffa00ddb6b3ed8] vfs_read at ffffffffab21efef
#15 [ffffa00ddb6b3f08] sys_read at ffffffffab21febf
#16 [ffffa00ddb6b3f50] system_call_fastpath at ffffffffab72579b
    RIP: 00007f6dd18d922d  RSP: 00007f6dc49fe880  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 0000000000000100  RCX: ffffffffffffffff
    RDX: 0000000000000100  RSI: 00007f6dc49fe210  RDI: 0000000000000033
    RBP: 0000000000000000   R8: 0000000000000100   R9: 000000005bf71284
    R10: 00007f6dcfad50e0  R11: 0000000000000293  R12: 0000000000000100
    R13: 00007f6dc49fe210  R14: 00007f6dc49fe0d0  R15: 0000000000000100
    ORIG_RAX: 0000000000000000  CS: 0033  SS: 002b
crash> 

A back-trace from another crash which has different RIP but same cause of crash
as page mapping was NULL

PID: 133843  TASK: ffff90eee2abcf10  CPU: 1   COMMAND: "java"
 #0 [ffff90ee79e637f0] machine_kexec at ffffffff95e60f9a
 #1 [ffff90ee79e63850] __crash_kexec at ffffffff95f13732
 #2 [ffff90ee79e63920] crash_kexec at ffffffff95f13820
 #3 [ffff90ee79e63938] oops_end at ffffffff96518798
 #4 [ffff90ee79e63960] no_context at ffffffff96507fbe
 #5 [ffff90ee79e639b0] __bad_area_nosemaphore at ffffffff96508055
 #6 [ffff90ee79e63a00] bad_area at ffffffff965083e5
 #7 [ffff90ee79e63a28] __do_page_fault at ffffffff9651b8ff
 #8 [ffff90ee79e63a90] do_page_fault at ffffffff9651b945
 #9 [ffff90ee79e63ac0] page_fault at ffffffff96517788
    [exception RIP: nfs_updatepage+441]
    RIP: ffffffffc08ec309  RSP: ffff90ee79e63b70  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffffdb6f850fd200  RCX: ffff90ed3960aa80
    RDX: 0000000000000001  RSI: 0000000000000000  RDI: ffff90e9af8bf0d8
    RBP: ffff90ee79e63bc8   R8: 000000000001ba80   R9: 0000000000000000
    R10: ffff90ee35e22100  R11: ffffdb6f9944dd80  R12: ffff90ee35e22140
    R13: 0000000000000c71  R14: ffff90e9af8bf0d8  R15: ffff90ee35e22100
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff90ee79e63bd0] nfs_write_end at ffffffffc08db531 [nfs]
#11 [ffff90ee79e63c20] generic_file_buffered_write at ffffffff95f93ae8
#12 [ffff90ee79e63ce8] __generic_file_aio_write at ffffffff95f96052
#13 [ffff90ee79e63d68] generic_file_aio_write at ffffffff95f962c9
#14 [ffff90ee79e63da8] nfs_file_write at ffffffffc08da99b [nfs]
#15 [ffff90ee79e63df0] do_sync_write at ffffffff9601a533
#16 [ffff90ee79e63ec8] vfs_write at ffffffff9601b010
#17 [ffff90ee79e63f08] sys_write at ffffffff9601be3f
#18 [ffff90ee79e63f50] system_call_fastpath at ffffffff9652082f
  • another pattern of crash:
[25730.889956] BUG: unable to handle kernel NULL pointer dereference at           (null)
[25730.892166] IP: [<ffffffffc0ca0309>] nfs_updatepage+0x1b9/0x8e0 [nfs]
[25730.894326] PGD 800000084da1e067 PUD 84da1d067 PMD 0 
[25730.896345] Oops: 0000 [#1] SMP 
[25730.898328] Modules linked in: binfmt_misc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ext4 mbcache jbd2 loop iscsi_target_mod scsi_transport_iscsi target_core_mod scsi_transport_srp scsi_tgt sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass ipmi_ssif crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ipmi_si glue_helper ablk_helper iTCO_wdt iTCO_vendor_support cryptd pcspkr ipmi_devintf ipmi_msghandler dcdbas sg mei_me shpchp lpc_ich mei acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci crct10dif_pclmul crct10dif_common libahci crc32c_intel tg3 libata i2c_core megaraid_sas be2net ptp pps_core
[25730.910833]  dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ib_core]
[25730.912988] CPU: 0 PID: 15036 Comm: test_enospc-4 Kdump: loaded Tainted: G               ------------ T 3.10.0-862.11.6.el7.x86_64 #1
[25730.917083] Hardware name: Dell Inc. PowerEdge R430/0HFG24, BIOS 1.5.4 10/05/2015
[25730.919166] task: ffff9baed5f6bf40 ti: ffff9bad68aac000 task.ti: ffff9bad68aac000
[25730.921239] RIP: 0010:[<ffffffffc0ca0309>]  [<ffffffffc0ca0309>] nfs_updatepage+0x1b9/0x8e0 [nfs]
[25730.923331] RSP: 0018:ffff9bad68aafb68  EFLAGS: 00010246
[25730.925375] RAX: 0000000000000000 RBX: ffffdb691ada7980 RCX: ffff9bac50cfa800
[25730.927430] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9baedb7e0e98
[25730.929452] RBP: ffff9bad68aafbc0 R08: 000000000001bac0 R09: ffff9baecbfc8e80
[25730.931457] R10: 00001766f08df591 R11: 0000000000000006 R12: ffff9baecbfc8ec0
[25730.933442] R13: 0000000000000688 R14: ffff9baedb7e0e98 R15: ffff9baecbfc8e80
[25730.935413] FS:  00007f956eb58740(0000) GS:ffff9baedd200000(0000) knlGS:0000000000000000
[25730.937360] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[25730.939285] CR2: 0000000000000000 CR3: 0000000711e00000 CR4: 00000000001607f0
[25730.941195] Call Trace:
[25730.943077]  [<ffffffffc0c8f531>] nfs_write_end+0x141/0x350 [nfs]
[25730.944922]  [<ffffffff9b796b08>] generic_file_buffered_write+0x198/0x2c0
[25730.946760]  [<ffffffff9bd1e3e9>] kretprobe_trampoline_holder+0x9/0x9
[25730.948594]  [<ffffffff9bd1e3e9>] kretprobe_trampoline_holder+0x9/0x9
[25730.950380]  [<ffffffff9bd1e3e9>] kretprobe_trampoline_holder+0x9/0x9
[25730.952137]  [<ffffffff9bd1e3e9>] kretprobe_trampoline_holder+0x9/0x9
[25730.953818]  [<ffffffff9b81f180>] vfs_write+0xc0/0x1f0
[25730.955525]  [<ffffffff9b820172>] SyS_pwrite64+0x92/0xc0
[25730.957208]  [<ffffffff9bd25a1b>] tracesys+0xa3/0xc9
[25730.958827] Code: 1f 40 00 4d 85 ff 0f 84 49 03 00 00 49 81 ff 00 f0 ff ff 0f 87 13 06 00 00 e9 7b ed 79 ff 00 01 00 0f 85 f8 05 00 00 48 8b 43 08 <4c> 8b 20 49 8d 84 24 88 00 00 00 48 89 c7 48 89 45 c8 e8 90 b2 
[25730.962481] RIP  [<ffffffffc0ca0309>] nfs_updatepage+0x1b9/0x8e0 [nfs]
[25730.964183]  RSP <ffff9bad68aafb68>
[25730.965793] CR2: 0000000000000000

Environment

  • Red Hat Enterprise Linux 7
  • seen on 3.10.0-862.11.6.el7.x86_64 / 3.10.0-862.20.2.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In