RHEL7.5: After updating to 3.10.0-862.el7 NFS client either sees CPU soft lockups during writeback on NFS or mmap system call sporadically returns -EBUSY

Solution Unverified - Updated -

Issue

  • NFS client soft lockup with in nfs_commit_inode
May 18 02:31:03 nfs-client kernel: NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [mysqlbackup:37825]
May 18 02:31:03 nfs-client kernel: Modules linked in: iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci shpchp binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx sd_mod crc_t10dif drm_kms_helper crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm serio_raw libata vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
May 18 02:31:03 nfs-client kernel: CPU: 3 PID: 37825 Comm: mysqlbackup Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
May 18 02:31:03 nfs-client kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
May 18 02:31:03 nfs-client kernel: task: ffff8c82a6e53f40 ti: ffff8c826ce00000 task.ti: ffff8c826ce00000
May 18 02:31:03 nfs-client kernel: RIP: 0010:[<ffffffffac915670>]  [<ffffffffac915670>] _raw_spin_lock+0x10/0x30
May 18 02:31:03 nfs-client kernel: RSP: 0018:ffff8c826ce03a58  EFLAGS: 00000246
May 18 02:31:03 nfs-client kernel: RAX: 0000000000000000 RBX: ffff8c826ce039f0 RCX: 0000000000000000
May 18 02:31:03 nfs-client kernel: RDX: 0000000000000001 RSI: ffff8c826ce03a88 RDI: ffff8c80d9cb3370
May 18 02:31:03 nfs-client kernel: RBP: ffff8c826ce03a78 R08: 000000000000002c R09: 0000000000000bdf
May 18 02:31:03 nfs-client kernel: R10: 0000000000000c0b R11: 0000000000000000 R12: ffff8c826ce03a10
May 18 02:31:03 nfs-client kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff8c838e6d8b00
May 18 02:31:03 nfs-client kernel: FS:  00007fc29effd700(0000) GS:ffff8c838e6c0000(0000) knlGS:0000000000000000
May 18 02:31:03 nfs-client kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 18 02:31:03 nfs-client kernel: CR2: 00007fc2780080e8 CR3: 0000000ad22f6000 CR4: 00000000001607e0
May 18 02:31:03 nfs-client kernel: Call Trace:
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e68f6>] ? nfs_scan_commit+0x26/0xc0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e6b47>] nfs_commit_inode+0x57/0x100 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e6ec2>] nfs_wb_single_page+0xb2/0x190 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e75c8>] nfs_updatepage+0x4d8/0x8e0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05d6531>] nfs_write_end+0x141/0x350 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffac3937a8>] generic_file_buffered_write+0x198/0x2c0
May 18 02:31:03 nfs-client kernel: [<ffffffffac395d12>] __generic_file_aio_write+0x1e2/0x400
May 18 02:31:03 nfs-client kernel: [<ffffffffac911c2f>] ? __mutex_lock_slowpath+0xcf/0x1d0
May 18 02:31:03 nfs-client kernel: [<ffffffffac395f89>] generic_file_aio_write+0x59/0xa0
May 18 02:31:03 nfs-client kernel: [<ffffffffc05d599b>] nfs_file_write+0xbb/0x1e0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffac41a1f3>] do_sync_write+0x93/0xe0
May 18 02:31:03 nfs-client kernel: [<ffffffffac41acd0>] vfs_write+0xc0/0x1f0
May 18 02:31:03 nfs-client kernel: [<ffffffffac41a484>] ? generic_file_llseek+0x24/0x30
May 18 02:31:03 nfs-client kernel: [<ffffffffac41baff>] SyS_write+0x7f/0xf0
May 18 02:31:03 nfs-client kernel: [<ffffffffac91f7d5>] system_call_fastpath+0x1c/0x21

Environment

  • Red Hat Enterprise Linux 7 (NFS client)
    • kernel 3.10.0-862.el7 and before kernel-3.10.0-862.2.3.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In