RHEL7.5: After updating to 3.10.0-862.el7 NFS client either sees CPU soft lockups during writeback on NFS or mmap system call sporadically returns -EBUSY

Solution Verified - Updated -

Issue

  • NFS client soft lockup with in nfs_commit_inode
May 18 02:31:03 nfs-client kernel: NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [mysqlbackup:37825]
May 18 02:31:03 nfs-client kernel: Modules linked in: iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci shpchp binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx sd_mod crc_t10dif drm_kms_helper crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm serio_raw libata vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
May 18 02:31:03 nfs-client kernel: CPU: 3 PID: 37825 Comm: mysqlbackup Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
May 18 02:31:03 nfs-client kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
May 18 02:31:03 nfs-client kernel: task: ffff8c82a6e53f40 ti: ffff8c826ce00000 task.ti: ffff8c826ce00000
May 18 02:31:03 nfs-client kernel: RIP: 0010:[<ffffffffac915670>]  [<ffffffffac915670>] _raw_spin_lock+0x10/0x30
May 18 02:31:03 nfs-client kernel: RSP: 0018:ffff8c826ce03a58  EFLAGS: 00000246
May 18 02:31:03 nfs-client kernel: RAX: 0000000000000000 RBX: ffff8c826ce039f0 RCX: 0000000000000000
May 18 02:31:03 nfs-client kernel: RDX: 0000000000000001 RSI: ffff8c826ce03a88 RDI: ffff8c80d9cb3370
May 18 02:31:03 nfs-client kernel: RBP: ffff8c826ce03a78 R08: 000000000000002c R09: 0000000000000bdf
May 18 02:31:03 nfs-client kernel: R10: 0000000000000c0b R11: 0000000000000000 R12: ffff8c826ce03a10
May 18 02:31:03 nfs-client kernel: R13: 0000000000000001 R14: 0000000000000001 R15: ffff8c838e6d8b00
May 18 02:31:03 nfs-client kernel: FS:  00007fc29effd700(0000) GS:ffff8c838e6c0000(0000) knlGS:0000000000000000
May 18 02:31:03 nfs-client kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 18 02:31:03 nfs-client kernel: CR2: 00007fc2780080e8 CR3: 0000000ad22f6000 CR4: 00000000001607e0
May 18 02:31:03 nfs-client kernel: Call Trace:
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e68f6>] ? nfs_scan_commit+0x26/0xc0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e6b47>] nfs_commit_inode+0x57/0x100 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e6ec2>] nfs_wb_single_page+0xb2/0x190 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05e75c8>] nfs_updatepage+0x4d8/0x8e0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffc05d6531>] nfs_write_end+0x141/0x350 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffac3937a8>] generic_file_buffered_write+0x198/0x2c0
May 18 02:31:03 nfs-client kernel: [<ffffffffac395d12>] __generic_file_aio_write+0x1e2/0x400
May 18 02:31:03 nfs-client kernel: [<ffffffffac911c2f>] ? __mutex_lock_slowpath+0xcf/0x1d0
May 18 02:31:03 nfs-client kernel: [<ffffffffac395f89>] generic_file_aio_write+0x59/0xa0
May 18 02:31:03 nfs-client kernel: [<ffffffffc05d599b>] nfs_file_write+0xbb/0x1e0 [nfs]
May 18 02:31:03 nfs-client kernel: [<ffffffffac41a1f3>] do_sync_write+0x93/0xe0
May 18 02:31:03 nfs-client kernel: [<ffffffffac41acd0>] vfs_write+0xc0/0x1f0
May 18 02:31:03 nfs-client kernel: [<ffffffffac41a484>] ? generic_file_llseek+0x24/0x30
May 18 02:31:03 nfs-client kernel: [<ffffffffac41baff>] SyS_write+0x7f/0xf0
May 18 02:31:03 nfs-client kernel: [<ffffffffac91f7d5>] system_call_fastpath+0x1c/0x21

Environment

  • Red Hat Enterprise Linux 7 (NFS client)
    • kernel 3.10.0-862.el7 and before kernel-3.10.0-862.2.3.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content