Soft lockup in the function isolate_lru_page() or mem_cgroup_move_account() on Red Hat Enterprise Linux 7

Issue

• Soft lockup in the function isolate_lru_page() with the following call traces:

[12862481.434431] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [systemd:1]
[12862481.435310] Modules linked in: macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag bonding vfat fat skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev mei_me mei lpc_ich shpchp ipmi_ssif wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables ext4 mbcache jbd2 dm_queue_length sd_mod crc_t10dif crct10dif_generic dm_multipath mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common crc32c_intel ttm fnic ahci libfcoe libahci drm libfc libata scsi_transport_fc enic scsi_tgt i2c_core dm_mirror dm_region_hash dm_log dm_mod
[12862481.435355] CPU: 3 PID: 1 Comm: systemd Not tainted 3.10.0-862.11.6.el7.x86_64 #1
[12862481.435356] Hardware name: Cisco Systems Inc UCSB-B200-M5/UCSB-B200-M5, BIOS B200M5.3.2.3f.0.0523181559 05/23/2018
[12862481.435358] task: ffff9932ee928000 ti: ffff9932ee91c000 task.ti: ffff9932ee91c000
[12862481.435359] RIP: 0010:[<ffffffff81faa496>]  [<ffffffff81faa496>] isolate_lru_page+0x86/0x1c0
[12862481.435366] RSP: 0018:ffff9932ee91fcf0  EFLAGS: 00000246
[12862481.435367] RAX: 000000000001c144 RBX: ffff99337fdce000 RCX: ffffffffffffff83
[12862481.435368] RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff99f1fffd9540
[12862481.435369] RBP: ffff9932ee91fd20 R08: fffffffffffffff9 R09: 0000000000000001
[12862481.435370] R10: 0000000000000001 R11: 0000000000000001 R12: ffff99d383384000
[12862481.435371] R13: ffffe4cdfd311b80 R14: 000000000000000e R15: 00000000c2101ed1
[12862481.435373] FS:  00007f51b7039940(0000) GS:ffff99907dac0000(0000) knlGS:0000000000000000
[12862481.435374] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[12862481.435375] CR2: 00007f50c99ff9e0 CR3: 0000005ebebf6000 CR4: 00000000007607e0
[12862481.435376] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[12862481.435377] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[12862481.435378] PKRU: 55555554
[12862481.435379] Call Trace:
[12862481.435382]  [<ffffffff81fa5ffe>] ? lru_cache_add+0xe/0x10
[12862481.435386]  [<ffffffff8201138d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[12862481.435388]  [<ffffffff82011794>] mem_cgroup_css_offline+0x84/0x140
[12862481.435392]  [<ffffffff81f1f47a>] cgroup_destroy_locked+0xea/0x370
[12862481.435393]  [<ffffffff81f1f722>] cgroup_rmdir+0x22/0x40
[12862481.435397]  [<ffffffff8202da4c>] vfs_rmdir+0xdc/0x150
[12862481.435400]  [<ffffffff82030fd1>] do_rmdir+0x1f1/0x220
[12862481.435402]  [<ffffffff8202169e>] ? ____fput+0xe/0x10
[12862481.435406]  [<ffffffff81ebab90>] ? task_work_run+0xc0/0xe0
[12862481.435408]  [<ffffffff82032206>] SyS_rmdir+0x16/0x20
[12862481.435412]  [<ffffffff8252579b>] system_call_fastpath+0x22/0x27
[12862481.435413] Code: 89 e6 48 89 df e8 2b 68 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

• Soft lockup in the function mem_cgroup_move_account() with the following call traces:

[7182367.931337] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:1]
[7182367.931370] Modules linked in: btrfs raid6_pq xor vfat msdos fat nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc xfs xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter nf_nat br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ip_vs nf_conntrack libcrc32c fuse bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm sb_edac intel_powerclamp coretemp intel_rapl iTCO_wdt iosf_mbi kvm_intel iTCO_vendor_support kvm irqbypass crc32_pclmul
[7182367.931416]  ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ocrdma(T) ipmi_ssif ib_core joydev hpwdt hpilo sg ioatdma pcspkr lpc_ich i2c_i801 shpchp dca ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq wmi acpi_power_meter auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic uas usb_storage mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw hpsa be2net nvme i2c_core nvme_core scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod
[7182367.931452] CPU: 1 PID: 1 Comm: systemd Kdump: loaded Tainted: G               ------------ T 3.10.0-862.11.6.el7.x86_64 #1
[7182367.931455] task: ffff92900e550000 ti: ffff92900e54c000 task.ti: ffff92900e54c000
[7182367.931457] RIP: 0010:[<ffffffff8fe0fcbc>]  [<ffffffff8fe0fcbc>] mem_cgroup_move_account+0x3c/0x160
[7182367.931465] RSP: 0018:ffff92900e54fce0  EFLAGS: 00000246
[7182367.931467] RAX: ffff92d0cfbf0440 RBX: ffff930d3ee55bc0 RCX: ffff92ce3772bc00
[7182367.931468] RDX: ffff9290852f1ee0 RSI: 0000000000000001 RDI: fffff1182c5c7b80
[7182367.931469] RBP: ffff92900e54fd20 R08: ffff92d0cfbf0400 R09: 0000000000000001
[7182367.931470] R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
[7182367.931471] R13: 000000009f9e140f R14: ffff92900e54fc38 R15: ffff92900e54fc38
[7182367.931473] FS:  00007fd6d2171940(0000) GS:ffff930d3ee40000(0000) knlGS:0000000000000000
[7182367.931474] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[7182367.931475] CR2: 00007f4f4c6ef860 CR3: 0000007e7aa0a000 CR4: 00000000003607e0
[7182367.931477] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[7182367.931478] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[7182367.931479] Call Trace:
[7182367.931487]  [<ffffffff8fdaa532>] ? isolate_lru_page+0x122/0x1c0
[7182367.931489]  [<ffffffff8fda5ffe>] ? lru_cache_add+0xe/0x10
[7182367.931492]  [<ffffffff8fe11418>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[7182367.931494]  [<ffffffff8fe11794>] mem_cgroup_css_offline+0x84/0x140
[7182367.931499]  [<ffffffff8fd1f47a>] cgroup_destroy_locked+0xea/0x370
[7182367.931500]  [<ffffffff8fd1f722>] cgroup_rmdir+0x22/0x40
[7182367.931506]  [<ffffffff8fe2da4c>] vfs_rmdir+0xdc/0x150
[7182367.931508]  [<ffffffff8fe30fd1>] do_rmdir+0x1f1/0x220
[7182367.931511]  [<ffffffff8fe2169e>] ? ____fput+0xe/0x10
[7182367.931516]  [<ffffffff8fcbab90>] ? task_work_run+0xc0/0xe0
[7182367.931518]  [<ffffffff8fe32206>] SyS_rmdir+0x16/0x20
[7182367.931523]  [<ffffffff9032579b>] system_call_fastpath+0x22/0x27
[7182367.931524] Code: 55 49 89 cd 41 54 49 89 fc 53 48 89 d3 48 83 ec 18 83 fe 01 4c 8b 77 08 76 0c 48 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 <19> c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02