BUG: soft lockup - CPU#19 stuck for 22s! [systemd:1]

Environment

• Red Hat Enterprise Linux 7.0
• 3.10.0-123.el7.x86_64
• systemd-208-11.el7.x86_64
• libcgroup-0.41-8.el7.x86_64

Issue

• System crashes several minutes after executing systemctl stop/disable capture.service

[Unit]
Description=capture.service
After=syslog.target network-online.target

[Service]
Environment=GOTRACEBACK=crash
LimitCORE=infinity
WorkingDirectory=/tmp/
ExecStart=/usr/sbin/capture
Restart=on-failure
RestartSec=10
MemoryLimit=1G

[Install]
WantedBy=multi-user.target

• About 4 out of 48 online servers encountered the above problem, the crash message is as follows

[30297823.418752] BUG: soft lockup - CPU#19 stuck for 22s! [systemd:1]
[30297823.419347] Modules linked in: fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 sch_sfq sch_htb cls_u32 sch_ingress vhost_net macvtap macvlan tun ip6table_filter ip6_tables ip_vs_ftp ip_vs nf_nat_ftp nf_conntrack_ftp act_mirred ifb dfi(OF) xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT iptable_filter ip_tables bridge uio ebtable_filter ebtables 8021q garp stp mrp llc openvswitch(OF) vxlan ip_tunnel gre sg iTCO_wdt iTCO_vendor_support ipmi_devintf mxm_wmi dcdbas coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper igb ixgbe ablk_helper cryptd ptp pps_core mdio ipmi_si ipmi_msghandler dca mei_me pcspkr acpi_power_meter
[30297823.419385]  lpc_ich mei shpchp mfd_core wmi mperf nfsd auth_rpcgss nfs_acl lockd sunrpc binfmt_misc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_common usb_storage mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm drm ahci libahci i2c_core libata megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[30297823.419404] CPU: 19 PID: 1 Comm: systemd Tainted: GF          O--------------   3.10.0-123.el7.x86_64 #1
[30297823.419405] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5 08/16/2017
[30297823.419406] task: ffff885e6d168000 ti: ffff885e6d164000 task.ti: ffff885e6d164000
[30297823.419408] RIP: 0010:[<ffffffff81153960>]  [<ffffffff81153960>] isolate_lru_page+0x80/0x190
[30297823.419415] RSP: 0018:ffff885e6d165d18  EFLAGS: 00000282
[30297823.419416] RAX: 0000000000017924 RBX: ffff88607ffd7e80 RCX: ffffffffffffff83
[30297823.419417] RDX: 0000000000000041 RSI: 0000000000000004 RDI: ffff88607ffd7e80
[30297823.419418] RBP: ffff885e6d165d48 R08: 0000000000000042 R09: 0000000000000001
[30297823.419419] R10: 0000000000000001 R11: 0000000000001000 R12: ffffea00a3ae3b80
[30297823.419419] R13: 000000000000000e R14: 000000003877cfd5 R15: ffff885e6d165c60
[30297823.419421] FS:  00007fa6ab8f7880(0000) GS:ffff885efe060000(0000) knlGS:0000000000000000
[30297823.419422] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[30297823.419423] CR2: 00007fa6ab908000 CR3: 0000005e60548000 CR4: 00000000001427e0
[30297823.419423] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[30297823.419424] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[30297823.419425] Stack:
[30297823.419426]  ffffffff8114f92e ffffea01477a2680 ffff88018bfb3430 ffff88607ffd8300
[30297823.419430]  ffff885eac2689a0 ffffea01477a26a0 ffff885e6d165dd0 ffffffff811a4eb5
[30297823.419433]  ffff88018bfb4140 0000000200000000 ffff88018bfb4108 ffff88018bfb4040
[30297823.419436] Call Trace:
[30297823.419439]  [<ffffffff8114f92e>] ? lru_cache_add+0xe/0x10
[30297823.419442]  [<ffffffff811a4eb5>] mem_cgroup_reparent_charges+0x1c5/0x450
[30297823.419444]  [<ffffffff811a52bf>] mem_cgroup_css_offline+0x4f/0x100
[30297823.419448]  [<ffffffff810d5e27>] cgroup_destroy_locked+0xe7/0x360
[30297823.419450]  [<ffffffff810d60c2>] cgroup_rmdir+0x22/0x40
[30297823.419454]  [<ffffffff811bd738>] vfs_rmdir+0xa8/0x100
[30297823.419456]  [<ffffffff811bd935>] do_rmdir+0x1a5/0x200
[30297823.419461]  [<ffffffff811b12ae>] ? ____fput+0xe/0x10
[30297823.419466]  [<ffffffff8108228c>] ? task_work_run+0xac/0xe0
[30297823.419472]  [<ffffffff81012a17>] ? do_notify_resume+0x97/0xb0
[30297823.419473]  [<ffffffff811c0d76>] SyS_rmdir+0x16/0x20
[30297823.419477]  [<ffffffff815f1699>] system_call_fastpath+0x16/0x1b
[30297823.419478] Code: 48 89 de 4c 89 e7 e8 a0 0d 05 00 49 8b 14 24 49 89 c7 83 e2 20 75 54 0f 1f 44 00 00 66 83 83 80 04 00 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 e8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f
[30297823.419494] Kernel panic - not syncing: softlockup: hung tasks
[30297823.419949] CPU: 19 PID: 1 Comm: systemd Tainted: GF          O--------------   3.10.0-123.el7.x86_64 #1
[30297823.420374] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5 08/16/2017
[30297823.420805]  ffffffff817ffb12 000000003877cfd5 ffff885efe063e18 ffffffff815e0fb8
[30297823.421278]  ffff885efe063e98 ffffffff815dab77 0000000000000008 ffff885efe063ea8
[30297823.421721]  ffff885efe063e48 000000003877cfd5 ffff885efe063e67 0000000000000046
[30297823.422198] Call Trace:
[30297823.422581]  <IRQ>  [<ffffffff815e0fb8>] dump_stack+0x19/0x1b
[30297823.423095]  [<ffffffff815dab77>] panic+0xd8/0x1e7
[30297823.423665]  [<ffffffff810f60f5>] watchdog_timer_fn+0x165/0x170
[30297823.424170]  [<ffffffff81089967>] __run_hrtimer+0x77/0x1d0
[30297823.424634]  [<ffffffff810f5f90>] ? watchdog_cleanup+0x10/0x10
[30297823.425162]  [<ffffffff8108a1a7>] hrtimer_interrupt+0xf7/0x240
[30297823.425622]  [<ffffffff81039627>] local_apic_timer_interrupt+0x37/0x60
[30297823.426149]  [<ffffffff815f39af>] smp_apic_timer_interrupt+0x3f/0x60
[30297823.426563]  [<ffffffff815f231d>] apic_timer_interrupt+0x6d/0x80
[30297823.427096]  <EOI>  [<ffffffff81153960>] ? isolate_lru_page+0x80/0x190
[30297823.427622]  [<ffffffff81153a10>] ? isolate_lru_page+0x130/0x190
[30297823.428157]  [<ffffffff8114f92e>] ? lru_cache_add+0xe/0x10
[30297823.428710]  [<ffffffff811a4eb5>] mem_cgroup_reparent_charges+0x1c5/0x450
[30297823.429250]  [<ffffffff811a52bf>] mem_cgroup_css_offline+0x4f/0x100
[30297823.429761]  [<ffffffff810d5e27>] cgroup_destroy_locked+0xe7/0x360
[30297823.430304]  [<ffffffff810d60c2>] cgroup_rmdir+0x22/0x40
[30297823.430916]  [<ffffffff811bd738>] vfs_rmdir+0xa8/0x100
[30297823.431463]  [<ffffffff811bd935>] do_rmdir+0x1a5/0x200
[30297823.432063]  [<ffffffff811b12ae>] ? ____fput+0xe/0x10
[30297823.432609]  [<ffffffff8108228c>] ? task_work_run+0xac/0xe0
[30297823.433200]  [<ffffffff81012a17>] ? do_notify_resume+0x97/0xb0
[30297823.433790]  [<ffffffff811c0d76>] SyS_rmdir+0x16/0x20
[30297823.434361]  [<ffffffff815f1699>] system_call_fastpath+0x16/0x1b

• We suspect that it is the problem described in mm/memcg: Don't stuck in mem_cgroup_reparent_charges() forever, but we cannot reproduce the same problem in our test environment

• The attachment contains the vmcore-dmesgs of the crashed 4 servers.