CPU stuck for xfrm_policy_flush in Red Hat Enterprise Linux 7

Solution Verified - Updated -

Issue

  • CPUs are being stuck. Kernel stacktrace shows XFRM related calls for the affected CPUs (xfrm_policy_flush). Node is not usable and must be reset or crashed. sosreport can't be taken in this state.
[276749.565205] NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [runc:[1:CHILD]:21984]
[276749.566221] Modules linked in: xt_multiport xt_set iptable_raw iptable_mangle ip_set_hash_ip ip_set_hash_net nfsv3 nfs_acl nfs lockd grace fscache veth ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv
6 nf_nat_ipv6 ip6_tables authenc esp4 ah4 xfrm4_mode_tunnel nf_conntrack_netlink br_netfilter bridge stp llc overlay(T) xt_statistic xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_recent ipt_REJECT n
f_reject_ipv4 xt_mark xt_comment xt_conntrack ip_set nfnetlink iptable_filter xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
 deflate drbg ansi_cprng twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 serpent_avx
2 serpent_avx_x86_64
[276749.571306]  serpent_sse2_x86_64 xts serpent_generic blowfish_generic blowfish_x86_64 blowfish_common cast5_avx_x86_64 cast5_generic cast_common des_generic cmac xcbc rmd160 sha512_ssse3 sha512_ge
neric crypto_null af_key vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr
 sg i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt sd_mod fb
_sys_fops ttm crc_t10dif crct10dif_generic ahci drm ata_piix libahci libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
[276749.577490] CPU: 14 PID: 21984 Comm: runc:[1:CHILD] Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.11.6.el7.x86_64 #1
[276749.578507] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
[276749.579507] task: ffff9259b58dbf40 ti: ffff925c79b38000 task.ti: ffff925c79b38000
[276749.580490] RIP: 0010:[<ffffffffbc75f269>]  [<ffffffffbc75f269>] __write_lock_failed+0x9/0x20
[276749.581476] RSP: 0018:ffff925c79b3bcd8  EFLAGS: 00000297
[276749.582441] RAX: ffff925c79b3bfd8 RBX: ffffffffffffff10 RCX: 0000000000000000
[276749.583373] RDX: ffff925c79b3bd5c RSI: 0000000000000001 RDI: ffff925e44f010bc
[276749.584291] RBP: ffff925c79b3bcd8 R08: 000000000001bb20 R09: ffffffffbca9f091
[276749.585193] R10: ffff9261bf69bb20 R11: ffffe1249c4aa000 R12: 0000000000000000
[276749.586079] R13: 0000000000000000 R14: 0000000000000002 R15: 0000000000000000
[276749.586956] FS:  00007ff793925740(0000) GS:ffff9261bf680000(0000) knlGS:0000000000000000
[276749.587834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[276749.588715] CR2: 00007ff792cfada1 CR3: 00000002f6a3a000 CR4: 00000000001607e0
[276749.589599] Call Trace:
[276749.590482]  [<ffffffffbcb1b44a>] _raw_write_lock_bh+0x2a/0x30
[276749.591375]  [<ffffffffbca92dea>] xfrm_policy_flush+0x3a/0x2f0
[276749.592272]  [<ffffffffbc6a1cfa>] ? __register_sysctl_table+0x5a/0x620
[276749.593152]  [<ffffffffbc5fd78d>] ? __kmalloc_track_caller+0x19d/0x240
[276749.594016]  [<ffffffffbca930f8>] xfrm_policy_fini+0x58/0x180
[276749.594894]  [<ffffffffbca934c0>] xfrm_net_init+0x240/0x370
[276749.595763]  [<ffffffffbc9e5914>] ops_init+0x44/0x150
[276749.596609]  [<ffffffffbc9e5ac3>] setup_net+0xa3/0x160
[276749.597453]  [<ffffffffbc9e6265>] copy_net_ns+0xb5/0x180
[276749.598302]  [<ffffffffbc4c2fb9>] create_new_namespaces+0xf9/0x180
[276749.599146]  [<ffffffffbc4c31fa>] unshare_nsproxy_namespaces+0x5a/0xc0
[276749.599990]  [<ffffffffbc493b13>] SyS_unshare+0x173/0x2e0
[276749.600835]  [<ffffffffbcb2579b>] system_call_fastpath+0x22/0x27
[276749.601687] Code: 00 00 e9 03 00 00 00 41 ff e7 e8 07 00 00 00 f3 90 0f ae e8 eb f9 4c 89 3c 24 c3 90 90 90 90 90 90 90 55 48 89 e5 f0 ff 07 f3 90 <83> 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 90 66 2e 0f 1f 84 00 00 00

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In