Server crashed with RIP: native_queued_spin_lock_slowpath+272

Issue

• Server crashed with RIP: native_queued_spin_lock_slowpath+272.

crash> log
[...]
[2221616.681792] BUG: unable to handle kernel paging request at 0000000000019600
[2221616.682227] IP: [<ffffffff810fdfa0>] native_queued_spin_lock_slowpath+0x110/0x200
[2221616.682565] PGD 0 
[2221616.682816] Oops: 0002 [#1] SMP 
[2221616.683060] Modules linked in: binfmt_misc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_filter xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bonding emcpdm(POE) emcpgpx(POE) emcpmpx(POE) emcp(POE) vfat fat iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether usbnet mii mei_me mei lpc_ich acpi_power_meter sg shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod mgag200 i2c_algo_bit drm_kms_helper lpfc syscopyarea sysfillrect sysimgblt fb_sys_fops
[2221616.685213]  ttm drm crc32c_intel ahci i40e libahci crc_t10dif crct10dif_generic libata crct10dif_pclmul scsi_transport_fc megaraid_sas i2c_core scsi_tgt ptp pps_core crct10dif_common dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[2221616.686573] CPU: 39 PID: 101271 Comm: systemd-cgtop Tainted: P           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
[2221616.687011] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
[2221616.687509] task: ffff88158c812f70 ti: ffff88a4b502c000 task.ti: ffff88a4b502c000
[2221616.687982] RIP: 0010:[<ffffffff810fdfa0>]  [<ffffffff810fdfa0>] native_queued_spin_lock_slowpath+0x110/0x200
[2221616.688519] RSP: 0018:ffff88a4b502fdc8  EFLAGS: 00010006
[2221616.689011] RAX: 0000000000001fd4 RBX: ffffffff81abb3a0 RCX: 0000000001390000
[2221616.689520] RDX: 0000000000019600 RSI: 00000000fea18bd0 RDI: ffff88b68e104f98
[2221616.690044] RBP: ffff88a4b502fdc8 R08: ffff88befe2d9600 R09: 0000000000000000
[2221616.690618] R10: ffff8801bfc07300 R11: 0000000000000246 R12: 0000000000000000
[2221616.691175] R13: ffff88748b81be00 R14: ffffffff81318e90 R15: ffff88bed0d860c0
[2221616.691743] FS:  00007efd99b72740(0000) GS:ffff88befe2c0000(0000) knlGS:0000000000000000
[2221616.692299] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2221616.692870] CR2: 0000000000019600 CR3: 0000009284026000 CR4: 00000000003607e0
[2221616.693498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2221616.694113] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2221616.694719] Call Trace:
[2221616.695352]  [<ffffffff816a8ff4>] queued_spin_lock_slowpath+0xb/0xf
[2221616.695956]  [<ffffffff816b6758>] _raw_spin_lock_irq+0x28/0x30
[2221616.696615]  [<ffffffff81319654>] blkcg_print_blkgs+0x74/0xe0
[2221616.697260]  [<ffffffff8131f286>] cfqg_print_rwstat+0x36/0x40
[2221616.697899]  [<ffffffff8110cfd6>] cgroup_seqfile_show+0x76/0x80
[2221616.698527]  [<ffffffff8122ac87>] ? seq_buf_alloc+0x17/0x40
[2221616.699185]  [<ffffffff8122b1b0>] seq_read+0x110/0x3e0
[2221616.699849]  [<ffffffff812054ef>] vfs_read+0x9f/0x170
[2221616.700539]  [<ffffffff812063bf>] SyS_read+0x7f/0xe0
[2221616.701184]  [<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
[2221616.701834] Code: 87 47 02 c1 e0 10 45 31 c9 85 c0 74 44 48 89 c2 c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 96 01 00 48 03 14 c5 a0 10 b2 81 <4c> 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 
[2221616.703308] RIP  [<ffffffff810fdfa0>] native_queued_spin_lock_slowpath+0x110/0x200
[2221616.704014]  RSP <ffff88a4b502fdc8>
[2221616.704702] CR2: 0000000000019600

• A hard lockup is detected at blkcg_print_blkgs+0x76.

[114915.664676] NMI watchdog: Watchdog detected hard LOCKUP on cpu 3
[114915.664713] Modules linked in:
[114915.665104]  ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache 8021q garp mrp stp llc bonding vfat fat iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl iosf_mbi dcdbas kvm irqbypass ipmi_ssif crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_i801 lpc_ich shpchp wmi ipmi_si ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops qede(OE) ttm ptp qedf(OE) libfcoe pps_core ahci qed(OE) qla2xxx(OE) libfc libahci crc8 drm libata i2c_core fjes emcpdm(POE) emcpgpx(POE) emcpmpx(POE) emcp(POE)
[114915.665159]  scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: emcpioc]
[114915.665166] CPU: 3 PID: 434357 Comm: sosreport Tainted: P           OE  ------------   3.10.0-514.41.1.el7.x86_64 #1
[114915.665168] Hardware name: Dell Inc. PowerEdge R640/08HT8T, BIOS 2.2.10 05/15/2019
[114915.665170] Call Trace:
[114915.665172]  <NMI>  [<ffffffff8168bb1f>] dump_stack+0x19/0x1b
[114915.665185]  [<ffffffff81132978>] watchdog_overflow_callback+0xe8/0x100
[114915.665189]  [<ffffffff81177e4e>] __perf_event_overflow+0x8e/0x1f0
[114915.665192]  [<ffffffff81178a94>] perf_event_overflow+0x14/0x20
[114915.665197]  [<ffffffff81009d98>] intel_pmu_handle_irq+0x1f8/0x4e0
[114915.665202]  [<ffffffff8131ed9c>] ? ioremap_page_range+0x27c/0x3e0
[114915.665207]  [<ffffffff811c2984>] ? vunmap_page_range+0x1c4/0x310
[114915.665211]  [<ffffffff813ce834>] ? ghes_copy_tofrom_phys+0x124/0x210
[114915.665214]  [<ffffffff813ce9c0>] ? ghes_read_estatus+0xa0/0x190
[114915.665219]  [<ffffffff8169402b>] perf_event_nmi_handler+0x2b/0x50
[114915.665223]  [<ffffffff81695ce7>] nmi_handle.isra.0+0x87/0x160
[114915.665226]  [<ffffffff81695f93>] do_nmi+0x1d3/0x410
[114915.665229]  [<ffffffff816950b9>] end_repeat_nmi+0x1e/0x7e
[114915.665234]  [<ffffffff81313430>] ? cfqg_print_leaf_weight_device+0x40/0x40
[114915.665237]  [<ffffffff8169353a>] ? _raw_spin_lock_irq+0x3a/0x60
[114915.665240]  [<ffffffff8169353a>] ? _raw_spin_lock_irq+0x3a/0x60
[114915.665243]  [<ffffffff8169353a>] ? _raw_spin_lock_irq+0x3a/0x60
[114915.665244]  <<EOE>>  [<ffffffff8130d396>] blkcg_print_blkgs+0x76/0xf0
[114915.665250]  [<ffffffff813132e6>] cfqg_print_rwstat_recursive+0x36/0x40
[114915.665254]  [<ffffffff8110c393>] cgroup_seqfile_show+0x73/0x80
[114915.665258]  [<ffffffff81225c37>] ? seq_buf_alloc+0x17/0x40
[114915.665261]  [<ffffffff8122613a>] seq_read+0x10a/0x3b0
[114915.665265]  [<ffffffff812019de>] vfs_read+0x9e/0x170
[114915.665268]  [<ffffffff812025af>] SyS_read+0x7f/0xe0
[114915.665271]  [<ffffffff8169d53d>] system_call_fastpath+0x16/0x1b