Soft lockup or Hard lockup occurs due to race between ring_buffer_detach() and ring_buffer_wakeup().
Issue
- Soft lockup occurs with following message:
[11596.213003] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [capsule8-sensor:2144]
[11596.213404] Modules linked in: sunrpc iosf_mbi crc32_pclmul crc32c_intel ghash_clmulni_intel ppdev snd_intel8x0 aesni_intel snd_ac97_codec lrw gf128mul glue_helper ac97_bus ablk_helper snd_pcm cryptd snd_timer pcspkr sg snd i2c_piix4 virtio_net soundcore parport_pc parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix crct10dif_pclmul libata crct10dif_common serio_raw virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
[11596.213473] CPU: 0 PID: 2144 Comm: capsule8-sensor Kdump: loaded Tainted: G W ------------ 3.10.0-957.27.2.el7.x86_64 #1
[11596.213475] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[11596.213477] task: ffff90da39a2b0c0 ti: ffff90da3b0ec000 task.ti: ffff90da3b0ec000
[11596.213478] RIP: 0010:[<ffffffffa43124ca>] [<ffffffffa43124ca>] generic_exec_single+0xfa/0x1b0
[11596.213484] RSP: 0018:ffff90da3b0efcf0 EFLAGS: 00000202
[11596.213485] RAX: 0000000000000002 RBX: ffff90da3b0efcc0 RCX: 0000000000000830
[11596.213486] RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
[11596.213488] RBP: ffff90da3b0efd38 R08: ffffffffa4a133c0 R09: 0000000000000000
[11596.213489] R10: ffff90da3b8786f0 R11: ffff90da3952f110 R12: 00000000725e687e
[11596.213490] R13: 0000000000000000 R14: 00ffddf5812d9940 R15: ffffffffa43f87c4
[11596.213493] FS: 00007f0646fd7ae8(0000) GS:ffff90da3fc00000(0000) knlGS:0000000000000000
[11596.213495] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11596.213496] CR2: 00007f0646e0d408 CR3: 00000000618e8000 CR4: 00000000000606f0
[11596.213501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11596.213502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11596.213504] Call Trace:
[11596.213508] [<ffffffffa43a16d0>] ? perf_cgroup_attach+0x60/0x60
[11596.213511] [<ffffffffa43a16d0>] ? perf_cgroup_attach+0x60/0x60
[11596.213513] [<ffffffffa43125df>] smp_call_function_single+0x5f/0xa0
[11596.213516] [<ffffffffa43a08a3>] cpu_function_call+0x43/0x60
[11596.213518] [<ffffffffa439f950>] ? retprobe_show+0x30/0x30
[11596.213520] [<ffffffffa43a55b1>] event_function_call+0x101/0x110
[11596.213553] [<ffffffffa43a6530>] ? event_sched_out.isra.94+0x200/0x200
[11596.213555] [<ffffffffa43a57c5>] perf_remove_from_context+0x25/0x90
[11596.213557] [<ffffffffa43a9589>] perf_event_release_kernel+0xe9/0x300
[11596.213559] [<ffffffffa43a97b0>] perf_release+0x10/0x20
[11596.213562] [<ffffffffa4444a3c>] __fput+0xec/0x260
[11596.213564] [<ffffffffa4444c9e>] ____fput+0xe/0x10
[11596.213567] [<ffffffffa42bf9eb>] task_work_run+0xbb/0xe0
[11596.213570] [<ffffffffa422bc65>] do_notify_resume+0xa5/0xc0
[11596.213573] [<ffffffffa4977134>] int_signal+0x12/0x17
[11596.213575] Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 bf f4 a4 48 89 df e8 1a bd 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
- Hard lockup occurs with following messages:
[26404900.348486] NMI watchdog: Watchdog detected hard LOCKUP on cpu 23
[26404900.355406] Modules linked in:
[26404900.359311] dccp_diag dccp udp_diag unix_diag af_packet_diag netlink_diag iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 bridge stp llc tcp_diag inet_diag oracleasm(O) rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core ipmi_devintf lpc_ich ipmi_si shpchp wmi ipmi_msghandler acpi_power_meter dm_round_robin sg binfmt_misc dm_multipath nfsd auth_rpcgss nfs_acl
[26404900.439366] lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fnic(OE) fb_sys_fops libfcoe ttm libfc crct10dif_pclmul drm scsi_transport_fc crct10dif_common i2c_core crc32c_intel scsi_tgt megaraid_sas enic(OE) fjes dm_mirror dm_region_hash dm_log dm_mod
[26404900.473261] CPU: 23 PID: 12448 Comm: oracle_12448_cs Tainted: G OE ------------ 3.10.0-514.6.1.el7.x86_64 #1
[26404900.486191] Hardware name: Cisco Systems Inc UCSB-B200-M4/UCSB-B200-M4, BIOS B200M4.2.2.6c.0.111720151647 11/17/2015
[26404900.498634] 0000000000000000 000000000a38ce9c ffff885fbf445ba8 ffffffff816861ac
[26404900.507623] ffff885fbf445bc0 ffffffff8112f217 ffff8830e7af8400 ffff885fbf445bf8
[26404900.516608] ffffffff811747ae 0000000000000001 ffff885fbf44a280 ffff8830e7af8400
[26404900.525586] Call Trace:
[26404900.528795] <NMI> [<ffffffff816861ac>] dump_stack+0x19/0x1b
[26404900.535726] [<ffffffff8112f217>] watchdog_overflow_callback+0xd7/0x100
[26404900.543604] [<ffffffff811747ae>] __perf_event_overflow+0x8e/0x1f0
[26404900.551006] [<ffffffff811753f4>] perf_event_overflow+0x14/0x20
[26404900.558108] [<ffffffff81009d88>] intel_pmu_handle_irq+0x1f8/0x4e0
[26404900.565500] [<ffffffff81319eec>] ? ioremap_page_range+0x27c/0x3e0
[26404900.572896] [<ffffffff811bef44>] ? vunmap_page_range+0x1c4/0x310
[26404900.580189] [<ffffffff811bf0a1>] ? unmap_kernel_range_noflush+0x11/0x20
[26404900.588165] [<ffffffff813c9604>] ? ghes_copy_tofrom_phys+0x124/0x210
[26404900.595846] [<ffffffff813c9790>] ? ghes_read_estatus+0xa0/0x190
[26404900.603044] [<ffffffff8168ddeb>] perf_event_nmi_handler+0x2b/0x50
[26404900.614593] [<ffffffff8168f219>] nmi_handle.isra.0+0x69/0xb0
[26404900.621509] [<ffffffff8168f393>] do_nmi+0x133/0x410
[26404900.627539] [<ffffffff8168e653>] end_repeat_nmi+0x1e/0x2e
[26404900.634152] [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.640473] [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.646792] [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.653112] <<EOE>> <IRQ> [<ffffffff81173a85>] perf_event_wakeup+0x65/0xe0
[26404900.661785] [<ffffffff81173bd7>] perf_pending_event+0xd7/0x110
[26404900.668884] [<ffffffff8116ae4f>] irq_work_run_list+0x4f/0x70
[26404900.675789] [<ffffffff8116ae89>] irq_work_run+0x19/0x40
[26404900.682208] [<ffffffff8102f645>] smp_trace_irq_work_interrupt+0x35/0xb0
[26404900.690185] [<ffffffff81697f9d>] trace_irq_work_interrupt+0x6d/0x80
[26404900.697767] <EOI>
Environment
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.