Soft lockup or Hard lockup occurs due to race between ring_buffer_detach() and ring_buffer_wakeup().

Solution Verified - Updated -

Issue

  • Soft lockup occurs with following message:
[11596.213003] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [capsule8-sensor:2144]
[11596.213404] Modules linked in: sunrpc iosf_mbi crc32_pclmul crc32c_intel ghash_clmulni_intel ppdev snd_intel8x0 aesni_intel snd_ac97_codec lrw gf128mul glue_helper ac97_bus ablk_helper snd_pcm cryptd snd_timer pcspkr sg snd i2c_piix4 virtio_net soundcore parport_pc parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix crct10dif_pclmul libata crct10dif_common serio_raw virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
[11596.213473] CPU: 0 PID: 2144 Comm: capsule8-sensor Kdump: loaded Tainted: G        W      ------------   3.10.0-957.27.2.el7.x86_64 #1
[11596.213475] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[11596.213477] task: ffff90da39a2b0c0 ti: ffff90da3b0ec000 task.ti: ffff90da3b0ec000
[11596.213478] RIP: 0010:[<ffffffffa43124ca>]  [<ffffffffa43124ca>] generic_exec_single+0xfa/0x1b0
[11596.213484] RSP: 0018:ffff90da3b0efcf0  EFLAGS: 00000202
[11596.213485] RAX: 0000000000000002 RBX: ffff90da3b0efcc0 RCX: 0000000000000830
[11596.213486] RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
[11596.213488] RBP: ffff90da3b0efd38 R08: ffffffffa4a133c0 R09: 0000000000000000
[11596.213489] R10: ffff90da3b8786f0 R11: ffff90da3952f110 R12: 00000000725e687e
[11596.213490] R13: 0000000000000000 R14: 00ffddf5812d9940 R15: ffffffffa43f87c4
[11596.213493] FS:  00007f0646fd7ae8(0000) GS:ffff90da3fc00000(0000) knlGS:0000000000000000
[11596.213495] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11596.213496] CR2: 00007f0646e0d408 CR3: 00000000618e8000 CR4: 00000000000606f0
[11596.213501] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11596.213502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11596.213504] Call Trace:
[11596.213508]  [<ffffffffa43a16d0>] ? perf_cgroup_attach+0x60/0x60
[11596.213511]  [<ffffffffa43a16d0>] ? perf_cgroup_attach+0x60/0x60
[11596.213513]  [<ffffffffa43125df>] smp_call_function_single+0x5f/0xa0
[11596.213516]  [<ffffffffa43a08a3>] cpu_function_call+0x43/0x60
[11596.213518]  [<ffffffffa439f950>] ? retprobe_show+0x30/0x30
[11596.213520]  [<ffffffffa43a55b1>] event_function_call+0x101/0x110
[11596.213553]  [<ffffffffa43a6530>] ? event_sched_out.isra.94+0x200/0x200
[11596.213555]  [<ffffffffa43a57c5>] perf_remove_from_context+0x25/0x90
[11596.213557]  [<ffffffffa43a9589>] perf_event_release_kernel+0xe9/0x300
[11596.213559]  [<ffffffffa43a97b0>] perf_release+0x10/0x20
[11596.213562]  [<ffffffffa4444a3c>] __fput+0xec/0x260
[11596.213564]  [<ffffffffa4444c9e>] ____fput+0xe/0x10
[11596.213567]  [<ffffffffa42bf9eb>] task_work_run+0xbb/0xe0
[11596.213570]  [<ffffffffa422bc65>] do_notify_resume+0xa5/0xc0
[11596.213573]  [<ffffffffa4977134>] int_signal+0x12/0x17
[11596.213575] Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 bf f4 a4 48 89 df e8 1a bd 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
  • Hard lockup occurs with following messages:
[26404900.348486] NMI watchdog: Watchdog detected hard LOCKUP on cpu 23
[26404900.355406] Modules linked in:
[26404900.359311]  dccp_diag dccp udp_diag unix_diag af_packet_diag netlink_diag iptable_filter fuse btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 bridge stp llc tcp_diag inet_diag oracleasm(O) rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core ipmi_devintf lpc_ich ipmi_si shpchp wmi ipmi_msghandler acpi_power_meter dm_round_robin sg binfmt_misc dm_multipath nfsd auth_rpcgss nfs_acl
[26404900.439366]  lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fnic(OE) fb_sys_fops libfcoe ttm libfc crct10dif_pclmul drm scsi_transport_fc crct10dif_common i2c_core crc32c_intel scsi_tgt megaraid_sas enic(OE) fjes dm_mirror dm_region_hash dm_log dm_mod
[26404900.473261] CPU: 23 PID: 12448 Comm: oracle_12448_cs Tainted: G           OE  ------------   3.10.0-514.6.1.el7.x86_64 #1
[26404900.486191] Hardware name: Cisco Systems Inc UCSB-B200-M4/UCSB-B200-M4, BIOS B200M4.2.2.6c.0.111720151647 11/17/2015
[26404900.498634]  0000000000000000 000000000a38ce9c ffff885fbf445ba8 ffffffff816861ac
[26404900.507623]  ffff885fbf445bc0 ffffffff8112f217 ffff8830e7af8400 ffff885fbf445bf8
[26404900.516608]  ffffffff811747ae 0000000000000001 ffff885fbf44a280 ffff8830e7af8400
[26404900.525586] Call Trace:
[26404900.528795]  <NMI>  [<ffffffff816861ac>] dump_stack+0x19/0x1b
[26404900.535726]  [<ffffffff8112f217>] watchdog_overflow_callback+0xd7/0x100
[26404900.543604]  [<ffffffff811747ae>] __perf_event_overflow+0x8e/0x1f0
[26404900.551006]  [<ffffffff811753f4>] perf_event_overflow+0x14/0x20
[26404900.558108]  [<ffffffff81009d88>] intel_pmu_handle_irq+0x1f8/0x4e0
[26404900.565500]  [<ffffffff81319eec>] ? ioremap_page_range+0x27c/0x3e0
[26404900.572896]  [<ffffffff811bef44>] ? vunmap_page_range+0x1c4/0x310
[26404900.580189]  [<ffffffff811bf0a1>] ? unmap_kernel_range_noflush+0x11/0x20
[26404900.588165]  [<ffffffff813c9604>] ? ghes_copy_tofrom_phys+0x124/0x210
[26404900.595846]  [<ffffffff813c9790>] ? ghes_read_estatus+0xa0/0x190
[26404900.603044]  [<ffffffff8168ddeb>] perf_event_nmi_handler+0x2b/0x50
[26404900.614593]  [<ffffffff8168f219>] nmi_handle.isra.0+0x69/0xb0
[26404900.621509]  [<ffffffff8168f393>] do_nmi+0x133/0x410
[26404900.627539]  [<ffffffff8168e653>] end_repeat_nmi+0x1e/0x2e
[26404900.634152]  [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.640473]  [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.646792]  [<ffffffff810bc069>] ? __wake_up+0x49/0x50
[26404900.653112]  <<EOE>>  <IRQ>  [<ffffffff81173a85>] perf_event_wakeup+0x65/0xe0
[26404900.661785]  [<ffffffff81173bd7>] perf_pending_event+0xd7/0x110
[26404900.668884]  [<ffffffff8116ae4f>] irq_work_run_list+0x4f/0x70
[26404900.675789]  [<ffffffff8116ae89>] irq_work_run+0x19/0x40
[26404900.682208]  [<ffffffff8102f645>] smp_trace_irq_work_interrupt+0x35/0xb0
[26404900.690185]  [<ffffffff81697f9d>] trace_irq_work_interrupt+0x6d/0x80
[26404900.697767]  <EOI> 

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content