Massive printk for writing a great number of LLT INFO messages to a slow serial console retults in a soft lockup
Issue
- Massive printk for writing a great number of LLT INFO messages to a slow serial console retults in a soft lockup
[922003.247913] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822636 HB pkts, current seqno 822636, from node 0 on link 1 (eth3)
[922003.248449] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822637 HB pkts, current seqno 822637, from node 0 on link 1 (eth3)
[922003.248513] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822639 HB pkts, current seqno 822639, from node 0 on link 1 (eth3)
[922003.248518] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822638 HB pkts, current seqno 822638, from node 0 on link 1 (eth3)
[922003.248860] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822640 HB pkts, current seqno 822640, from node 0 on link 1 (eth3)
[922003.248864] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822641 HB pkts, current seqno 822641, from node 0 on link 1 (eth3)
[922003.249213] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822642 HB pkts, current seqno 822642, from node 0 on link 1 (eth3)
[922003.745421] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822643 HB pkts, current seqno 822643, from node 0 on link 1 (eth3)
[922004.004089] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822644 HB pkts, current seqno 822644, from node 0 on link 1 (eth3)
[922004.005057] systemd-journald[1173]: /dev/kmsg buffer overrun, some messages lost.
[922028.086001] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [swapper/21:0]
[922028.086001] Modules linked in: [...]
[922028.086001] CPU: 21 PID: 0 Comm: swapper/21 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.92.1.el7.x86_64 #1
[922028.086001] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008 12/07/2018
[922028.086001] task: ffff8db8d95f3180 ti: ffff8db8d9608000 task.ti: ffff8db8d9608000
[922028.086001] RIP: 0010:[<ffffffffb811eacf>] [<ffffffffb811eacf>] native_queued_spin_lock_slowpath+0x1f/0x200
[922028.086001] RSP: 0018:ffff8dc81f343d88 EFLAGS: 00000293
[922028.086001] RAX: 0000000000000001 RBX: 0000000000000286 RCX: 0000000000000001
[922028.086001] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffb9bdd4f00004
[922028.086001] RBP: ffff8dc81f343d88 R08: ffffffffb881e7c0 R09: ffff8db8d95f3180
[922028.086001] R10: 0000000044212a63 R11: ffffb9bdd4f00000 R12: ffff8dc81f343cf8
[922028.086001] R13: ffffffffb87c9042 R14: ffff8dc81f343d88 R15: 0000000000000008
[922028.086001] FS: 0000000000000000(0000) GS:ffff8dc81f340000(0000) knlGS:0000000000000000
[922028.086001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[922028.086001] CR2: 00007f3353b6b570 CR3: 0000000bbff70000 CR4: 00000000003606e0
[922028.086001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[922028.086001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[922028.086001] Call Trace:
[922028.086001] <IRQ>
[922028.086001] [<ffffffffb87ac21a>] queued_spin_lock_slowpath+0xb/0x13
[922028.086001] [<ffffffffb87ba777>] _raw_spin_lock_bh+0x37/0x40
[922028.086001] [<ffffffffc0754644>] llt_timer_func+0x424/0x18a0 [llt]
[922028.086001] [<ffffffffc0755ac0>] ? llt_timer_func+0x18a0/0x18a0 [llt]
[922028.086001] [<ffffffffc0755bd1>] llt_timer_handler+0x111/0x170 [llt]
[922028.086001] [<ffffffffb80b0b08>] call_timer_fn+0x38/0x110
[922028.086001] [<ffffffffc0755ac0>] ? llt_timer_func+0x18a0/0x18a0 [llt]
[922028.086001] [<ffffffffb80b31fd>] run_timer_softirq+0x25d/0x340
[922028.086001] [<ffffffffb80a9595>] __do_softirq+0xf5/0x290
[922028.086001] [<ffffffffb87c8aac>] call_softirq+0x1c/0x30
[922028.086001] [<ffffffffb8030825>] do_softirq+0x65/0xa0
[922028.086001] [<ffffffffb80a9945>] irq_exit+0x115/0x120
[922028.086001] [<ffffffffb87c9ffb>] hv_stimer0_vector_handler+0x5b/0x70
[922028.086001] [<ffffffffb87c9042>] hv_stimer0_callback_vector+0x172/0x180
[922028.086001] <EOI>
[922028.086001] [<ffffffffb87b9bc0>] ? __sched_text_end+0x4/0x4
[922028.086001] [<ffffffffb87b9e1b>] ? native_safe_halt+0xb/0x30
[922028.086001] [<ffffffffb87b9bde>] default_idle+0x1e/0xd0
[922028.086001] [<ffffffffb8039570>] arch_cpu_idle+0x20/0xc0
[922028.086001] [<ffffffffb810820a>] cpu_startup_entry+0x14a/0x1e0
[922028.086001] [<ffffffffb805d3a7>] start_secondary+0x1f7/0x270
[922028.086001] [<ffffffffb80000d5>] start_cpu+0x5/0x14
[922028.086001] Code: ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 8b 17 <85> d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc cc cc
Environment
- Red Hat Enterprise Linux
- Veritas InfoScale
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.