Massive printk for writing a great number of LLT INFO messages to a slow serial console retults in a soft lockup

Solution Unverified - Updated -

Issue

  • Massive printk for writing a great number of LLT INFO messages to a slow serial console retults in a soft lockup
[922003.247913] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822636 HB pkts, current seqno 822636, from node 0 on link 1 (eth3)
[922003.248449] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822637 HB pkts, current seqno 822637, from node 0 on link 1 (eth3)
[922003.248513] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822639 HB pkts, current seqno 822639, from node 0 on link 1 (eth3)
[922003.248518] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822638 HB pkts, current seqno 822638, from node 0 on link 1 (eth3)
[922003.248860] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822640 HB pkts, current seqno 822640, from node 0 on link 1 (eth3)
[922003.248864] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822641 HB pkts, current seqno 822641, from node 0 on link 1 (eth3)
[922003.249213] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822642 HB pkts, current seqno 822642, from node 0 on link 1 (eth3)
[922003.745421] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822643 HB pkts, current seqno 822643, from node 0 on link 1 (eth3)
[922004.004089] LLT INFO V-14-1-10023 llt_msg_recv1: missed 822644 HB pkts, current seqno 822644, from node 0 on link 1 (eth3)
[922004.005057] systemd-journald[1173]: /dev/kmsg buffer overrun, some messages lost.
[922028.086001] NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [swapper/21:0]
[922028.086001] Modules linked in: [...]
[922028.086001] CPU: 21 PID: 0 Comm: swapper/21 Kdump: loaded Tainted: P           OE  ------------   3.10.0-1160.92.1.el7.x86_64 #1
[922028.086001] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[922028.086001] task: ffff8db8d95f3180 ti: ffff8db8d9608000 task.ti: ffff8db8d9608000
[922028.086001] RIP: 0010:[<ffffffffb811eacf>]  [<ffffffffb811eacf>] native_queued_spin_lock_slowpath+0x1f/0x200
[922028.086001] RSP: 0018:ffff8dc81f343d88  EFLAGS: 00000293
[922028.086001] RAX: 0000000000000001 RBX: 0000000000000286 RCX: 0000000000000001
[922028.086001] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffb9bdd4f00004
[922028.086001] RBP: ffff8dc81f343d88 R08: ffffffffb881e7c0 R09: ffff8db8d95f3180
[922028.086001] R10: 0000000044212a63 R11: ffffb9bdd4f00000 R12: ffff8dc81f343cf8
[922028.086001] R13: ffffffffb87c9042 R14: ffff8dc81f343d88 R15: 0000000000000008
[922028.086001] FS:  0000000000000000(0000) GS:ffff8dc81f340000(0000) knlGS:0000000000000000
[922028.086001] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[922028.086001] CR2: 00007f3353b6b570 CR3: 0000000bbff70000 CR4: 00000000003606e0
[922028.086001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[922028.086001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[922028.086001] Call Trace:
[922028.086001]  <IRQ> 
[922028.086001]  [<ffffffffb87ac21a>] queued_spin_lock_slowpath+0xb/0x13
[922028.086001]  [<ffffffffb87ba777>] _raw_spin_lock_bh+0x37/0x40
[922028.086001]  [<ffffffffc0754644>] llt_timer_func+0x424/0x18a0 [llt]
[922028.086001]  [<ffffffffc0755ac0>] ? llt_timer_func+0x18a0/0x18a0 [llt]
[922028.086001]  [<ffffffffc0755bd1>] llt_timer_handler+0x111/0x170 [llt]
[922028.086001]  [<ffffffffb80b0b08>] call_timer_fn+0x38/0x110
[922028.086001]  [<ffffffffc0755ac0>] ? llt_timer_func+0x18a0/0x18a0 [llt]
[922028.086001]  [<ffffffffb80b31fd>] run_timer_softirq+0x25d/0x340
[922028.086001]  [<ffffffffb80a9595>] __do_softirq+0xf5/0x290
[922028.086001]  [<ffffffffb87c8aac>] call_softirq+0x1c/0x30
[922028.086001]  [<ffffffffb8030825>] do_softirq+0x65/0xa0
[922028.086001]  [<ffffffffb80a9945>] irq_exit+0x115/0x120
[922028.086001]  [<ffffffffb87c9ffb>] hv_stimer0_vector_handler+0x5b/0x70
[922028.086001]  [<ffffffffb87c9042>] hv_stimer0_callback_vector+0x172/0x180
[922028.086001]  <EOI> 
[922028.086001]  [<ffffffffb87b9bc0>] ? __sched_text_end+0x4/0x4
[922028.086001]  [<ffffffffb87b9e1b>] ? native_safe_halt+0xb/0x30
[922028.086001]  [<ffffffffb87b9bde>] default_idle+0x1e/0xd0
[922028.086001]  [<ffffffffb8039570>] arch_cpu_idle+0x20/0xc0
[922028.086001]  [<ffffffffb810820a>] cpu_startup_entry+0x14a/0x1e0
[922028.086001]  [<ffffffffb805d3a7>] start_secondary+0x1f7/0x270
[922028.086001]  [<ffffffffb80000d5>] start_cpu+0x5/0x14
[922028.086001] Code: ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 8b 17 <85> d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc cc cc 

Environment

  • Red Hat Enterprise Linux
  • Veritas InfoScale

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content