Machine slowdown or lockup, massive rate of logging seen on console

Solution Verified - Updated -

Issue

  • Slowdown of the machine is observed, accompanied by stack traces and massive logging on the console (tty). This can go as far as leading to a crash or softlockup.
  • Console logging has been known to produce soft lockups, hard lockups, NTP or time instability, and can cause various other similar issues.
  • The behavior is triggered by any high rate logging to the tty console (which is framebuffer by default), a typical situation is using iptables LOG target, and generating or encountering massive traffic that is matched by the -j LOG rules.
  • One example of such traffic seen was being generated for security auditing, another was generated "naturally" in a Windows network by Samba NetBIOS broadcast.
  • Hardlockup or soft lockup panic in framebuffer due to repeated kernel printk.
  • Lockup in framebuffer console like:
Call Trace:
 [<ffffffff80182784>] cfb_fillrect+0x1ed/0x2f6
 [<ffffffff8017beb0>] fbcon_redraw+0x130/0x169
 [<ffffffff8017f07e>] bit_putcs+0x42b/0x481
 [<ffffffff80156347>] __next_cpu+0x19/0x28
 [<ffffffff8017f564>] bit_cursor+0x490/0x4ac
 [<ffffffff8017a2cb>] fbcon_putcs+0x20e/0x260
 [<ffffffff8017ec53>] bit_putcs+0x0/0x481
 [<ffffffff8017be53>] fbcon_redraw+0xd3/0x169
  • Lockups detected like NMI Watchdog detected LOCKUP on CPU #

Environment

  • Red Hat Enterprise Linux
  • Default kernel.printk configuration (console log level is 6)
  • Heavy kernel logging to console

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content