Unable to login to the system (eg. with ssh) after rsyslog crashes, it looks frozen

Solution Verified - Updated -

Issue

  • System hangs after sending a large amount of syslog messages via TCP
  • I'm not able to login through ssh, but the server answers ping requests
  • abrt-addon-ccpp user space deadlock when rsyslog daemon crashes
  • rsyslogd crashes and no core dump is saved by abrt
  • Below are the common call traces found on the logs when the problem happens:
INFO: task rsyslogd:13421 blocked for more than 120 seconds.
      Not tainted 2.6.32-431.29.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsyslogd      D 0000000000000003     0 13421      1 0x00000080
 ffff88204eb15c98 0000000000000082 ffff882053935a28 ffff882053933500
 ffff88204eb15c68 ffffffff810af000 ffff882053935a28 0000000000000000
 ffff882050d105f8 ffff88204eb15fd8 000000000000fbc8 ffff882050d105f8
Call Trace:
 [<ffffffff810af000>] ? exit_robust_list+0x90/0x160
 [<ffffffff810767e5>] exit_mm+0x95/0x180
 [<ffffffff81076c2f>] do_exit+0x15f/0x870
 [<ffffffff81077398>] do_group_exit+0x58/0xd0
 [<ffffffff8108cd46>] get_signal_to_deliver+0x1f6/0x460
 [<ffffffff8100a265>] do_signal+0x75/0x800
 [<ffffffff810149b9>] ? read_tsc+0x9/0x20
 [<ffffffff811a04b8>] ? poll_select_copy_remaining+0xf8/0x150
 [<ffffffff8100aa80>] do_notify_resume+0x90/0xc0
 [<ffffffff8100b341>] int_signal+0x12/0x17

Environment

  • Red Hat Enterprise Linux 6
  • abrt-addon-ccpp package installed and abrt-ccpp service enabled

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content