sosreport hangs while collecting statistics from the "/proc/timer_*" files.

Solution Verified - Updated -

Issue

  • sosreport hangs while collecting statistics from the /proc/timer_list file on the busy system.
  • Huge number of connections over IPsec create software timers. Some third party tool operates in such a way that it registers >2-3 Lakh
    timers in kernel.
  • Observed softlockups while reading /proc/timer_list when we try to run sosreport on such as busy server.
[506290.233300] Kernel panic - not syncing: softlockup: hung tasks
[506290.233302] CPU: 12 PID: 11898 Comm: sosreport Tainted: G             L ------------   3.10.0-693.17.1.el7.x86_64 #1
[506290.233303] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015
[506290.233303] Call Trace:
[506290.233308]  <IRQ>  [<ffffffff816a6071>] dump_stack+0x19/0x1b
[506290.233312]  [<ffffffff8169ff34>] panic+0xe8/0x20d
[506290.233316]  [<ffffffff8102d76f>] ? show_regs+0x5f/0x210
[506290.233319]  [<ffffffff81131531>] watchdog_timer_fn+0x221/0x230
[506290.233320]  [<ffffffff81131310>] ? watchdog+0x40/0x40
[506290.233323]  [<ffffffff810b6864>] __hrtimer_run_queues+0xd4/0x260
[506290.233325]  [<ffffffff810b6dff>] hrtimer_interrupt+0xaf/0x1d0
[506290.233329]  [<ffffffff81053a05>] local_apic_timer_interrupt+0x35/0x60
[506290.233333]  [<ffffffff816bea4d>] smp_apic_timer_interrupt+0x3d/0x50
[506290.233335]  [<ffffffff816b9d32>] apic_timer_interrupt+0x232/0x240
[506290.233338]  <EOI>  [<ffffffff810b6770>] ? hrtimer_start_range_ns+0x3c0/0x3c0
[506290.233340]  [<ffffffff816adce5>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[506290.233342]  [<ffffffff810ef4f7>] print_active_timers+0xf7/0x370
[506290.233344]  [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233346]  [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233348]  [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233350]  [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233352]  [<ffffffff810ef893>] print_cpu+0x123/0x700
[506290.233354]  [<ffffffff810f0310>] timer_list_show+0x60/0x110
[506290.233356]  [<ffffffff812286f0>] seq_read+0x250/0x3b0
[506290.233358]  [<ffffffff812725fd>] proc_reg_read+0x3d/0x80
[506290.233359]  [<ffffffff81202b5c>] vfs_read+0x9c/0x170
[506290.233361]  [<ffffffff81203a1f>] SyS_read+0x7f/0xe0
[506290.233362]  [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[506290.233364]  [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214

Environment

  • Red Hat Enterprise Linux 7
  • Kernel version 3.10.0-693.17.1.el7.x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content