sosreport hangs while collecting statistics from the "/proc/timer_*" files.
Issue
- sosreport hangs while collecting statistics from the
/proc/timer_list
file on the busy system. - Huge number of connections over IPsec create software timers. Some third party tool operates in such a way that it registers >2-3 Lakh
timers in kernel. - Observed
softlockups
while reading/proc/timer_list
when we try to run sosreport on such as busy server.
[506290.233300] Kernel panic - not syncing: softlockup: hung tasks
[506290.233302] CPU: 12 PID: 11898 Comm: sosreport Tainted: G L ------------ 3.10.0-693.17.1.el7.x86_64 #1
[506290.233303] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015
[506290.233303] Call Trace:
[506290.233308] <IRQ> [<ffffffff816a6071>] dump_stack+0x19/0x1b
[506290.233312] [<ffffffff8169ff34>] panic+0xe8/0x20d
[506290.233316] [<ffffffff8102d76f>] ? show_regs+0x5f/0x210
[506290.233319] [<ffffffff81131531>] watchdog_timer_fn+0x221/0x230
[506290.233320] [<ffffffff81131310>] ? watchdog+0x40/0x40
[506290.233323] [<ffffffff810b6864>] __hrtimer_run_queues+0xd4/0x260
[506290.233325] [<ffffffff810b6dff>] hrtimer_interrupt+0xaf/0x1d0
[506290.233329] [<ffffffff81053a05>] local_apic_timer_interrupt+0x35/0x60
[506290.233333] [<ffffffff816bea4d>] smp_apic_timer_interrupt+0x3d/0x50
[506290.233335] [<ffffffff816b9d32>] apic_timer_interrupt+0x232/0x240
[506290.233338] <EOI> [<ffffffff810b6770>] ? hrtimer_start_range_ns+0x3c0/0x3c0
[506290.233340] [<ffffffff816adce5>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[506290.233342] [<ffffffff810ef4f7>] print_active_timers+0xf7/0x370
[506290.233344] [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233346] [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233348] [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233350] [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233352] [<ffffffff810ef893>] print_cpu+0x123/0x700
[506290.233354] [<ffffffff810f0310>] timer_list_show+0x60/0x110
[506290.233356] [<ffffffff812286f0>] seq_read+0x250/0x3b0
[506290.233358] [<ffffffff812725fd>] proc_reg_read+0x3d/0x80
[506290.233359] [<ffffffff81202b5c>] vfs_read+0x9c/0x170
[506290.233361] [<ffffffff81203a1f>] SyS_read+0x7f/0xe0
[506290.233362] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[506290.233364] [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
Environment
- Red Hat Enterprise Linux 7
- Kernel version 3.10.0-693.17.1.el7.x86_64
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.