sosreport hangs while collecting statistics from the "/proc/timer_*" files.
Issue
- sosreport hangs while collecting statistics from the
/proc/timer_listfile on the busy system. - Huge number of connections over IPsec create software timers. Some third party tool operates in such a way that it registers >2-3 Lakh
timers in kernel. - Observed
softlockupswhile reading/proc/timer_listwhen we try to run sosreport on such as busy server.
[506290.233300] Kernel panic - not syncing: softlockup: hung tasks
[506290.233302] CPU: 12 PID: 11898 Comm: sosreport Tainted: G L ------------ 3.10.0-693.17.1.el7.x86_64 #1
[506290.233303] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015
[506290.233303] Call Trace:
[506290.233308] <IRQ> [<ffffffff816a6071>] dump_stack+0x19/0x1b
[506290.233312] [<ffffffff8169ff34>] panic+0xe8/0x20d
[506290.233316] [<ffffffff8102d76f>] ? show_regs+0x5f/0x210
[506290.233319] [<ffffffff81131531>] watchdog_timer_fn+0x221/0x230
[506290.233320] [<ffffffff81131310>] ? watchdog+0x40/0x40
[506290.233323] [<ffffffff810b6864>] __hrtimer_run_queues+0xd4/0x260
[506290.233325] [<ffffffff810b6dff>] hrtimer_interrupt+0xaf/0x1d0
[506290.233329] [<ffffffff81053a05>] local_apic_timer_interrupt+0x35/0x60
[506290.233333] [<ffffffff816bea4d>] smp_apic_timer_interrupt+0x3d/0x50
[506290.233335] [<ffffffff816b9d32>] apic_timer_interrupt+0x232/0x240
[506290.233338] <EOI> [<ffffffff810b6770>] ? hrtimer_start_range_ns+0x3c0/0x3c0
[506290.233340] [<ffffffff816adce5>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[506290.233342] [<ffffffff810ef4f7>] print_active_timers+0xf7/0x370
[506290.233344] [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233346] [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233348] [<ffffffff81092180>] ? __tasklet_hi_schedule+0xa0/0xa0
[506290.233350] [<ffffffff810b6782>] ? hrtimer_start+0x12/0x20
[506290.233352] [<ffffffff810ef893>] print_cpu+0x123/0x700
[506290.233354] [<ffffffff810f0310>] timer_list_show+0x60/0x110
[506290.233356] [<ffffffff812286f0>] seq_read+0x250/0x3b0
[506290.233358] [<ffffffff812725fd>] proc_reg_read+0x3d/0x80
[506290.233359] [<ffffffff81202b5c>] vfs_read+0x9c/0x170
[506290.233361] [<ffffffff81203a1f>] SyS_read+0x7f/0xe0
[506290.233362] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[506290.233364] [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
Environment
- Red Hat Enterprise Linux 7
- Kernel version 3.10.0-693.17.1.el7.x86_64
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
