System crashed after softlockup traces caused by postgres process.
Issue
- System rebooted unexpectedly, RCA required.
- System crashed after softlockup message:
BUG: soft lockup - CPU#1 stuck for 70s! [postgres:553375]
CPU 1
..
Pid: 553375, comm: postgres Not tainted 2.6.32-504.8.1.el6.x86_64 #1 ...
RIP: 0010:[<ffffffff8152ce91>] [<ffffffff8152ce91>] _spin_lock+0x21/0x30
..
Process postgres (pid: 553375, threadinfo ffff882213882000, task ffff88204025aaa0)
Stack:
...
Call Trace:
[<ffffffff8115a6c2>] ? page_referenced+0xa2/0x360
[<ffffffff8113e451>] ? shrink_active_list+0x1e1/0x370
[<ffffffff8113f1d5>] ? shrink_mem_cgroup_zone+0x3c5/0x540
[<ffffffff8117eecd>] ? mem_cgroup_iter+0xfd/0x280
[<ffffffff8113f3ca>] ? shrink_zone+0x7a/0x180
[<ffffffff8113f5e5>] ? do_try_to_free_pages+0x115/0x610
[<ffffffff81131c8c>] ? get_page_from_freelist+0x15c/0x870
[<ffffffff8113fcb2>] ? try_to_free_pages+0x92/0x120
[<ffffffff811340be>] ? __alloc_pages_nodemask+0x47e/0x8d0
[<ffffffff8116c79a>] ? alloc_pages_vma+0x9a/0x150
[<ffffffff8114f6fd>] ? handle_pte_fault+0x73d/0xb00
[<ffffffff810a3da4>] ? hrtimer_start_range_ns+0x14/0x20
[<ffffffff8114fcea>] ? handle_mm_fault+0x22a/0x300
[<ffffffff8104d0d8>] ? __do_page_fault+0x138/0x480
[<ffffffff81448795>] ? sys_sendto+0x185/0x190
[<ffffffff8152ffde>] ? do_page_fault+0x3e/0xa0
[<ffffffff8152d395>] ? page_fault+0x25/0x30
Environment
- Red Hat Enterprise Linux 6
- postgresql-8.4.20-1.el6_5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.