System panicked in jbd2_journal_grab_journal_head() due to memory corruption.
Issue
- A couple of servers rebooted unexpectedly; would like root cause analysis
- No changes were made to the servers prior to the reboot
- System panicked with:
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/virtual/net/bond0/bonding/slaves
CPU 17
Modules linked in: iptable_filter ip_tables sunrpc ipmi_devintf bonding 8021q garp stp llc ipv6 power_meter hpwdt hpilo onload(U) sfc_char(U) sfc_resource(U) sfc_affinity(U) sfc_tune(U) sfc(U) i2c_algo_bit i2c_core ptp pps_core mdio sg tg3 microcode serio_raw iTCO_wdt iTCO_vendor_support ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif hpsa dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
Pid: 311, comm: khugepaged Not tainted 2.6.32-358.18.1.el6.x86_64 #1 HP ProLiant DL560 Gen8
RIP: 0010:[<ffffffffa0065f29>] [<ffffffffa0065f29>] jbd2_journal_grab_journal_head+0x9/0x40 [jbd2]
RSP: 0018:ffff881027c6d5f0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 3836343030303030 RCX: 00c0000000000809
RDX: 4220313030b63634 RSI: ffffea0044edb9d8 RDI: 3836343030303030
RBP: ffff881027c6d5f0 R08: ffffea0044edba00 R09: 00c000000000082c
R10: ffffea006422acd0 R11: 0000000000000000 R12: 0000000000000000
R13: ffff881c5e4542d0 R14: ffffea0044edb9d8 R15: ffff880fd664539c
FS: 0000000000000000(0000) GS:ffff882078820000(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fbd31519010 CR3: 0000000001a85000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process khugepaged (pid: 311, threadinfo ffff881027c6c000, task ffff881027c4e080)
Stack:
ffff881027c6d640 ffffffffa005ccf8 ffff881027c6d798 00000000eda61df8
<d> ffff881ceda61de0 ffffea0044edb9d8 ffff880fd6645000 ffffea0044edba00
<d> ffffea0044edb9d8 0000000000000001 ffff881027c6d670 ffffffffa00828b5
Call Trace:
[<ffffffffa005ccf8>] jbd2_journal_try_to_free_buffers+0x48/0x150 [jbd2]
[<ffffffffa00828b5>] ext4_releasepage+0x55/0x90 [ext4]
[<ffffffff81119100>] try_to_release_page+0x30/0x60
[<ffffffff811328e7>] shrink_page_list.clone.3+0x517/0x650
[<ffffffff81133403>] shrink_inactive_list+0x343/0x830
[<ffffffff8112fe41>] ? lru_cache_add_lru+0x21/0x40
[<ffffffff81131e2f>] ? putback_lru_page+0x8f/0x100
[<ffffffff81170ae0>] ? mem_cgroup_get_reclaim_stat_from_page+0x20/0x70
[<ffffffff81133c9e>] shrink_mem_cgroup_zone+0x3ae/0x610
[<ffffffff8117286d>] ? mem_cgroup_iter+0xfd/0x280
[<ffffffff81133f63>] shrink_zone+0x63/0xb0
[<ffffffff811340c5>] do_try_to_free_pages+0x115/0x610
[<ffffffff81134792>] try_to_free_pages+0x92/0x120
[<ffffffff8112bf78>] __alloc_pages_nodemask+0x478/0x8d0
[<ffffffff81160d3a>] alloc_pages_vma+0x9a/0x150
[<ffffffff8117a1f7>] khugepaged+0x767/0x1310
[<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff81179a90>] ? khugepaged+0x0/0x1310
[<ffffffff81096a36>] kthread+0x96/0xa0
[<ffffffff8100c0ca>] child_rip+0xa/0x20
[<ffffffff810969a0>] ? kthread+0x0/0xa0
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Code: fd ff ff eb 94 4c 89 e7 e8 25 05 15 e1 0f 1f 44 00 00 eb c6 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 <f0> 0f ba 2f 17 19 c0 85 c0 75 1c 48 8b 17 31 c0 f7 c2 00 00 01
RIP [<ffffffffa0065f29>] jbd2_journal_grab_journal_head+0x9/0x40 [jbd2]
RSP <ffff881027c6d5f0>
Environment
- Red Hat Enterprise Linux 6.4
- ext4/jbd2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.