Page table corruption in RHEL5

Issue

Page table corruption, like most memory corruption can have many different symptoms, however there are several common ones:

• Kernel BUG at mm/memory.c:428. Example oops message below, but note that the many different backtraces are possible.

Kernel BUG at mm/memory.c:428
invalid opcode: 0000 [1] SMP 
last sysfs file: ...
CPU 0 
Modules linked in: ...
Pid: XXXX, comm: XXXX Tainted: G     ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff8000c900>]  [<ffffffff8000c900>] vm_normal_page+0x76/0x83
RSP: 0018:ffff8101f7b39c20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810009003400 RCX: ffff8101f8210ac0
RDX: 9b92aae1938facbd RSI: 00000000eb000000 RDI: ffff81021007f608
RBP: ffff810106d52188 R08: ffff810107e67d01 R09: ffff81000001b600
R10: ffff8101f7b39b38 R11: ffff81023be63140 R12: 80000001f3c07067
R13: 00000000eb000000 R14: ffff8101f3c08000 R15: ffff81023be63140
FS:  0000000000000000(0000) GS:ffffffff80431000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000cbdd7bd8 CR3: 0000000000201000 CR4: 00000000000006a0
Process java (pid: 5010, threadinfo ffff8101f7b38000, task ffff810208ec5080)
Stack:  ffffffff80007abb 0000000000000000 ffff8101f7b39d08 ffffffffffffffff
 0000000000000000 ffff81021007f608 ffff8101f7b39d10 0000000000227000
 0000000000000000 0000000108ec5268 ffff81023be63140 00000000f4180000
Call Trace:
 [<ffffffff80007abb>] unmap_vmas+0x3e4/0x903
 [<ffffffff80039e8e>] exit_mmap+0x87/0x104
 [<ffffffff8003c01d>] mmput+0x30/0x82
 [<ffffffff800158e8>] do_exit+0x2e7/0x931
 [<ffffffff80048e75>] cpuset_exit+0x0/0x88
 [<ffffffff8002b3b8>] get_signal_to_deliver+0x465/0x494
 [<ffffffff8005a91a>] do_notify_resume+0x9c/0x7af
 [<ffffffff8008ee72>] default_wake_function+0x0/0xe
 [<ffffffff8002ffb6>] __up_write+0x27/0xf2
 [<ffffffff800bab0e>] audit_syscall_exit+0x329/0x344
 [<ffffffff800a74d3>] compat_sys_futex+0x104/0x124
 [<ffffffff8005d32e>] int_signal+0x12/0x17


Code: 0f 0b 68 43 03 2c 80 c2 ac 01 eb fe c3 41 56 48 c7 c0 80 ad 
RIP  [<ffffffff8000c900>] vm_normal_page+0x76/0x83
 RSP <ffff8101f7b39c20>

• Bad swap offset entry messages. For example

May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry df0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry ff0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry ef0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry 770000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry ff0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry fb0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry be0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry de0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry fd0000000000
May 17 16:07:48 hostname kernel: swap_free: Bad swap offset entry fe0000000000

• Bad pte messages. For example

Bad pte = 8b3ea828581aae5a, process = XXXX, vm_flags = 100073, vaddr = eb12af50

Call Trace:
 [<ffffffff800091de>] __handle_mm_fault+0xa6f/0x103b
 [<ffffffff80067202>] do_page_fault+0x499/0x842
 [<ffffffff80063002>] thread_return+0x62/0xfe
 [<ffffffff8005dde9>] error_exit+0x0/0x84

VM: killing process XXXX