RHEL6: kernel crash in NFS code, RIP nfs_access_cache_shrinker called from shrink_slab by ksmtuned
Issue
- We've been seeing this same crash repeatedly on one of our RHEL6 machines, kernel 2.6.32-220.9.1.el6.x86_64. It seems to happen always in NFS code and it seems it's trying to access a pointer in memory in a misaligned address which definitely looks wrong.
- Here's an oops from one crash:
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.4/0000:06:00.0/0000:07:00.0/local_cpus
CPU 7
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipmi_devintf ipmi_si ipmi_msghandler
cpufreq_ondemand acpi_cpufreq freq_table mperf
openafs(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U)
vhost_net macvtap macvlan tun kvm_intel kvm sg bnx2 microcode cdc_ether usbnet mii serio_raw i2c_i801
i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext3 jbd mbcache sd_mod
crc_t10dif qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: mptctl]
Pid: 6217, comm: ksmtuned Tainted: P ---------------- 2.6.32-220.9.1.el6.x86_64 #1 IBM BladeCenter HS22 -[7870AC1]-/68Y8163
RIP: 0010:[<ffffffffa0786d8c>] [<ffffffffa0786d8c>] nfs_access_cache_shrinker+0xbc/0x210 [nfs]
RSP: 0018:ffff8806d8a75808 EFLAGS: 00010283
RAX: ffff880b9cb4973f RBX: ffff88066446b140 RCX: ffff88066446b218
RDX: ffff88066446b218 RSI: ff8805469bbbd8ff RDI: ffff880b9cb49757
RBP: ffff8806d8a75858 R08: ffffffffa07d1d30 R09: ffff880bfae3a880
R10: ffff8806f3f1f880 R11: 0000000000000000 R12: ffff880648f6a100
R13: ffff88066446b228 R14: ffff88066446b3b8 R15: 0000000000000044
FS: 00007f3941e87700(0000) GS:ffff8806954c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fffa7d0ed54 CR3: 00000006d8a69000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ksmtuned (pid: 6217, threadinfo ffff8806d8a74000, task ffff8806aabeab40)
Stack:
ffff8801f698ed40 ffff880648f6a5f0 ffff8801f698ed58 ffff8801caa48318
<0> ffff8806d8a75848 ffffffffa07d2000 0000000000053550 0000000000023d0c
<0> 00000000000200da 0000000000012ae0 ffff8806d8a758b8 ffffffff8112995a
Call Trace:
[<ffffffff8112995a>] shrink_slab+0x12a/0x1a0
[<ffffffff8112c004>] zone_reclaim+0x284/0x410
[<ffffffff8112cd20>] ? isolate_pages_global+0x0/0x350
[<ffffffff811225d4>] get_page_from_freelist+0x694/0x820
[<ffffffff81123851>] __alloc_pages_nodemask+0x111/0x940
[<ffffffff8116a2d8>] ? __mem_cgroup_try_charge+0x78/0x420
[<ffffffff81158c2a>] alloc_pages_vma+0x9a/0x150
[<ffffffff8113aa4d>] do_wp_page+0xfd/0x8d0
[<ffffffff8113b669>] ? __do_fault+0x449/0x510
[<ffffffff8113b9fd>] handle_pte_fault+0x2cd/0xb50
[<ffffffff810471c7>] ? pte_alloc_one+0x37/0x50
[<ffffffff8113c464>] handle_mm_fault+0x1e4/0x2b0
[<ffffffff81042b79>] __do_page_fault+0x139/0x480
[<ffffffff814f253e>] ? do_page_fault+0x3e/0xa0
[<ffffffff814ef8f5>] ? page_fault+0x25/0x30
[<ffffffff81080331>] ? do_sigaction+0x91/0x1d0
[<ffffffff814f253e>] do_page_fault+0x3e/0xa0
[<ffffffff814ef8f5>] page_fault+0x25/0x30
Code: f6 0f 84 20 01 00 00 49 81 ec e8 00 00 00 44 8d 7e ff e9 b4 00 00 00 0f 1f 84 00 00 00 00 00 48 8d 47 e8 48 8b 70 18 48 8b 50 20 <48> 89 56 08 48 89 32 48 8d 75 c0 48 8b 55 c0 48 89 4d b8 48 89
RIP [<ffffffffa0786d8c>] nfs_access_cache_shrinker+0xbc/0x210 [nfs]
RSP <ffff8806d8a75808>
Environment
- Red Hat Enterprise Linux 6.2
- 2.6.32-220.9.1.el6.x86_64
- Proprietary kernel modules loaded
openafs(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.