RHEL6: kernel crash in NFS code, RIP nfs_access_cache_shrinker called from shrink_slab by ksmtuned

Solution Unverified - Updated -

Issue

  • We've been seeing this same crash repeatedly on one of our RHEL6 machines, kernel 2.6.32-220.9.1.el6.x86_64. It seems to happen always in NFS code and it seems it's trying to access a pointer in memory in a misaligned address which definitely looks wrong.
  • Here's an oops from one crash:
general protection fault: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.4/0000:06:00.0/0000:07:00.0/local_cpus
CPU 7
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipmi_devintf ipmi_si ipmi_msghandler
 cpufreq_ondemand acpi_cpufreq freq_table mperf 
openafs(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) 
vhost_net macvtap macvlan tun kvm_intel kvm sg bnx2 microcode cdc_ether usbnet mii serio_raw i2c_i801 
i2c_core iTCO_wdt iTCO_vendor_support ioatdma dca i7core_edac edac_core shpchp ext3 jbd mbcache sd_mod 
crc_t10dif qla2xxx scsi_transport_fc scsi_tgt mptsas mptscsih mptbase scsi_transport_sas dm_mirror 
dm_region_hash dm_log dm_mod [last unloaded: mptctl]

Pid: 6217, comm: ksmtuned Tainted: P           ----------------   2.6.32-220.9.1.el6.x86_64 #1 IBM BladeCenter HS22 -[7870AC1]-/68Y8163
RIP: 0010:[<ffffffffa0786d8c>]  [<ffffffffa0786d8c>] nfs_access_cache_shrinker+0xbc/0x210 [nfs]
RSP: 0018:ffff8806d8a75808  EFLAGS: 00010283
RAX: ffff880b9cb4973f RBX: ffff88066446b140 RCX: ffff88066446b218
RDX: ffff88066446b218 RSI: ff8805469bbbd8ff RDI: ffff880b9cb49757
RBP: ffff8806d8a75858 R08: ffffffffa07d1d30 R09: ffff880bfae3a880
R10: ffff8806f3f1f880 R11: 0000000000000000 R12: ffff880648f6a100
R13: ffff88066446b228 R14: ffff88066446b3b8 R15: 0000000000000044
FS:  00007f3941e87700(0000) GS:ffff8806954c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fffa7d0ed54 CR3: 00000006d8a69000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ksmtuned (pid: 6217, threadinfo ffff8806d8a74000, task ffff8806aabeab40)
Stack:
 ffff8801f698ed40 ffff880648f6a5f0 ffff8801f698ed58 ffff8801caa48318
<0> ffff8806d8a75848 ffffffffa07d2000 0000000000053550 0000000000023d0c
<0> 00000000000200da 0000000000012ae0 ffff8806d8a758b8 ffffffff8112995a
Call Trace:
 [<ffffffff8112995a>] shrink_slab+0x12a/0x1a0
 [<ffffffff8112c004>] zone_reclaim+0x284/0x410
 [<ffffffff8112cd20>] ? isolate_pages_global+0x0/0x350
 [<ffffffff811225d4>] get_page_from_freelist+0x694/0x820
 [<ffffffff81123851>] __alloc_pages_nodemask+0x111/0x940
 [<ffffffff8116a2d8>] ? __mem_cgroup_try_charge+0x78/0x420
 [<ffffffff81158c2a>] alloc_pages_vma+0x9a/0x150
 [<ffffffff8113aa4d>] do_wp_page+0xfd/0x8d0
 [<ffffffff8113b669>] ? __do_fault+0x449/0x510
 [<ffffffff8113b9fd>] handle_pte_fault+0x2cd/0xb50
 [<ffffffff810471c7>] ? pte_alloc_one+0x37/0x50
 [<ffffffff8113c464>] handle_mm_fault+0x1e4/0x2b0
 [<ffffffff81042b79>] __do_page_fault+0x139/0x480
 [<ffffffff814f253e>] ? do_page_fault+0x3e/0xa0
 [<ffffffff814ef8f5>] ? page_fault+0x25/0x30
 [<ffffffff81080331>] ? do_sigaction+0x91/0x1d0
 [<ffffffff814f253e>] do_page_fault+0x3e/0xa0
 [<ffffffff814ef8f5>] page_fault+0x25/0x30
Code: f6 0f 84 20 01 00 00 49 81 ec e8 00 00 00 44 8d 7e ff e9 b4 00 00 00 0f 1f 84 00 00 00 00 00 48 8d 47 e8 48 8b 70 18 48 8b 50 20 <48> 89 56 08 48 89 32 48 8d 75 c0 48 8b 55 c0 48 89 4d b8 48 89
RIP  [<ffffffffa0786d8c>] nfs_access_cache_shrinker+0xbc/0x210 [nfs]
 RSP <ffff8806d8a75808>

Environment

  • Red Hat Enterprise Linux 6.2
    • 2.6.32-220.9.1.el6.x86_64
  • Proprietary kernel modules loaded
openafs(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) 

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content