Server is crashed by NMI watchdog due to spinlock contention in "vxfs" module code
Issue
- The server has crashed with the following messages in the kernel log:
NMI Watchdog detected LOCKUP on CPU 0
CPU 0
Modules linked in: sr_mod joydev cdc_ether usbnet nfs fscache vxodm(PFU) vxfen(PU) gab(PU) llt(PU) nfsd lockd nfs_acl
auth_rpcgss sunrpc autofs4 ipmi_devintf ipmi_si ipmi_msghandler dmpjbod(PU) dmpap(PU) dmpaa(PU) vxspec(PFU) vxio(PFU)
vxdmp(PU) cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ipv6 xfrm_nalgo crypto_api vxportal(PFU) fdd(PFU) vxfs(PU)
exportfs dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi
acpi_memhotplug ac parport_pc lp parport sg lpfc i7core_edac ata_piix i2c_i801 tpm_tis ide_cd tpm edac_mc libata i2c_core
tpm_bios scsi_transport_fc pcspkr bnx2 cdrom dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror
dm_log dm_mod usb_storage mptspi scsi_transport_spi mptsas mptscsih scsi_transport_sas mptbase shpchp megaraid_sas sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 28697, comm: osh Tainted: PF 2.6.18-238.1.1.el5 #1
RIP: 0010:[<ffffffff80064bcb>] [<ffffffff80064bcb>] .text.lock.spinlock+0x11/0x30
RSP: 0018:ffff8111a007f670 EFLAGS: 00000086
RAX: 0000000000000282 RBX: 0000000000008000 RCX: 0000000000008180
RDX: ffff8103f86f2480 RSI: ffff8111a007f6f0 RDI: ffff810719af1498
RBP: ffff81061ab91800 R08: 0000000000000001 R09: 00000000cee24d78
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000008180
R13: ffff8108bc903bc0 R14: ffff8106aba1f000 R15: ffff8103f86f2480
FS: 00002af52c44eec0(0000) GS:ffffffff80425000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000001b494a68 CR3: 00000011a004d000 CR4: 00000000000006e0
Process osh (pid: 28697, threadinfo ffff8111a007e000, task ffff811359c3e100)
Stack: ffffffff885a32ef ffff8111a007f6f0 0000000000008000 ffff81061ab91800
0000000000008180 ffff8108bc903bc0 0000000000008180 ffff8107cee247f0
ffffffff885a3a21 00000001cee24d78 ffff8106aba1f000 ffff81082e2f2000
Call Trace:
[<ffffffff885a32ef>] :vxfs:vx_delicache_reuse+0x33/0x3a9
[<ffffffff885a3a21>] :vxfs:vx_ialloc+0x3bc/0x3d7
[<ffffffff885618a5>] :vxfs:vx_dirmakeinode+0x1fd/0x3dd
[<ffffffff88561b77>] :vxfs:vx_dircreate+0xf2/0x22b
[<ffffffff8003bdd1>] remove_wait_queue+0x1c/0x2c
[<ffffffff88563332>] :vxfs:vx_dircreate_tran+0x423/0xbab
[<ffffffff8864aafc>] :vxfs:vx_pd_create+0xa48/0xdf4
[<ffffffff8856a2bc>] :vxfs:vx_exh_hash32+0x0/0x2c
[<ffffffff885705bb>] :vxfs:vx_dexh_keycheck+0x0/0x2d9
[<ffffffff8856a2e8>] :vxfs:vx_dexh_markbad+0x0/0x1f
[<ffffffff8864af55>] :vxfs:vx_create1_pd+0xad/0x206
[<ffffffff8864b199>] :vxfs:vx_create1+0x8e/0xe7
[<ffffffff885a01ed>] :vxfs:vx_create_vp+0x16e/0x3df
[<ffffffff885a0581>] :vxfs:vx_create+0x123/0x1a9
[<ffffffff885c2071>] :vxfs:vx_rwsleep_rec_unlock+0x13/0xad
[<ffffffff88605a81>] :vxfs:vx_access+0x125/0x155
[<ffffffff8003a704>] vfs_create+0xe6/0x158
[<ffffffff8001b1f2>] open_namei+0x19d/0x712
[<ffffffff80027511>] do_filp_open+0x1c/0x38
[<ffffffff80019f7e>] do_sys_open+0x44/0xbe
[<ffffffff8005d116>] system_call+0x7e/0x83
Code: 7e f9 e9 f9 fe ff ff f3 90 83 3f 00 7e f9 e9 f8 fe ff ff f3
Environment
- Red Hat Enterprise Linux 5
- Veritas Cluster Suite
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.