RHEL 5 system suffers multiple panics/reboots with Veritas vxfs

Solution Verified - Updated -

Issue

System suffered multiple panics over the course of a couple of days. Latest vmcore revealed vxfs was stuck waiting on something.

NMI Watchdog detected LOCKUP on CPU 40
CPU 40 
Modules linked in: nfs nfs_acl vxodm(PFU) autofs4 dmpjbod(PU) dmpap(PU) dmpaa(PU) vxspec(PFU) vxio(PFU) vxdmp(PU) lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api cpufreq_ondemand acpi_cpufreq freq_table mperf vxportal(PFU) fdd(PFU) vxfs(PU) exportfs dm_mirror dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev qla2xxx cdc_ether i2c_i801 tpm_tis sg ide_cd bnx2 tpm usbnet i2c_core i7core_edac tpm_bios scsi_transport_fc cdrom pcspkr edac_mc dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache usb_storage ata_piix libata shpchp megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 2033, comm: osh Tainted: PF    ---- 2.6.18-274.12.1.el5 #1
RIP: 0010:[<ffffffff80064bd8>]  [<ffffffff80064bd8>] .text.lock.spinlock+0xe/0x30
RSP: 0018:ffff811d6e9e7620  EFLAGS: 00000086
RAX: 0000000000000296 RBX: ffff81307e785000 RCX: 0000000000000800
RDX: ffff811a852c3bc0 RSI: ffff8119e3505d40 RDI: ffff81307d538318
RBP: 0000000000000000 R08: 0000000000008000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8119e3505d40
R13: ffff81307d5382c0 R14: ffff81207fb0a000 R15: ffff8119e3505d40
FS:  00002ac4ab475ec0(0000) GS:ffff81207ff8dac0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000379009a830 CR3: 0000001d7587c000 CR4: 00000000000006a0
Process osh (pid: 2033, threadinfo ffff811d6e9e6000, task ffff811d7587b7e0)
Stack:  ffffffff8850767c ffff811a852c3bc0 ffff81207fb0a000 0000000000000001
 0000000000000001 000000016e9e7908 0000000000000800 0000000000000000
 ffff8119e3505d40 ffff811d6e9e7898 ffff81207fb0a000 0000000000000800
Call Trace:
 [<ffffffff8850767c>] :vxfs:vx_extentalloc_delicache+0x34/0x556
 [<ffffffff8850c0d5>] :vxfs:vx_extentalloc+0x490/0x7d4
 [<ffffffff8000f470>] __alloc_pages+0x78/0x308
 [<ffffffff8852b439>] :vxfs:vx_bmap_ext4+0x292/0x646
 [<ffffffff88528f71>] :vxfs:vx_bmap_alloc+0x1f4/0x292
 [<ffffffff885e7509>] :vxfs:vx_mkimtran+0x208/0x2da
 [<ffffffff885702ea>] :vxfs:vx_write_alloc3+0x2aa/0x907
 [<ffffffff885709d4>] :vxfs:vx_tran_write_alloc+0x8d/0x9c
 [<ffffffff8856fd79>] :vxfs:vx_write_alloc2+0x55/0x70
 [<ffffffff8867f107>] :vxfs:vx_write_alloc+0x27e/0x31f
 [<ffffffff88682847>] :vxfs:vx_write1+0x712/0xd63
 [<ffffffff8862e1ba>] :vxfs:vx_uiomove+0x588/0x5ca
 [<ffffffff8003bb71>] remove_wait_queue+0x1c/0x2c
 [<ffffffff88683e06>] :vxfs:vx_write_common_slow+0x553/0xb91
 [<ffffffff88528941>] :vxfs:vx_bmap_cache_check+0x99/0xeb
 [<ffffffff88529121>] :vxfs:vx_bmap_lookup+0x112/0x334
 [<ffffffff88684851>] :vxfs:vx_write_common+0x328/0x396
 [<ffffffff8862f7bd>] :vxfs:vx_vop_write+0x14/0x19
 [<ffffffff8863349e>] :vxfs:vx_write+0x21b/0x274
 [<ffffffff80016b92>] vfs_write+0xce/0x174
 [<ffffffff8001745b>] sys_write+0x45/0x6e
 [<ffffffff8005d116>] system_call+0x7e/0x83

Environment

  • Red Hat Enterprise Linux (RHEL) 5
  • Veritas File System

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content