Kernel panic due to CPU hardlockup because of a Veritas thread
Issue
-
Server is unresponsive and panics with the following messages:
NMI Watchdog detected LOCKUP on CPU 6 CPU 6 Modules linked in: mptctl mptbase vxodm(PFU) vxgms(PU) amf(PU) vxglm(PU) vxfen(PU) gab(PU) llt(PU) nfs nfs_acl dmpaa(PU) vxspec(PU) vxio(PU) vxdmp(PU) lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api vxcafs(PU) vxportal(PU) fdd(PU) vxfs(PU) exportfs dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sg pcspkr tg3 hpilo bnx2x serio_raw tpm_tis tpm 8021q tpm_bios mdio dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc shpchp cciss(U) sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 32558, comm: vx_worklist_thr Tainted: PF ---- 2.6.18-348.el5 #1 RIP: 0010:[<ffffffff80064bf7>] [<ffffffff80064bf7>] .text.lock.spinlock+0x1d/0x30 RSP: 0000:ffff812f71327868 EFLAGS: 00000086 RAX: ffff8129623820c0 RBX: 0000000000000001 RCX: ffff812b19d26280 RDX: ffff813fe4a62cc0 RSI: ffff812f71327898 RDI: ffff813fe4a62cf8 RBP: ffff813ff3b34d18 R08: 0000000000000000 R09: 0000000000000000 R10: ffff818080809308 R11: ffffffff8000bd86 R12: 0000000000000800 R13: 0000000000000010 R14: 0000000002c1f750 R15: 0000000000001000 FS: 00002aab7e2bb6e0(0000) GS:ffff810241c93140(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 000000003a087000 CR3: 000000d33c22e000 CR4: 00000000000006a0 Process vx_worklist_thr (pid: 32558, threadinfo ffff812f71326000, task ffff8133646390c0) Stack: ffffffff8000bdfb 0000000000001000 00000000f3b340c8 ffff81be10518d30 ffff81414093ebb8 0000000000001000 ffff8129623820c0 0000000000000000 0000000000000000 0000000000000000 0000000000000010 ffff8129623820c0 Call Trace: [<ffffffff8000bdfb>] __make_request+0x75/0x4ce [<ffffffff8001c839>] generic_make_request+0x211/0x228 [<ffffffff8002389f>] mempool_alloc+0x31/0xe7 [<ffffffff80063002>] thread_return+0x62/0xfe [<ffffffff80033695>] submit_bio+0xe6/0xed [<ffffffff886b11b1>] :vxfs:vx_dev_strategy+0x3eb/0x487 [<ffffffff88dd2087>] :vxglm:vxg_free_svar_qitem+0x1b/0x1d [<ffffffff88de00e1>] :vxglm:vxg_lock_ilock_omnibus+0x32/0x205 [<ffffffff88536635>] :vxfs:vx_bc_bwrite+0x14d/0x193 [<ffffffff885399bd>] :vxfs:vx_bwrite+0x1d0/0x22f [<ffffffff8861def6>] :vxfs:vx_async_iupdat+0x31f/0x331 [<ffffffff8855dbab>] :vxfs:vx_getblk_clust+0x42f/0x43e [<ffffffff8853a0e9>] :vxfs:vx_getblk_cmn+0x106/0x150 [<ffffffff8853a148>] :vxfs:vx_getblk+0x15/0x1a [<ffffffff8861e15f>] :vxfs:vx_iupdat_local+0x257/0x513 [<ffffffff8861e15f>] :vxfs:vx_iupdat_local+0x257/0x513 [<ffffffff886b2649>] :vxfs:vx_tranpgfree+0x3e/0x93 [<ffffffff886b4021>] :vxfs:vx_trancommit2+0xe8/0xfc [<ffffffff80157c4c>] __next_cpu+0x19/0x28 [<ffffffff8008de9c>] find_busiest_group+0x20d/0x621 [<ffffffff88575d15>] :vxfs:vx_cfs_iupdat+0x45/0x119 [<ffffffff8860cfff>] :vxfs:vx_iflush_list+0x682/0x983 [<ffffffff8860d36c>] :vxfs:vx_iflush+0x6c/0x7c [<ffffffff8860d300>] :vxfs:vx_iflush+0x0/0x7c [<ffffffff8860a217>] :vxfs:vx_workitem_process+0x13/0x34 [<ffffffff8860a3ea>] :vxfs:vx_worklist_process+0x1b2/0x1f3 [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9 [<ffffffff8860dd6b>] :vxfs:vx_worklist_thread+0x6b/0xa9 [<ffffffff88667481>] :vxfs:vx_kthread_init+0x57/0x5e [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9 [<ffffffff8005dfc1>] child_rip+0xa/0x11 [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9 [<ffffffff8866742a>] :vxfs:vx_kthread_init+0x0/0x5e [<ffffffff8005dfb7>] child_rip+0x0/0x11
Environment
-
Red Hat Enterprise Linux 5
-
Veritas Filesystem
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.