Kernel panic due to CPU hardlockup because of a Veritas thread

Solution Unverified - Updated -

Issue

  • Server is unresponsive and panics with the following messages:

    NMI Watchdog detected LOCKUP on CPU 6
    CPU 6 
    Modules linked in: mptctl mptbase vxodm(PFU) vxgms(PU) amf(PU) vxglm(PU) vxfen(PU) gab(PU) llt(PU) nfs nfs_acl dmpaa(PU) vxspec(PU) vxio(PU) vxdmp(PU) lockd sunrpc bonding ipv6 xfrm_nalgo crypto_api vxcafs(PU) vxportal(PU) fdd(PU) vxfs(PU) exportfs dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sg pcspkr tg3 hpilo bnx2x serio_raw tpm_tis tpm 8021q tpm_bios mdio dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc shpchp cciss(U) sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
    Pid: 32558, comm: vx_worklist_thr Tainted: PF    ---- 2.6.18-348.el5 #1
    RIP: 0010:[<ffffffff80064bf7>]  [<ffffffff80064bf7>] .text.lock.spinlock+0x1d/0x30
    RSP: 0000:ffff812f71327868  EFLAGS: 00000086
    RAX: ffff8129623820c0 RBX: 0000000000000001 RCX: ffff812b19d26280
    RDX: ffff813fe4a62cc0 RSI: ffff812f71327898 RDI: ffff813fe4a62cf8
    RBP: ffff813ff3b34d18 R08: 0000000000000000 R09: 0000000000000000
    R10: ffff818080809308 R11: ffffffff8000bd86 R12: 0000000000000800
    R13: 0000000000000010 R14: 0000000002c1f750 R15: 0000000000001000
    FS:  00002aab7e2bb6e0(0000) GS:ffff810241c93140(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    CR2: 000000003a087000 CR3: 000000d33c22e000 CR4: 00000000000006a0
    Process vx_worklist_thr (pid: 32558, threadinfo ffff812f71326000, task ffff8133646390c0)
    Stack:  ffffffff8000bdfb 0000000000001000 00000000f3b340c8 ffff81be10518d30
    ffff81414093ebb8 0000000000001000 ffff8129623820c0 0000000000000000
    0000000000000000 0000000000000000 0000000000000010 ffff8129623820c0
    Call Trace:
    [<ffffffff8000bdfb>] __make_request+0x75/0x4ce
    [<ffffffff8001c839>] generic_make_request+0x211/0x228
    [<ffffffff8002389f>] mempool_alloc+0x31/0xe7
    [<ffffffff80063002>] thread_return+0x62/0xfe
    [<ffffffff80033695>] submit_bio+0xe6/0xed
    [<ffffffff886b11b1>] :vxfs:vx_dev_strategy+0x3eb/0x487
    [<ffffffff88dd2087>] :vxglm:vxg_free_svar_qitem+0x1b/0x1d
    [<ffffffff88de00e1>] :vxglm:vxg_lock_ilock_omnibus+0x32/0x205
    [<ffffffff88536635>] :vxfs:vx_bc_bwrite+0x14d/0x193
    [<ffffffff885399bd>] :vxfs:vx_bwrite+0x1d0/0x22f
    [<ffffffff8861def6>] :vxfs:vx_async_iupdat+0x31f/0x331
    [<ffffffff8855dbab>] :vxfs:vx_getblk_clust+0x42f/0x43e
    [<ffffffff8853a0e9>] :vxfs:vx_getblk_cmn+0x106/0x150
    [<ffffffff8853a148>] :vxfs:vx_getblk+0x15/0x1a
    [<ffffffff8861e15f>] :vxfs:vx_iupdat_local+0x257/0x513
    [<ffffffff8861e15f>] :vxfs:vx_iupdat_local+0x257/0x513
    [<ffffffff886b2649>] :vxfs:vx_tranpgfree+0x3e/0x93
    [<ffffffff886b4021>] :vxfs:vx_trancommit2+0xe8/0xfc
    [<ffffffff80157c4c>] __next_cpu+0x19/0x28
    [<ffffffff8008de9c>] find_busiest_group+0x20d/0x621
    [<ffffffff88575d15>] :vxfs:vx_cfs_iupdat+0x45/0x119
    [<ffffffff8860cfff>] :vxfs:vx_iflush_list+0x682/0x983
    [<ffffffff8860d36c>] :vxfs:vx_iflush+0x6c/0x7c
    [<ffffffff8860d300>] :vxfs:vx_iflush+0x0/0x7c
    [<ffffffff8860a217>] :vxfs:vx_workitem_process+0x13/0x34
    [<ffffffff8860a3ea>] :vxfs:vx_worklist_process+0x1b2/0x1f3
    [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9
    [<ffffffff8860dd6b>] :vxfs:vx_worklist_thread+0x6b/0xa9
    [<ffffffff88667481>] :vxfs:vx_kthread_init+0x57/0x5e
    [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9
    [<ffffffff8005dfc1>] child_rip+0xa/0x11
    [<ffffffff8860dd00>] :vxfs:vx_worklist_thread+0x0/0xa9
    [<ffffffff8866742a>] :vxfs:vx_kthread_init+0x0/0x5e
    [<ffffffff8005dfb7>] child_rip+0x0/0x11
    

Environment

  • Red Hat Enterprise Linux 5

  • Veritas Filesystem

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content