System crashed :vxfs:vx_ilist_chunkclean+0x170/0x605

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 5

Issue

  • System rebooted with the below logs
...
vxfs: msgcnt 30216 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/ditfd1vg/ditfd1lv file system full (8 block extent)
vxfs: msgcnt 30217 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/ditfd1vg/ditfd1lv file system full (1 block extent)

Unable to handle kernel NULL pointer dereference at 0000000000000298 RIP: 
 [<ffffffff884b43d7>] :vxfs:vx_ilist_chunkclean+0x170/0x605
PGD 193478067 PUD 193477067 PMD 0 
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:11.0/0000:02:02.0/irq
CPU 2 
Modules linked in: vxodm(PFU) nfs nfs_acl autofs4 vxspec(PFU) vxio(PFU) vxdmp(PU) lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi acpiphp vxportal(PFU) fdd(PFU) vxfs(PU) exportfs dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac lp tpm_tis tpm i2c_piix4 floppy parport_pc tpm_bios serio_raw ide_cd shpchp i2c_core cdrom e1000 sg parport pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 3283, comm: vx_worklist_thr Tainted: PF    --------------------    2.6.18-348.30.1.el5 #1
RIP: 0010:[<ffffffff884b43d7>]  [<ffffffff884b43d7>] :vxfs:vx_ilist_chunkclean+0x170/0x605
RSP: 0000:ffff810239d05df0  EFLAGS: 00010086
RAX: ffff81009a3e1758 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff81009a3e1578 RDI: 0000000000000000
RBP: 0000000000000002 R08: ffff81009a3e1000 R09: 0000000000000001
R10: 0000000000010be1 R11: 0000000000000000 R12: ffff8102392927f8
R13: 0000000000000000 R14: ffff810239292850 R15: 0000000000000258
FS:  00002aec7236c6e0(0000) GS:ffff810107f410c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000298 CR3: 0000000193a09000 CR4: 00000000000006a0
Process vx_worklist_thr (pid: 3283, threadinfo ffff810239d04000, task ffff81023aa4e7f0)
Stack:  0000000000000001 00000002efdae000 000004b000000000 00025f7100000000
 0000000100000001 0000000000000293 0000000100200200 ffff81004e751000
 0000000000000002 0000000000000019 0000000000000005 ffff81004e751000
Call Trace:
 [<ffffffff884b624b>] :vxfs:vx_inode_free_list+0x5e/0x1f0
 [<ffffffff884a3337>] :vxfs:vx_ifree_scan_list+0x6f/0x7b
 [<ffffffff884a2320>] :vxfs:vx_workitem_process+0xb/0x1b
 [<ffffffff884a24f7>] :vxfs:vx_worklist_process+0x1c7/0x2a8
 [<ffffffff884a5e51>] :vxfs:vx_worklist_thread+0x0/0x98
 [<ffffffff884a5eab>] :vxfs:vx_worklist_thread+0x5a/0x98
 [<ffffffff884f291d>] :vxfs:vx_kthread_init+0x69/0x74
 [<ffffffff884a5e51>] :vxfs:vx_worklist_thread+0x0/0x98
 [<ffffffff8005dfc1>] child_rip+0xa/0x11
 [<ffffffff884a5e51>] :vxfs:vx_worklist_thread+0x0/0x98
 [<ffffffff884f28b4>] :vxfs:vx_kthread_init+0x0/0x74
 [<ffffffff8005dfb7>] child_rip+0x0/0x11


Code: 4c 39 a1 98 02 00 00 75 0a 8b 44 24 1c 89 81 7c 01 00 00 45 
RIP  [<ffffffff884b43d7>] :vxfs:vx_ilist_chunkclean+0x170/0x605
 RSP <ffff810239d05df0>

      KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/2.6.18-348.30.1.el5/vmlinux
LOAD AVERAGE: 1.09, 1.06, 1.02
     MACHINE: x86_64  (2264 Mhz)
       PANIC: "Unable to handle kernel NULL pointer dereference at 0000000000000298 RIP: "
     COMMAND: "vx_worklist_thr"
       STATE: TASK_RUNNING (PANIC)

Resolution

  • System crashed in vx_ilist_chunkclean function in vxfs module which is shipped by Veritas.
  • Please contact a Veritas to find out root cause and a possible solution.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.