System kernel panic, appears "kernel BUG at /SFHA_build/fsbuild/6.0.1/6.0.500.100/2014-08-27_23.00.19/linux/vxfs/non-debug/rhel6/x86/kernel/smp/vx_strategy.c:1481!"
Issue
- System crashed inside vxfs
vx_dev_strategy
with kernel BUG statement similar to the following
<2>kernel BUG at /SFHA_build/fsbuild/6.0.1/6.0.500.200/2014-12-04_16.07.02/linux/vxfs/non-debug/rhel6/x86/kernel/smp_u6/vx_strategy.c:1481!
<4>invalid opcode: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/pci0000:60/0000:60:05.4/irq
<4>CPU 54
<4>Modules linked in: dm_snapshot dm_bufio bridge mptctl mptbase vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs iTCO_wdt iTCO_vendor_support microcode serio_raw hpilo hpwdt ixgbe ptp pps_core mdio lpc_ich mfd_core ioatdma dca sg power_meter acpi_ipmi ipmi_si ipmi_msghandler shpchp ext4 jbd2 mbcache dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) sd_mod hpsa lpfc scsi_transport_fc scsi_tgt crc_t10dif dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 19359, comm: multitool Tainted: P --------------- 2.6.32-504.8.1.el6.x86_64 #1 HP ProLiant DL560 Gen8
<4>RIP: 0010:[<ffffffffa08c094e>] [<ffffffffa08c094e>] vx_dev_strategy+0x68e/0x6f0 [vxfs]
<4>RSP: 0000:ffff88072dfe0798 EFLAGS: 00010297
<4>RAX: ffff88072dfe0000 RBX: 0000000000000600 RCX: ffff8801e6e52aa0
<4>RDX: ffff884040018e68 RSI: ffff88400f6ecae0 RDI: ffff886012ab7bc0
<4>RBP: ffff88072dfe0888 R08: 000000000008f664 R09: 000000000c70afc8
<4>R10: 000000000000001b R11: 0000000000003000 R12: 0000000000000000
<4>R13: 0000000000000000 R14: ffffea000c0c67a8 R15: 0000000000000000
<4>FS: 00007f9a0b94e720(0000) GS:ffff8840b0dc0000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 00007f9a0b536000 CR3: 000000600ee32000 CR4: 00000000000407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process multitool (pid: 19359, threadinfo ffff88072dfe0000, task ffff8801e6e52aa0)
<4>Stack:
<4> ffff88072dfe07a8 0000000000001609 ffff880371420000 0000000000000000
<4><d> 0000000000000001 0000000000000000 00000001ffffffff ffff8801e6e52aa0
<4><d> ffff8801e6e52aa0 ffff88200c013a00 ffff8801e6e52aa0 0000000000000000
<4>Call Trace:
<4> [<ffffffffa08c0f05>] vx_snap_strategy+0x295/0x2d0 [vxfs]
<4> [<ffffffffa08356fd>] vx_logbuf_write+0x17d/0x1b0 [vxfs]
<4> [<ffffffffa083590f>] vx_logbuf_io+0x1df/0x340 [vxfs]
<4> [<ffffffffa08366ae>] vx_logbuf_flush+0x22e/0x2d0 [vxfs]
<4> [<ffffffffa0836832>] vx_logflush+0xe2/0x140 [vxfs]
- Also found below similar kind of trace
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
IP: [<ffffffffa050cde5>] vx_dev_strategy+0x495/0x7c0 [vxfs]
PGD 15f146067 PUD faa586067 PMD 0
Thread overran stack, or stack corrupted
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:80/0000:80:01.0/0000:81:00.1/temp1_input
CPU 11
[..]
Pid: 42600, comm: java Tainted: P --------------- 2.6.32-504.el6.x86_64 #1 HP ProLiant DL380 Gen9/ProLiant DL380 Gen9
RIP: 0010:[<ffffffffa050cde5>] [<ffffffffa050cde5>] vx_dev_strategy+0x495/0x7c0 [vxfs]
[..]
Call Trace:
[<ffffffffa050d67d>] vx_snap_strategy+0x2ed/0x330 [vxfs]
[<ffffffffa04f85bc>] vx_io_startnowait+0x10c/0x3c0 [vxfs]
[<ffffffffa0403b66>] ? vx_io_setup+0x56/0xe0 [vxfs]
[<ffffffffa04fce4b>] vx_nalloc_getpage_lnx+0xa8b/0x14b0 [vxfs]
[<ffffffff81146d5e>] ? __inc_zone_page_state+0x2e/0x30
[<ffffffffa0c98e00>] ? vxg_range_lockwf_fastsub+0x60/0x140 [vxglm]
[<ffffffffa0407a59>] vx_do_getpage+0x1c19/0x2500 [vxfs]
[<ffffffffa0c85c00>] ? vxg_lock_fastsub+0x0/0xe0 [vxglm]
[<ffffffffa030f4f8>] ? vx_sum_upd_efree1+0x88/0xa0 [vxfs]
[<ffffffffa04ad400>] ? vx_alloc+0x8b0/0xd10 [vxfs]
[<ffffffffa051108e>] ? vx_subtranalloc+0xde/0x270 [vxfs]
[<ffffffffa04d7208>] vx_do_read_ahead+0x1a8/0x220 [vxfs]
[<ffffffffa0453aca>] vx_read_ahead+0x3ca/0xed0 [vxfs]
[<ffffffffa0409505>] ? vx_glm_range_lock+0x25/0x50 [vxfs]
[<ffffffffa0403543>] ? vx_glmrange_rangelock+0x33/0x90 [vxfs]
[<ffffffffa035d581>] ? vx_ipglock+0x31/0x40 [vxfs]
[<ffffffffa04052d2>] ? vx_getpage_rangelock+0x142/0x150 [vxfs]
[<ffffffffa04067a1>] vx_do_getpage+0x961/0x2500 [vxfs]
[<ffffffffa04036f1>] ? vx_genglm_lock+0x31/0x40 [vxfs]
[<ffffffffa0435400>] ? vx_iglock3+0x70/0x110 [vxfs]
[<ffffffff8100988e>] ? __switch_to+0x26e/0x320
[<ffffffffa0408741>] vx_getpage1+0x401/0x940 [vxfs]
[<ffffffff8117d170>] ? mem_cgroup_get_reclaim_stat_from_page+0x20/0x70
[<ffffffffa04b1651>] vx_fault+0x2c1/0x6c0 [vxfs]
[<ffffffff811807d7>] ? mem_cgroup_update_file_mapped+0x17/0x90
[<ffffffff8114eae4>] __do_fault+0x54/0x530
[<ffffffff8114f0b7>] handle_pte_fault+0xf7/0xb00
[<ffffffff81182028>] ? mem_cgroup_cache_charge+0xb8/0xc0
[<ffffffffa0c98ef2>] ? vxg_range_lock_fastsub+0x12/0x40 [vxglm]
[<ffffffffa0c98fcc>] ? vxg_range_unlock_fastsub+0x2c/0x40 [vxglm]
[<ffffffffa0c97ded>] ? vxg_lock_ilock_omnibus+0x3d/0x2a0 [vxglm]
[<ffffffff8114fcea>] handle_mm_fault+0x22a/0x300
[<ffffffff8114feea>] __get_user_pages+0x12a/0x430
[<ffffffffa0c88064>] ? vxg_api_lock+0x54/0xa0 [vxglm]
[<ffffffff81150289>] get_user_pages+0x49/0x50
[<ffffffffa04a5ab1>] vx_prefault_uio+0x1c1/0x2c0 [vxfs]
[<ffffffffa0409000>] ? vx_glm_trylock+0x30/0x80 [vxfs]
[<ffffffffa0539572>] vx_write_default+0x3e2/0x1610 [vxfs]
[<ffffffffa05343a9>] ? vx_write_alloc+0x409/0x6e0 [vxfs]
[<ffffffffa053ae41>] vx_write1+0x6a1/0x1940 [vxfs]
[<ffffffffa0c88064>] ? vxg_api_lock+0x54/0xa0 [vxglm]
[<ffffffffa0c85c00>] ? vxg_lock_fastsub+0x0/0xe0 [vxglm]
[<ffffffffa0c90001>] ? vxg_recv_release+0x61/0x160 [vxglm]
[<ffffffffa053c41f>] vx_write_common_slow+0x33f/0xfd0 [vxfs]
[<ffffffffa053d887>] vx_write_common+0x5f7/0x9f0 [vxfs]
[<ffffffff8152d375>] ? page_fault+0x25/0x30
[<ffffffffa04a5248>] ? vx_prefault_uio_readable+0x98/0xf0 [vxfs]
[<ffffffffa04b9c7a>] vx_write+0x20a/0x370 [vxfs]
[<ffffffff81156225>] ? do_mmap_pgoff+0x335/0x380
[<ffffffff8118e058>] vfs_write+0xb8/0x1a0
[<ffffffff8118eae2>] sys_pwrite64+0x82/0xa0
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: b3 cb e0 48 8b 95 48 ff ff ff 48 8b 42 08 44 8b 78 1c 45 85 ff 0f 88 dd 02 00 00 81 78 18 00 10 00 00 0f 87 c8 02 00 00 48 8b 10 <48> 3b 42 08 0f 85 bf 02 00 00 4d 85 e4 0f 85 10 fd ff ff 44 8b
RIP [<ffffffffa050cde5>] vx_dev_strategy+0x495/0x7c0 [vxfs]
RSP <ffff881a7358e7d8>
CR2: 0000000000000008
Environment
- Red Hat Enterprise Linux 6
- seen on various kernels including 2.6.32-220.el6 and 2.6.32-504.8.1.el6,
- vxfs
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.