Server hung due to high CPU utilization
Issue
- VCS cluster node was hung in the console due to high CPU utilization.
- Console messages during the hung:
BUG: soft lockup - CPU#4 stuck for 67s! [vxpath_links:29485]
Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl vxfen(P)(U) vxodm(P)(U) vxgms(P)(U) vxglm(P)(U) gab(P)(U) llt(P)(U) dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) autofs4 sunrpc vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs ext3 jbd dm_round_robin dm_multipath power_meter igb dca ptp pps_core sg microcode serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt bnx2 i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif lpfc scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxfen]
CPU 4
Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl vxfen(P)(U) vxodm(P)(U) vxgms(P)(U) vxglm(P)(U) gab(P)(U) llt(P)(U) dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) autofs4 sunrpc vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs ext3 jbd dm_round_robin dm_multipath power_meter igb dca ptp pps_core sg microcode serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt bnx2 i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif lpfc scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxfen]
Pid: 29485, comm: vxpath_links Tainted: P W --------------- 2.6.32-358.11.1.el6.x86_64 #1 HP ProLiant DL380 G7
RIP: 0010:[<ffffffff8151076e>] [<ffffffff8151076e>] lock_kernel+0x2e/0x50
RSP: 0018:ffff880548f99cb8 EFLAGS: 00000297
RAX: 0000000000000000 RBX: ffff880548f99cb8 RCX: 0000000000002392
RDX: 0000000000002398 RSI: ffff88054d272780 RDI: ffff8817faf69d48
RBP: ffffffff8100bb8e R08: ffff880542a97a80 R09: ffff882ffb5a0500
R10: 000000000000000a R11: 0000000000000246 R12: 0000000300000004
R13: ffff882ffba2ac80 R14: ffffffff81190220 R15: ffff880548f99c68
FS: 00007f25cfc98700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000047b4d0 CR3: 0000000542a95000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vxpath_links (pid: 29485, threadinfo ffff880548f98000, task ffff88054d27caa0)
Stack:
ffff880548f99ce8 ffffffff81321f04 ffff882ffae7e940 ffff882ffae7e940
<d> ffff8817faf69d48 0000000000000000 ffff880548f99d48 ffffffff81185245
<d> ffff880548f99d58 ffffffff8119ac27 ffff88041fb53005 ffff88054d272780
Call Trace:
[<ffffffff81321f04>] ? memory_open+0x24/0xa0
[<ffffffff81185245>] ? chrdev_open+0x125/0x230
[<ffffffff8119ac27>] ? __d_lookup+0xa7/0x150
[<ffffffff81185120>] ? chrdev_open+0x0/0x230
[<ffffffff8117e20a>] ? __dentry_open+0x10a/0x360
[<ffffffff8121cdef>] ? security_inode_permission+0x1f/0x30
[<ffffffff8117e574>] ? nameidata_to_filp+0x54/0x70
[<ffffffff81192f70>] ? do_filp_open+0x6d0/0xdc0
[<ffffffff81510a05>] ? page_fault+0x25/0x30
[<ffffffff8119f622>] ? alloc_fd+0x92/0x160
[<ffffffff8117dfb9>] ? do_sys_open+0x69/0x140
[<ffffffff8117e0d0>] ? sys_open+0x20/0x30
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Code: 8b 04 25 c0 cb 00 00 8b 40 1c 48 89 e5 83 c0 01 75 2d ba 00 00 01 00 f0 0f c1 15 1e 3a 57 00 0f b7 ca c1 ea 10 39 d1 74 16 f3 90 <0f> b7 0d 0b 3a 57 00 eb f1 83 3d 02 3a 57 00 00 75 ec eb d3 65
Call Trace:
[<ffffffff81184871>] ? cdev_get+0x31/0xe0
[<ffffffff81321f04>] ? memory_open+0x24/0xa0
[<ffffffff81185245>] ? chrdev_open+0x125/0x230
[<ffffffff8119ac27>] ? __d_lookup+0xa7/0x150
[<ffffffff81185120>] ? chrdev_open+0x0/0x230
[<ffffffff8117e20a>] ? __dentry_open+0x10a/0x360
[<ffffffff8121cdef>] ? security_inode_permission+0x1f/0x30
[<ffffffff8117e574>] ? nameidata_to_filp+0x54/0x70
[<ffffffff81192f70>] ? do_filp_open+0x6d0/0xdc0
[<ffffffff81510a05>] ? page_fault+0x25/0x30
[<ffffffff8119f622>] ? alloc_fd+0x92/0x160
[<ffffffff8117dfb9>] ? do_sys_open+0x69/0x140
[<ffffffff8117e0d0>] ? sys_open+0x20/0x30
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 6.4
- Veritas cluster
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
