Server hung due to high CPU utilization
Issue
- VCS cluster node was hung in the console due to high CPU utilization.
- Console messages during the hung:
BUG: soft lockup - CPU#4 stuck for 67s! [vxpath_links:29485]
Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl vxfen(P)(U) vxodm(P)(U) vxgms(P)(U) vxglm(P)(U) gab(P)(U) llt(P)(U) dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) autofs4 sunrpc vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs ext3 jbd dm_round_robin dm_multipath power_meter igb dca ptp pps_core sg microcode serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt bnx2 i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif lpfc scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxfen]
CPU 4
Modules linked in: bridge nfs lockd fscache auth_rpcgss nfs_acl vxfen(P)(U) vxodm(P)(U) vxgms(P)(U) vxglm(P)(U) gab(P)(U) llt(P)(U) dmpjbod(P)(U) dmpap(P)(U) dmpaa(P)(U) autofs4 sunrpc vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) cpufreq_ondemand freq_table pcc_cpufreq bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs ext3 jbd dm_round_robin dm_multipath power_meter igb dca ptp pps_core sg microcode serio_raw iTCO_wdt iTCO_vendor_support hpilo hpwdt bnx2 i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif lpfc scsi_transport_fc scsi_tgt hpsa radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vxfen]
Pid: 29485, comm: vxpath_links Tainted: P W --------------- 2.6.32-358.11.1.el6.x86_64 #1 HP ProLiant DL380 G7
RIP: 0010:[<ffffffff8151076e>] [<ffffffff8151076e>] lock_kernel+0x2e/0x50
RSP: 0018:ffff880548f99cb8 EFLAGS: 00000297
RAX: 0000000000000000 RBX: ffff880548f99cb8 RCX: 0000000000002392
RDX: 0000000000002398 RSI: ffff88054d272780 RDI: ffff8817faf69d48
RBP: ffffffff8100bb8e R08: ffff880542a97a80 R09: ffff882ffb5a0500
R10: 000000000000000a R11: 0000000000000246 R12: 0000000300000004
R13: ffff882ffba2ac80 R14: ffffffff81190220 R15: ffff880548f99c68
FS: 00007f25cfc98700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000047b4d0 CR3: 0000000542a95000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process vxpath_links (pid: 29485, threadinfo ffff880548f98000, task ffff88054d27caa0)
Stack:
ffff880548f99ce8 ffffffff81321f04 ffff882ffae7e940 ffff882ffae7e940
<d> ffff8817faf69d48 0000000000000000 ffff880548f99d48 ffffffff81185245
<d> ffff880548f99d58 ffffffff8119ac27 ffff88041fb53005 ffff88054d272780
Call Trace:
[<ffffffff81321f04>] ? memory_open+0x24/0xa0
[<ffffffff81185245>] ? chrdev_open+0x125/0x230
[<ffffffff8119ac27>] ? __d_lookup+0xa7/0x150
[<ffffffff81185120>] ? chrdev_open+0x0/0x230
[<ffffffff8117e20a>] ? __dentry_open+0x10a/0x360
[<ffffffff8121cdef>] ? security_inode_permission+0x1f/0x30
[<ffffffff8117e574>] ? nameidata_to_filp+0x54/0x70
[<ffffffff81192f70>] ? do_filp_open+0x6d0/0xdc0
[<ffffffff81510a05>] ? page_fault+0x25/0x30
[<ffffffff8119f622>] ? alloc_fd+0x92/0x160
[<ffffffff8117dfb9>] ? do_sys_open+0x69/0x140
[<ffffffff8117e0d0>] ? sys_open+0x20/0x30
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Code: 8b 04 25 c0 cb 00 00 8b 40 1c 48 89 e5 83 c0 01 75 2d ba 00 00 01 00 f0 0f c1 15 1e 3a 57 00 0f b7 ca c1 ea 10 39 d1 74 16 f3 90 <0f> b7 0d 0b 3a 57 00 eb f1 83 3d 02 3a 57 00 00 75 ec eb d3 65
Call Trace:
[<ffffffff81184871>] ? cdev_get+0x31/0xe0
[<ffffffff81321f04>] ? memory_open+0x24/0xa0
[<ffffffff81185245>] ? chrdev_open+0x125/0x230
[<ffffffff8119ac27>] ? __d_lookup+0xa7/0x150
[<ffffffff81185120>] ? chrdev_open+0x0/0x230
[<ffffffff8117e20a>] ? __dentry_open+0x10a/0x360
[<ffffffff8121cdef>] ? security_inode_permission+0x1f/0x30
[<ffffffff8117e574>] ? nameidata_to_filp+0x54/0x70
[<ffffffff81192f70>] ? do_filp_open+0x6d0/0xdc0
[<ffffffff81510a05>] ? page_fault+0x25/0x30
[<ffffffff8119f622>] ? alloc_fd+0x92/0x160
[<ffffffff8117dfb9>] ? do_sys_open+0x69/0x140
[<ffffffff8117e0d0>] ? sys_open+0x20/0x30
[<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 6.4
- Veritas cluster
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.