RHEL 6.3 System Hangs

Solution Verified - Updated -

Issue

Another rhel63 server hang up, with the following message showing up in dmesg output:

It's networking is still working, we could ssh from this hang-up server to another server, but we could not ssh into this hanging server.

Some NFS mount hang, nslookup not working, ldap not working, initially we could not create an sosreport on this server, after waiting for quite some time, we created an sosreport "sosreport-lx-chopt118a-20120718105258-a92b.tar.xz", which will be uploaded shortly. The server is still up, please let us know what you need to gather before we reboot this server.

[root@lx-chopt118a ~]# uname -r 
2.6.32-279.0.1.el6.x86_64 

dmesg ouput

[46559.744299] ip_tables: (C) 2000-2006 Netfilter Core Team
[174395.495832] TVSHMPublisher.[20178]: segfault at 0 ip (null) sp 00007fd853ffeb78 error 14
[174395.537100] ------------[ cut here ]------------
[174395.537107] WARNING: at kernel/softirq.c:159 local_bh_enable_ip+0x7d/0xb0() (Not tainted)
[174395.537109] Hardware name: ProLiant BL460c G7
[174395.537110] Modules linked in: iptable_filter ip_tables nfs lockd fscache nfs_acl auth_rpcgss netconsole configfs autofs4 sunrpc bonding 8021q garp stp llc ipv6 power_meter be2net microcode serio_raw iTCO_wdt iTCO_vendor_support hpilo sg hpwdt i7core_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif hpsa dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
[174395.537134] Pid: 4827, comm: OMVolServer.BAK Not tainted 2.6.32-279.0.1.el6.x86_64 #1
[174395.537136] Call Trace:
[174395.537143]  [<ffffffff8106b747>] ? warn_slowpath_common+0x87/0xc0
[174395.537146]  [<ffffffff8106b79a>] ? warn_slowpath_null+0x1a/0x20
[174395.537149]  [<ffffffff810747ed>] ? local_bh_enable_ip+0x7d/0xb0
[174395.537154]  [<ffffffff8150041b>] ? _spin_unlock_bh+0x1b/0x20
[174395.537161]  [<ffffffffa00ee974>] ? be_process_mcc+0x74/0x230 [be2net]
[174395.537165]  [<ffffffffa00ea68c>] ? be_poll_tx_mcc+0x16c/0x290 [be2net]
[174395.537169]  [<ffffffff8144fe76>] ? netpoll_poll_dev+0xd6/0x490
[174395.537177]  [<ffffffffa01ce4a5>] ? bond_poll_controller+0x75/0x80 [bonding]
[174395.537180]  [<ffffffff8144fde5>] ? netpoll_poll_dev+0x45/0x490
[174395.537183]  [<ffffffff81450437>] ? netpoll_send_skb_on_dev+0x157/0x240
[174395.537189]  [<ffffffffa01cf8c4>] ? bond_dev_queue_xmit+0x1f4/0x210 [bonding]
[174395.537194]  [<ffffffffa01cfbe4>] ? bond_start_xmit+0x304/0x5f0 [bonding]
[174395.537199]  [<ffffffff8143b1ec>] ? dev_hard_start_xmit+0x2bc/0x3f0
[174395.537203]  [<ffffffff8143fa36>] ? dev_queue_xmit+0x596/0x6f0
[174395.537207]  [<ffffffff8147609c>] ? ip_finish_output2+0xdc/0x2a0
[174395.537210]  [<ffffffff814759ed>] ? ip_fragment+0x27d/0x850
[174395.537213]  [<ffffffff81475fc0>] ? ip_finish_output2+0x0/0x2a0
[174395.537217]  [<ffffffff814772ef>] ? ip_finish_output+0x21f/0x310
[174395.537220]  [<ffffffff81477688>] ? ip_mc_output+0x1e8/0x220
[174395.537223]  [<ffffffff81476795>] ? ip_local_out+0x25/0x30
[174395.537225]  [<ffffffff814767bb>] ? ip_send_skb+0x1b/0x80
[174395.537229]  [<ffffffff81498b69>] ? udp_send_skb+0x119/0x3d0
[174395.537232]  [<ffffffff81498a51>] ? udp_send_skb+0x1/0x3d0
[174395.537235]  [<ffffffff81475500>] ? ip_generic_getfrag+0x0/0xb0
[174395.537239]  [<ffffffff8149971d>] ? udp_sendmsg+0x2ed/0x8f0
[174395.537242]  [<ffffffff814a1bda>] ? inet_sendmsg+0x4a/0xb0
[174395.537246]  [<ffffffff81428123>] ? sock_sendmsg+0x123/0x150
[174395.537249]  [<ffffffff810629d3>] ? dequeue_entity+0x113/0x2e0
[174395.537254]  [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40
[174395.537258]  [<ffffffff8100988e>] ? __switch_to+0x26e/0x320
[174395.537264]  [<ffffffff8105b483>] ? perf_event_task_sched_out+0x33/0x80
[174395.537267]  [<ffffffff81096481>] ? lock_hrtimer_base+0x31/0x60
[174395.537270]  [<ffffffff814285f9>] ? sys_sendto+0x139/0x190
[174395.537273]  [<ffffffff81096230>] ? hrtimer_wakeup+0x0/0x30
[174395.537278]  [<ffffffff810d6b02>] ? audit_syscall_entry+0x272/0x2a0
[174395.537281]  [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
[174395.537284] ---[ end trace 80ea71f1b4570e7e ]---
[174397.002678]  in TVSHMPublisher.BAK[400000+bb4000]

Environment

Red Hat Enterprise Linux

6.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.