RHEL5: Kernel panic - not syncing: GAB: Port h halting system due to client process failure.

Issue

• Red Hat Enterprise Linux 5 running Veritas Cluster Server (VCS) crashed with following messages.

GAB WARNING V-15-1-20057 Port h process 14272 inactive 29 sec
GAB WARNING V-15-1-20058 Port h process 14272: heartbeat failed, killing process
GAB INFO V-15-1-20059 Port h heartbeat interval 30000 msec. Statistics:
GAB INFO V-15-1-20129 Port h: heartbeats in 0 ~ 6000 msec: 122753311
GAB INFO V-15-1-20129 Port h: heartbeats in 6000 ~ 12000 msec: 0
GAB INFO V-15-1-20129 Port h: heartbeats in 12000 ~ 18000 msec: 0
GAB INFO V-15-1-20129 Port h: heartbeats in 18000 ~ 24000 msec: 0
GAB INFO V-15-1-20129 Port h: heartbeats in 24000 ~ 30000 msec: 0
GAB INFO V-15-1-20088 System information:
GAB INFO V-15-1-20089   number of cpu:                 24
GAB INFO V-15-1-20090   physical memory:               98845604 K
GAB INFO V-15-1-20091   free memory:                   222368 K
GAB INFO V-15-1-20041 Port h: client process failure: killing process
GAB WARNING V-15-1-20035 Port h attempting to kill process due to client process failure
GAB WARNING V-15-1-20035 Port h attempting to kill process due to client process failure
GAB WARNING V-15-1-20035 Port h attempting to kill process due to client process failure
GAB WARNING V-15-1-20138 Port h isolated due to client process failure
Kernel panic - not syncing: GAB: Port h halting system due to client process failure at [14:1019]

• Server rebooted abruptly with no clue in /var/log/messages.