Kernel panic - not syncing: GAB: Port d halting system due to network failure at [XX:XXXX]

Issue

• Red Hat Enterprise Linux running Veritas Cluster Server (VCS) crashed with following messages.

LLT INFO V-14-1-10205 link 0 (eth1) node 0 in trouble LLT INFO V-14-1-10205 link 1 (eth3) node 0 in trouble LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 8 sec (696686965) LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 8 sec (696614993) LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 9 sec (696686965) LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 9 sec (696614993) LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 10 sec (696686965) LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 10 sec (696614993) LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 11 sec (696686965) LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 11 sec (696614993) LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 12 sec (696686965) LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 12 sec (696614993) LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 13 sec (696686965) LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth1) node 0. 4 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth1) node 0. 3 more to go.
LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 13 sec (696614993) LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth1) node 0. 2 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (eth3) node 0. 4 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth1) node 0. 1 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (eth3) node 0. 3 more to go.
LLT INFO V-14-1-10032 link 0 (eth1) node 0 inactive 14 sec (696686965) LLT INFO V-14-1-10510 sent hbreq (NULL) on link 0 (eth1) node 0. 0 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (eth3) node 0. 2 more to go.
LLT INFO V-14-1-10509 link 0 (eth1) node 0 expired LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (eth3) node 0. 1 more to go.
LLT INFO V-14-1-10510 sent hbreq (NULL) on link 1 (eth3) node 0. 0 more to go.
LLT INFO V-14-1-10032 link 1 (eth3) node 0 inactive 14 sec (696614993) LLT INFO V-14-1-10509 link 1 (eth3) node 0 expired GAB INFO V-15-1-20239 Initiating FFDC data collection
GAB INFO V-15-1-20036 Port h[GAB_USER_CLIENT (refcount 0)] gen   263813 membership ;1
GAB INFO V-15-1-20036 Port u[GAB_USER_CLIENT (refcount 0)] gen   263814 membership ;1
GAB INFO V-15-1-20036 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen   26380f membership ;1
GAB INFO V-15-1-20036 Port f[GAB_LEGACY_CLIENT (refcount 0)] gen   263816 membership ;1
GAB INFO V-15-1-20036 Port w[GAB_USER_CLIENT (refcount 0)] gen   263811 membership ;1
GAB INFO V-15-1-20036 Port d[GAB_LEGACY_CLIENT (refcount 0)] gen   26380f membership ;1
GAB INFO V-15-1-20036 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   26380e membership ;1
GAB INFO V-15-1-20036 Port a[GAB_Control (refcount 1)] gen   26380e membership ;1
GAB INFO V-15-1-20036 Port b[VxFen (refcount 2)] gen   26380c membership ;1
LLT INFO V-14-1-10024 link 0 (eth1) node 0 active LLT INFO V-14-1-10024 link 1 (eth3) node 0 active
GAB INFO V-15-1-20036 Port w[GAB_USER_CLIENT (refcount 0)] gen   263811 membership ;1
GAB INFO V-15-1-20038 Port w[GAB_USER_CLIENT (refcount 0)] gen   263811 k_jeopardy 0
GAB INFO V-15-1-20040 Port w[GAB_USER_CLIENT (refcount 0)] gen   263811    visible 0
GAB INFO V-15-1-20036 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen   26380f membership ;1
GAB INFO V-15-1-20038 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen   26380f k_jeopardy 0
GAB INFO V-15-1-20040 Port v[GAB_LEGACY_CLIENT (refcount 0)] gen   26380f    visible 0
GAB INFO V-15-1-20036 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   26380e membership ;1
GAB INFO V-15-1-20038 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   26380e k_jeopardy 0
GAB INFO V-15-1-20040 Port y[GAB_LEGACY_CLIENT (refcount 0)] gen   26380e    visible 0
GAB INFO V-15-1-20036 Port b[VxFen (refcount 2)] gen   26380c membership ;1
GAB INFO V-15-1-20038 Port b[VxFen (refcount 2)] gen   26380c k_jeopardy 0
GAB INFO V-15-1-20040 Port b[VxFen (refcount 2)] gen   26380c    visible 0
GAB INFO V-15-1-20036 Port u[GAB_USER_CLIENT (refcount 0)] gen   263814 membership ;1
GAB INFO V-15-1-20038 Port u[GAB_USER_CLIENT (refcount 0)] gen   263814 k_jeopardy 0
GAB INFO V-15-1-20040 Port u[GAB_USER_CLIENT (refcount 0)] gen   263814    visible 0
GAB INFO V-15-1-20036 Port f[GAB_LEGACY_CLIENT (refcount 0)] gen   263816 membership ;1
GAB INFO V-15-1-20038 Port f[GAB_LEGACY_CLIENT (refcount 0)] gen   263816 k_jeopardy 0
GAB INFO V-15-1-20040 Port f[GAB_LEGACY_CLIENT (refcount 0)] gen   263816    visible 0
GAB INFO V-15-1-20036 Port h[GAB_USER_CLIENT (refcount 0)] gen   263813 membership ;1
GAB INFO V-15-1-20038 Port h[GAB_USER_CLIENT (refcount 0)] gen   263813 k_jeopardy 0
GAB INFO V-15-1-20040 Port h[GAB_USER_CLIENT (refcount 0)] gen   263813    visible 0
GAB INFO V-15-1-20033 Port d[GAB_LEGACY_CLIENT (refcount 0)] nid 0     [3:0:0:0]    [3:0:0:0]    [3:0:0:0]    [1:0:0:0]    [0:0:0:0]  22   26380f   20 9
GAB INFO V-15-1-20033 Port d[GAB_LEGACY_CLIENT (refcount 0)] nid 1     [3:0:0:0]    [3:0:0:0]    [3:0:0:0]    [2:0:0:0]    [0:0:0:0]  22   26380f   20 37
Kernel panic - not syncing: GAB: Port d halting system due to network failure at [14:2166] [..]