Intermittent connection timeouts when connecting to a clustered resource

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux Server 5 (with the High Availability or Resilient Storage Add Ons)
  • Red Hat Enterprise Linux Server 6 (with the High Availability or Resilient Storage Add Ons)

Issue

  • Users report timeout issues attempting to connect to a failover resource

  • Cluster service is going into recovery every 60 seconds with messages like the following in /var/log/messages

    node1 clurgmgrd: [11448]: <warning> Failed to ping 192.168.1.52 
node1 clurgmgrd[11448]: <notice> status on ip 192.168.1.52 returned 1 (generic error) 
node1 clurgmgrd[11448]: <notice> Stopping service service:myservice 
node1 clurgmgrd[11448]: <notice> Service service:myservice is recovering 
node1 clurgmgrd[11448]: <notice> Recovering failed service service:myservice 
node1 clurgmgrd[11448]: <notice> Service service:myservice started

Resolution

  • Alter iptables rules so that they no longer drop ICMP from the interface holding the ip resource.

Root Cause

Cluster performs a ping of ip resources every 60 seconds. A firewall rule was blocking those ping requests. When the ping fails, cluster attempts to recover the ip resource by stopping and starting it, resulting in connection timeout/losses on the client side. iptables was configured to drop all ICMP if the rate of pings was over a certain rate, so the issue was intermittent.

Diagnostic Steps

  • Check the firewall rules to see if there are any limiting rules on icmp:
# cat /etc/sysconfig/iptables |grep -i icmp
   -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
   -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT
   -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT
   -A INPUT -p icmp -m icmp --icmp-type 8 -m limit --limit 2/sec -j ACCEPT
   -A INPUT -p icmp -j firewall

In the example above the pings are limited to 2/sec which prevents the cluster from working correctly.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments