“HQ214016: Failed to create netty connection:java.net.NoRouteToHostException: No route to host” repeated in the server.log

Issue

• On a clusterd HornetQ messaging environment, I'm seeing “HQ214016: Failed to create netty connection:java.net.NoRouteToHostException: No route to host” ERROR in the server log repeatedly.
  Here's the full stack trace :

• We are getting below No route to host error on our Prod Jboss cluster.

ERROR [org.hornetq.core.client] (Thread-5 (HornetQ-server-HornetQServerImpl::serverUUID=ddd256a8-412d-11e3-a26b-29c23655ddca-947242391)) HQ214016: Failed to create netty connection: java.net.NoRouteToHostException: No route to host
    at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) [rt.jar:1.6.0_20]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) [rt.jar:1.6.0_20]
    at java.net.Socket.connect(Socket.java:546) [rt.jar:1.6.0_20]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:105) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:65) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.Channels.connect(Channels.java:634) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:207) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:628) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1223) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1069) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:828) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:616) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:600) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:578) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_20]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_20]
    at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]

• I noticed that this issue started just after a server reboo. I further verified the connectivity and found its OK when we telnet on Jboss specific ports between the nodes.