“HQ214016: Failed to create netty connection:java.net.NoRouteToHostException: No route to host” repeated in the server.log

Solution Verified - Updated -

Issue

  • On a clusterd HornetQ messaging environment, I'm seeing “HQ214016: Failed to create netty connection:java.net.NoRouteToHostException: No route to host” ERROR in the server log repeatedly.
    Here's the full stack trace :

  • We are getting below No route to host error on our Prod Jboss cluster.

ERROR [org.hornetq.core.client] (Thread-5 (HornetQ-server-HornetQServerImpl::serverUUID=ddd256a8-412d-11e3-a26b-29c23655ddca-947242391)) HQ214016: Failed to create netty connection: java.net.NoRouteToHostException: No route to host
    at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) [rt.jar:1.6.0_20]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) [rt.jar:1.6.0_20]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) [rt.jar:1.6.0_20]
    at java.net.Socket.connect(Socket.java:546) [rt.jar:1.6.0_20]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:105) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:65) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.Channels.connect(Channels.java:634) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:207) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [netty-3.6.2.Final-redhat-1.jar:3.6.2.Final-redhat-1]
    at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:628) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1223) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1069) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:828) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:616) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:600) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:578) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_20]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_20]
    at java.lang.Thread.run(Thread.java:636) [rt.jar:1.6.0_20]
  • I noticed that this issue started just after a server reboo. I further verified the connectivity and found its OK when we telnet on Jboss specific ports between the nodes.

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6
    • 7
  • Red Hat JBoss Fuse Service Works (FSW)
    • 6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.