HQ222024: Could not complete operations on IO context OperationContextImpl

Solution Verified - Updated -

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 6.x

Issue

The following exception can be seen in the server log file:

2014-03-25 16:54:28,640 WARN  [org.hornetq.core.server] (Thread-26 (HornetQ-server-HornetQServerImpl::serverUUID=f079f93b-af65-11e3-a09f-592001d35ed1-1329633628)) HQ222024: Could not complete operations on IO context OperationContextImpl [2037424067] [minimalStore=0, storeLineUp=0, stored=0, minimalReplicated=1, replicationLineUp=1, replicated=0, paged=0, minimalPage=0, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=java.util.concurrent.Executors$FinalizableDelegatedExecutorService@5108f97e]Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
  • We are setting up a 2 node cluster each running as standalone nodes. However we're experiencing following error :
HQ214016: Failed to create netty connection: java.net.SocketTimeoutException: connect timed out

How to solve this? We're running both nodes on Windows Server 2008R2 x64 within same subnet.

Resolution

Fix the network.

Root Cause

  • In this particular case the issue was caused by network misconfiguration. There was a firewall between to cluster nodes that was preventing cluster from formation.

  • The message HQ222024 was caused by the fact that the system could not keep up with executor creation rate. As soon as one was created it was used up by netty trying to establish a connection, activity that was continuously failing.

Diagnostic Steps

The issue can be recognised by the following message signiture:

2014-03-25 17:02:55,960 WARN  [org.hornetq.core.server] (Thread-26 (HornetQ-server-HornetQServerImpl::serverUUID=f079f93b-af65-11e3-a09f-592001d35ed1-1329633628)) HQ222024: Could not complete operations on IO context OperationContextImpl [2037424067] [minimalStore=0, storeLineUp=0, stored=0, minimalReplicated=1, replicationLineUp=1, replicated=0, paged=0, minimalPage=0, pageLineUp=0, errorCode=-1, errorMessage=null, executorsPending=0, executor=java.util.concurrent.Executors$FinalizableDelegatedExecutorService@5108f97e]Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=IOAsyncTask(PageSubscriptionImpl) ]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0,  task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]
Task = TaskHolder [storeLined=0, replicationLined=1, pageLined=0, task=org.hornetq.core.journal.impl.SimpleWaitIOCallback]

. . .

2014-03-25 17:02:59,127 ERROR [org.hornetq.core.client] (Thread-19 (HornetQ-server-HornetQServerImpl::serverUUID=f079f93b-af65-11e3-a09f-592001d35ed1-1329633628)) HQ214016: Failed to create netty connection: java.net.SocketTimeoutException: connect timed out
    at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51]
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) [rt.jar:1.7.0_51]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51]
    at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:109) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:69) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.jboss.netty.channel.Channels.connect(Channels.java:634) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:207) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [netty-3.6.7.Final-redhat-1.jar:3.6.7.Final-redhat-1]
    at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:624) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1223) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1069) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1998) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1824) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:667) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:657) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:635) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:107) [hornetq-core-client-2.3.14.Final-redhat-1.jar:2.3.14.Final-redhat-1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.