Getting SocketTimeoutException error when HotRod Java client trying to reconnect on Red Hat JBoss Data Grid
Environment
- Red Hat Data Grid
- 7.3.x
- Hot Rod Java client
Issue
- Getting timed out when JDG client tries to reconnect on JDG server
- After stopping JDG server node, Hot Rod Java client gets blocked and launches a
SocketTimeoutException
client exception - Hot Rod client is not retrying when a server goes down launching
Connection reset by peer
Resolution
- To solve this issue update the Hot Rod Java client to 7.3.5 version as described in the
Fix Version/s:
on JIRA JDG-3357 - Set the
socket_timeout
,connect_timeout
,max_retries
in the client-side configuration.
This can be done one thehotrod-application.properties
file (orhotrod-client.properties
), as below:
infinispan.client.hotrod.socket_timeout = 5000
infinispan.client.hotrod.connect_timeout = 5000
infinispan.client.hotrod.max_retries = 5
Or programmatically:
.maxRetries(5)
.socketTimeout(80000)
.connectionTimeout(80000)
The values should be less than the max-idle
defined in the clustered.xml
file.
Root Cause
The Hot Rod client registers a default timeout for operations using the socket timeout (60000
milliseconds by default). However, when this timeout is reached, the retry logic is not invoked causing requests to die and servers failing immediately without retry.
This issue was tracked by the following JIRA:
- Upstream community JIRA: ISPN-10429
- Internal product JIRA: JDG-3357
Diagnostic Steps
Search for SocketTimeoutException
error in the Hot Rod Java client log as follow:
Exception in thread "main" org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException: GetOperation{api-general-filestore, key=[B0x033E03323531, flags=0} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.Util.rewrap(Util.java:54)
at org.infinispan.client.hotrod.impl.Util.await(Util.java:27)
at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:418)
at br.com.itau.jdg.JdgTest.main(JdgTest.java:19)
Caused by: java.net.SocketTimeoutException: GetOperation{api-general-filestore, key=[B0x033E03323531, flags=0} timed out after 60000 ms
at org.infinispan.client.hotrod.impl.operations.HotRodOperation.run(HotRodOperation.java:172)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:331)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments