Using EAP 6.2 client jars to connect to AS 6.2 EAP fails with hang
Hi all,
our customer is experiencing problems when connecting to remote ejb's under load (about 40 clients). After some time no client is able to connect to the server anymore and the following exception occurred:
java.lang.RuntimeException: Operation failed with status WAITING at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:94) at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:77) at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51) at org.jboss.ejb.client.remoting.MaxAttemptsReconnectHandler.tryConnect(MaxAttemptsReconnectHandler.java:63) at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:59) at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1301)
A thread dump of the server was taken right after the problem occurred and a second one an our later (no change).
It seems that the thread "sap-iwp-01t1" read-1" is waiting to lock the resource 0x00000000b007dfd0>, which is locked by the thread "sap-iwp-01t1" task-3":
"Remoting "sap-iwp-01t1" read-1" prio=10 tid=0x00007f9b88022870 nid=0x4ab5 waiting for monitor entry [0x00007f9b245a4000] java.lang.Thread.State: BLOCKED (on object monitor) at org.jboss.as.ejb3.deployment.DeploymentRepository.removeListener(DeploymentRepository.java:104) - waiting to lock 0x00000000b007dfd0> (a org.jboss.as.ejb3.deployment.DeploymentRepository) "Remoting "sap-iwp-01t1" task-3" prio=10 tid=0x00007f9a40052ab0 nid=0x4cd8 in Object.wait() [0x00007f9b277f6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x00000000ca35ff98> (a org.xnio.streams.BufferPipeOutputStream) at java.lang.Object.wait(Object.java:503) at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:106) at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:126) ... at org.jboss.as.ejb3.deployment.DeploymentRepository.addListener(DeploymentRepository.java:99) - locked 0x00000000b007dfd0> (a org.jboss.as.ejb3.deployment.DeploymentRepository) at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.startReceiving(VersionOneProtocolChannelReceiver.java:97)
Our customer did not change the configuration for the remoting worker thread pool. As far as i know the following is the standard configuration:
...
The client is using the following code to connect to the server via the EJB client API:
static synchronized void setContextSelector(final ClassLoader classLoader, final String appServerURL, final String appServerUser, final String appServerPassword) { Properties props = new Properties(); props.put("endpoint.name", "client-endpoint"); props.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false"); props.put("remote.connections", "default"); props.put("remote.connection.default.host", host); props.put("remote.connection.default.port", port); props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER"); props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true"); props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false"); props.put("remote.connection.default.username", appServerUser); props.put("remote.connection.default.password", appServerPassword); final Class c = Class.forName("org.jboss.ejb.client.EJBClientContext", false, classLoader); final Method m = c.getMethod("setSelector", new Class[]{Class.forName("org.jboss.ejb.client.ContextSelector")}); m.invoke(null, new Object[]{createContextSelector(classLoader, props)}); } private static Object createContextSelector(final ClassLoader classLoader, final Properties props) throws ClassNotFoundException, NoSuchMethodException, InstantiationException, IllegalAccessException, InvocationTargetException { final Class c1 = Class.forName("org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration", false, classLoader); final Object o = c1.getConstructor(new Class[]{Properties.class}).newInstance(new Object[]{props}); final Class c2 = Class.forName("org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector", false, classLoader); return c2.getConstructor(new Class[]{Class.forName("org.jboss.ejb.client.EJBClientConfiguration")}).newInstance(new Object[]{o}); }
In addition we use the remote-naming API to look up some jndi entries (Strings).
Another problem is that the firewall cuts off inactive connections. Maybe the configuration of an keep alive event could be a solution:
remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=
remote.connection.default.connect.options.org.xnio.Options.KEEP_ALIVE=true
The JBoss Version our customer is using is JBOSS 6.2 EAP (no patches applied).
What could be the reason for this error? Should our customer change the worker thread pool configuration?
Responses