Using EAP 6.2 client jars to connect to AS 6.2 EAP fails with hang

Latest response

Hi all,

our customer is experiencing problems when connecting to remote ejb's under load (about 40 clients). After some time no client is able to connect to the server anymore and the following exception occurred:

java.lang.RuntimeException: Operation failed with status WAITING
  at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:94)
  at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:77)
  at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
  at org.jboss.ejb.client.remoting.MaxAttemptsReconnectHandler.tryConnect(MaxAttemptsReconnectHandler.java:63)
  at org.jboss.ejb.client.remoting.EJBClientContextConnectionReconnectHandler.reconnect(EJBClientContextConnectionReconnectHandler.java:59)
  at org.jboss.ejb.client.EJBClientContext$ReconnectAttempt.run(EJBClientContext.java:1301)

A thread dump of the server was taken right after the problem occurred and a second one an our later (no change).

It seems that the thread "sap-iwp-01t1" read-1" is waiting to lock the resource <0x00000000b007dfd0>, which is locked by the thread "sap-iwp-01t1" task-3":

    "Remoting "sap-iwp-01t1" read-1" prio=10 tid=0x00007f9b88022870 nid=0x4ab5 waiting for monitor entry [0x00007f9b245a4000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jboss.as.ejb3.deployment.DeploymentRepository.removeListener(DeploymentRepository.java:104)
    - waiting to lock <0x00000000b007dfd0> (a org.jboss.as.ejb3.deployment.DeploymentRepository)

    "Remoting "sap-iwp-01t1" task-3" prio=10 tid=0x00007f9a40052ab0 nid=0x4cd8 in Object.wait() [0x00007f9b277f6000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000ca35ff98> (a org.xnio.streams.BufferPipeOutputStream)
    at java.lang.Object.wait(Object.java:503)
    at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:106)
    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:126)
...
    at org.jboss.as.ejb3.deployment.DeploymentRepository.addListener(DeploymentRepository.java:99)
    - locked <0x00000000b007dfd0> (a org.jboss.as.ejb3.deployment.DeploymentRepository)
    at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.startReceiving(VersionOneProtocolChannelReceiver.java:97)

Our customer did not change the configuration for the remoting worker thread pool. As far as i know the following is the standard configuration:

<subsystem xmlns="urn:jboss:domain:remoting:1.1">
   <worker-thread-pool read-threads="1" task-core-threads="4" task-keepalive="60" task-limit="16384" task-max-threads="16" write-threads="1"/>
   ...
</subsystem>

The client is using the following code to connect to the server via the EJB client API:

    static synchronized void setContextSelector(final ClassLoader classLoader, final String appServerURL, final String appServerUser, final String appServerPassword) {
    Properties props = new Properties();
    props.put("endpoint.name", "client-endpoint");
    props.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
    props.put("remote.connections", "default");
    props.put("remote.connection.default.host", host);
    props.put("remote.connection.default.port", port);
    props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
    props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "true");
    props.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
    props.put("remote.connection.default.username", appServerUser);
    props.put("remote.connection.default.password", appServerPassword);

    final Class c = Class.forName("org.jboss.ejb.client.EJBClientContext", false, classLoader);
    final Method m = c.getMethod("setSelector", new Class[]{Class.forName("org.jboss.ejb.client.ContextSelector")});
    m.invoke(null, new Object[]{createContextSelector(classLoader, props)});
  }

  private static Object createContextSelector(final ClassLoader classLoader, final Properties props) throws ClassNotFoundException, 
  NoSuchMethodException, InstantiationException, IllegalAccessException, InvocationTargetException
  {
    final Class c1 = Class.forName("org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration", false, classLoader);
    final Object o = c1.getConstructor(new Class[]{Properties.class}).newInstance(new Object[]{props});
    final Class c2 = Class.forName("org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector", false, classLoader);
    return c2.getConstructor(new Class[]{Class.forName("org.jboss.ejb.client.EJBClientConfiguration")}).newInstance(new Object[]{o});
  }

In addition we use the remote-naming API to look up some jndi entries (Strings).

Another problem is that the firewall cuts off inactive connections. Maybe the configuration of an keep alive event could be a solution:

remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=
remote.connection.default.connect.options.org.xnio.Options.KEEP_ALIVE=true

The JBoss Version our customer is using is JBOSS 6.2 EAP (no patches applied).

What could be the reason for this error? Should our customer change the worker thread pool configuration?

EH

Responses