EAP 6 EJB connections hanging for a long time if the remote server fails

Issue

• If a cluster member is no longer available, it takes a long time until the caller invocation times out
  • When a node was killed (power off) we can see that there are still established TCP connections to the dead instance via the remoting port, 4447.
  • An EJB invocation will hung about 15min, no matter whether the cluster-view has changed due to the node failure
  • There are many threads waiting on connections if a remote instance failed

"http-/127.0.0.1:8080-1" daemon prio=10 tid=0x00007f0520071800 nid=0x2286 in Object.wait() [0x00007f059f58c000]
   java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on <0x00000000e83ccf38> (a java.lang.Object)
  at java.lang.Object.wait(Object.java:503)
  at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:397)
  - locked <0x00000000e83ccf38> (a java.lang.Object)
  at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:140)
  at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)
  at com.sun.proxy.$Proxy31.invoke(Unknown Source)

• We have JBoss EAP 6.1 in standalone mode calling to a EJB remoting cluster in JBoss EAP 6.2 domain. The first machine is in a DMZ and the firewall is closed the connection to EJB remotoing after a timeout of inactivity. We know that there is an option jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL and KEEP_ALIVE but we dont know how to use it. So we tried to put in the file jboss-ejb-client.xml but we have this error:

ERROR [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000001: Failed to start service jboss.deployment.unit."test.war".STRUCTURE: org.jboss.msc.service.StartException in service jboss.deployment.unit."test.war".STRUCTURE: JBAS018733: Failed to process phase STRUCTURE of deployment "test.war"
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:127) [jboss-as-server-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: JBAS011096: Exception while parsing jboss-ejb-client.xml file found at /jboss/eap/standalone/deployments/test.war/WEB-INF/jboss-ejb-client.xml
        at org.jboss.as.ee.structure.EJBClientDescriptorParsingProcessor.parse(EJBClientDescriptorParsingProcessor.java:147)
        at org.jboss.as.ee.structure.EJBClientDescriptorParsingProcessor.parse(EJBClientDescriptorParsingProcessor.java:125)
        at org.jboss.as.ee.structure.EJBClientDescriptorParsingProcessor.deploy(EJBClientDescriptorParsingProcessor.java:107)
        at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:120) [jboss-as-server-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8]
        ... 5 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[9,10]
Message: JBAS011094: Unexpected element '{urn:jboss:ejb-client:1.0}clusters' encountered
        at org.jboss.as.ee.structure.EJBClientDescriptor10Parser.unexpectedElement(EJBClientDescriptor10Parser.java:275)
        at org.jboss.as.ee.structure.EJBClientDescriptor10Parser.parseClientContext(EJBClientDescriptor10Parser.java:171)
        at org.jboss.as.ee.structure.EJBClientDescriptor10Parser.readElement(EJBClientDescriptor10Parser.java:138)
        at org.jboss.as.ee.structure.EJBClientDescriptor10Parser.readElement(EJBClientDescriptor10Parser.java:60)
        at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110)
        at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69)
        at org.jboss.as.ee.structure.EJBClientDescriptorParsingProcessor.parse(EJBClientDescriptorParsingProcessor.java:141)
        ... 8 more