CLI multiple :start() fail after a deploy in EAP 6
Issue
-we are using a ruby script to start in parallel 15 JBoss instances on a domain configuration.
We have 1 domain controller and 15 host controller machines each with 1 JBoss instance. All instances are owned by a single server group.
The script creates 15 threads each calling jboss-cli.sh to launch at the same a :start(blocking=true) on each instance. We do not use the server group :start() operation because it's not synchronous.
The problem:
We found that after a deploy (managed), the ruby script fails because many threads "jboss-cli.sh" is failing with this exception:
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:280)
at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:246)
at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.jboss.modules.Module.run(Module.java:270)
at org.jboss.modules.Main.main(Main.java:411)
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at localhost:9999
at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:951)
at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:790)
at org.jboss.as.cli.impl.CommandContextImpl.connectController(CommandContextImpl.java:766)
at org.jboss.as.cli.impl.CliLauncher.initCommandContext(CliLauncher.java:278)
... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:129)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
at org.jboss.as.cli.impl.CommandContextImpl.tryConnection(CommandContextImpl.java:929)
... 11 more
Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:131)
at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256)
at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
at org.jboss.as.protocol.mgmt.FutureManagementChannel$Establishing.getChannel(FutureManagementChannel.java:176)
at org.jboss.as.cli.impl.CLIModelControllerClient.getOrCreateChannel(CLIModelControllerClient.java:155)
at org.jboss.as.cli.impl.CLIModelControllerClient$2.getChannel(CLIModelControllerClient.java:117)
at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:115)
at org.jboss.as.protocol.mgmt.ManagementChannelHandler.executeRequest(ManagementChannelHandler.java:90)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeRequest(AbstractModelControllerClient.java:236)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:141)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
... 13 more
We are guessing that DC, for some reason, is temporarily refusing connections from jboss-cli.sh instances.
Note that the DC does not need to be restarted: later, if we start the same ruby script (with same parallelism) the start finish successfully.
- is really DC limiting connections or is it a bug related to the deploy?
- is there a workaround to start all the 15 instances at the same time in a synchronized manner?
Environment
- Redhat Enterprise Application Platform (EAP) 6.1.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.