Server instance can't start after host-contoroller and server instance processes are killed at once in EAP 6 domain mode

Solution Verified - Updated -

Issue

  • When host-contoroller and server instance processes are killed at once in domain mode, the host-controller process is auto-restarted successfully by process-controller but the server instance process is not auto-restarted with the following ERROR in process-controller.log:
INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012010: Process 'Host Controller' finished with an exit status of 137
INFO  [org.jboss.as.process-controller.server] (reaper for Host Controller) JBAS012020: Waiting 1 seconds until trying to restart process Host Controller. 
INFO  [org.jboss.as.process.Server:server-one.status] (reaper for Server:server-one) JBAS012010: Process 'Server:server-one' finished with an exit status of 137
INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012017: Starting process 'Host Controller'
ERROR [org.jboss.as.process.Server:server-one.status] (ProcessController-threads - 4) JBAS012007: Failed to send reconnect message to process 'Server:server-one' input stream: java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method) [rt.jar:1.6.0_24]
        at java.io.FileOutputStream.write(FileOutputStream.java:297) [rt.jar:1.6.0_24]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) [rt.jar:1.6.0_24]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) [rt.jar:1.6.0_24]
        at org.jboss.as.process.ManagedProcess.reconnect(ManagedProcess.java:170) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.ProcessController.sendReconnectProcess(ProcessController.java:336) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.ProcessControllerServerHandler$InitMessageHandler$ConnectedMessageHandler.handleMessage(ProcessControllerServerHandler.java:224) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.protocol.ConnectionImpl.safeHandleMessage(ConnectionImpl.java:269) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.protocol.ConnectionImpl$1$1.run(ConnectionImpl.java:223) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT] 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]
        at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]

After the above happens, server instance process cannot start. When I try to invoke stop and start operation by cli (For instance, /host=slave/server-config=server-one:stop and /host=slave/server-config=server-one:start), the following ERROR is thrown in host-controller.log:

INFO  [org.jboss.as.host.controller] (domain-connection-threads - 1) JBAS010923: Stopping server server-one
ERROR [org.jboss.as.controller.management-operation] (domain-connection-threads - 1) JBAS014612: Operation ("start") failed - address: ([
    ("host" => "slave"),
    ("server-config" => "server-one")
]): java.lang.IllegalStateException
        at org.jboss.as.host.controller.ManagedServer.start(ManagedServer.java:192) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.ServerInventoryImpl.startServer(ServerInventoryImpl.java:166) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.DomainModelControllerService$DelegatingServerInventory.startServer(DomainModelControllerService.java:557) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.operations.ServerStartHandler$1.execute(ServerStartHandler.java:80) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]        
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeDirect(OperationCoordinatorStepHandler.java:161) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:112) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:126) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:111) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:112) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.execute(TransactionalProtocolOperationHandler.java:95) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:286) [jboss-as-protocol-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:491) [jboss-as-protocol-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]
        at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 6.0.x
  • Domain mode

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content