Fabric Child Container can't be started - java.lang.IllegalStateException: Instance already started

Solution Verified - Updated -

Issue

  • VM running a parent/child containers went down.
  • After VM came back, one of the child containers is in dodgy state.
  • It is not running, there is no PID and Fabric shows it as "starting" and not connected.
JBossFuse:admin@fuseesb-uat-jonpl> container-stop -f proxy-server-ad
The list of container names: [proxy-server-ad]
Container 'proxy-server-ad' already stopped.
JBossFuse:admin@fuseesb-uat-jonpl> container-info proxy-server-ad
Name:                          proxy-server-ad
Version:                       1.13
Connected:                     false
Type:                          karaf
Root:                          false
Ensemble Server:               false
Managed:                       true
Process ID:
Resolver:                      localhostname
Network Address:               some.host
Local Network Address:         XXX.XXX.XXX.XXX
Public Network Address:
Local Hostname:                some.host
Public Hostname:
SSH Url:                       some.host:8102
JMX Url:                       service:jmx:rmi://some.host:44445/jndi/rmi://some.host:1100/karaf-proxy-server-ad
Http Url:                      http://some.host:8188
Jolokia Url:                   http://some.host:8188/jolokia
Provision Status:              starting
JBossFuse:admin@fuseesb-uat-jonpl> container-list|grep proxy-server-ad
  proxy-server-ad     1.13       karaf   no           default                                             starting
JBossFuse:admin@fuseesb-uat-jonpl> container-start -f proxy-server-ad
The list of container names: [proxy-server-ad]
Error executing command: Instance already started
  • child container (proxy-server-ad) logs are not updated at any point (last updated before VM outage).
  • Log from parent container when trying to reach out and start child container:
2017-06-14 14:03:10,343 | INFO  | Thread-826       | FabricServiceImpl                | 142 - io.fabric8.fabric-core - 1.2.0.redhat-621159 | Starting container proxy-server-ad
2017-06-14 14:03:10,345 | INFO  | Thread-826       | ChildContainerProvider           | 142 - io.fabric8.fabric-core - 1.2.0.redhat-621159 | Using container controller io.fabric8.service.child.ChildContainerProvider$1@28487c81
2017-06-14 14:03:10,568 | ERROR | Thread-826       | Console                          | 38 - org.apache.karaf.shell.console - 2.4.0.redhat-621159 | Exception caught while executing command
java.lang.IllegalStateException: Instance already started
        at org.apache.karaf.admin.internal.AdminServiceImpl$6.call(AdminServiceImpl.java:436)
        at org.apache.karaf.admin.internal.AdminServiceImpl$1.call(AdminServiceImpl.java:199)
        at org.apache.karaf.util.properties.FileLockUtils$2.call(FileLockUtils.java:103)
        at org.apache.karaf.util.properties.FileLockUtils.execute(FileLockUtils.java:69)
        at org.apache.karaf.util.properties.FileLockUtils.execute(FileLockUtils.java:97)
        at org.apache.karaf.admin.internal.AdminServiceImpl.execute(AdminServiceImpl.java:196)
        at org.apache.karaf.admin.internal.AdminServiceImpl.startInstance(AdminServiceImpl.java:428)
        at org.apache.karaf.admin.internal.InstanceImpl.start(InstanceImpl.java:92)
        at org.apache.karaf.admin.management.internal.AdminServiceMBeanImpl.startInstance(AdminServiceMBeanImpl.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at javax.management.StandardMBean.invoke(StandardMBean.java:405)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.karaf.management.internal.MBeanInvocationHandler.invoke(MBeanInvocationHandler.java:60)
        at com.sun.proxy.$Proxy3.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Environment

  • Red Hat JBoss Fuse
    • 6.2.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content