Fabric Child Container can't be started - java.lang.IllegalStateException: Instance already started
Issue
- VM running a parent/child containers went down.
- After VM came back, one of the child containers is in dodgy state.
- It is not running, there is no PID and Fabric shows it as "starting" and not connected.
JBossFuse:admin@fuseesb-uat-jonpl> container-stop -f proxy-server-ad
The list of container names: [proxy-server-ad]
Container 'proxy-server-ad' already stopped.
JBossFuse:admin@fuseesb-uat-jonpl> container-info proxy-server-ad
Name: proxy-server-ad
Version: 1.13
Connected: false
Type: karaf
Root: false
Ensemble Server: false
Managed: true
Process ID:
Resolver: localhostname
Network Address: some.host
Local Network Address: XXX.XXX.XXX.XXX
Public Network Address:
Local Hostname: some.host
Public Hostname:
SSH Url: some.host:8102
JMX Url: service:jmx:rmi://some.host:44445/jndi/rmi://some.host:1100/karaf-proxy-server-ad
Http Url: http://some.host:8188
Jolokia Url: http://some.host:8188/jolokia
Provision Status: starting
JBossFuse:admin@fuseesb-uat-jonpl> container-list|grep proxy-server-ad
proxy-server-ad 1.13 karaf no default starting
JBossFuse:admin@fuseesb-uat-jonpl> container-start -f proxy-server-ad
The list of container names: [proxy-server-ad]
Error executing command: Instance already started
- child container (proxy-server-ad) logs are not updated at any point (last updated before VM outage).
- Log from parent container when trying to reach out and start child container:
2017-06-14 14:03:10,343 | INFO | Thread-826 | FabricServiceImpl | 142 - io.fabric8.fabric-core - 1.2.0.redhat-621159 | Starting container proxy-server-ad
2017-06-14 14:03:10,345 | INFO | Thread-826 | ChildContainerProvider | 142 - io.fabric8.fabric-core - 1.2.0.redhat-621159 | Using container controller io.fabric8.service.child.ChildContainerProvider$1@28487c81
2017-06-14 14:03:10,568 | ERROR | Thread-826 | Console | 38 - org.apache.karaf.shell.console - 2.4.0.redhat-621159 | Exception caught while executing command
java.lang.IllegalStateException: Instance already started
at org.apache.karaf.admin.internal.AdminServiceImpl$6.call(AdminServiceImpl.java:436)
at org.apache.karaf.admin.internal.AdminServiceImpl$1.call(AdminServiceImpl.java:199)
at org.apache.karaf.util.properties.FileLockUtils$2.call(FileLockUtils.java:103)
at org.apache.karaf.util.properties.FileLockUtils.execute(FileLockUtils.java:69)
at org.apache.karaf.util.properties.FileLockUtils.execute(FileLockUtils.java:97)
at org.apache.karaf.admin.internal.AdminServiceImpl.execute(AdminServiceImpl.java:196)
at org.apache.karaf.admin.internal.AdminServiceImpl.startInstance(AdminServiceImpl.java:428)
at org.apache.karaf.admin.internal.InstanceImpl.start(InstanceImpl.java:92)
at org.apache.karaf.admin.management.internal.AdminServiceMBeanImpl.startInstance(AdminServiceMBeanImpl.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at javax.management.StandardMBean.invoke(StandardMBean.java:405)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.karaf.management.internal.MBeanInvocationHandler.invoke(MBeanInvocationHandler.java:60)
at com.sun.proxy.$Proxy3.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Environment
- Red Hat JBoss Fuse
- 6.2.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.