Issue

We have Fuse ESB/MQ two node fabric cluster with multiple child containers running in active/passive mode. The failover id done via dB lock on Oracle 10g database, as per:

https://access.redhat.com/site/documentation/en-US/Fuse_ESB_Enterprise/7.1/html/Configuring_and_Running_Fuse_ESB_Enterprise/files/ESBRuntimeFailoverJdbc.html

Yesterday, we had an issue where the child containers restarted themselves on losing connection to the dB. The active/passive mode did not work as there was no connection for the lock to happen.
So, the question is, is it a feature of Fuse that the containers will restart on losing connection to the dB? or is it a bug?

Here is the log from the time it happened.

2013-05-29 18:33:22,677 | WARN  | Thread-2         | OracleJDBCLock                   | apache.karaf.main.OracleJDBCLock   97 |  -  -  | Failed to acquire database lock: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

2013-05-29 18:33:22,678 | INFO  | Thread-2         | Main                             | org.apache.karaf.main.Main       1392 |  -  -  | Lost the lock, reducing start level to 1
2013-05-29 18:33:22,690 | INFO  | FelixStartLevel  | BlueprintCamelContext            | e.camel.impl.DefaultCamelContext 1596 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Apache Camel 2.10.0.fuse-71-051 (CamelContext: payroll) is shutting down
2013-05-29 18:33:22,693 | INFO  | FelixStartLevel  | DefaultShutdownStrategy          | mel.impl.DefaultShutdownStrategy  165 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Starting to graceful shutdown 3 routes (timeout 300 seconds)
2013-05-29 18:33:22,708 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-processed-to-archive shutdown complete, was consuming from: Endpoint[file:///mnt/esb/prod/internal/payroll/internal/processed?delay=30000&delete=true&exclude=.*%7E%24&readLock=changed]
2013-05-29 18:33:22,709 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-internal-to-destination shutdown complete, was consuming from: Endpoint[file:///mnt/esb/prod/internal/payroll/internal/?delay=30000&exclude=.*%7E%24&move=processed&readLock=changed]
2013-05-29 18:33:22,711 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-source-to-internal shutdown complete, was consuming from: Endpoint[file:///mnt/corpdfs/fps01/Payroll%20BPO/PAYROLLFINAL34T?delay=60000&delete=true&exclude=.*%7E%24&readLock=changed]
2013-05-29 18:33:22,711 | INFO  | FelixStartLevel  | DefaultShutdownStrategy          | mel.impl.DefaultShutdownStrategy  213 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Graceful shutdown of 3 routes completed in 0 seconds
2013-05-29 18:33:22,716 | INFO  | FelixStartLevel  | DefaultTypeConverter             | verter.BaseTypeConverterRegistry  522 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | TypeConverterRegistry utilization[attempts=14, hits=14, misses=0, failures=0] mappings[total=179, misses=0]
2013-05-29 18:33:22,718 | INFO  | FelixStartLevel  | BlueprintCamelContext            | e.camel.impl.DefaultCamelContext 1662 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Apache Camel 2.10.0.fuse-71-051 (CamelContext: payroll) is shutdown in 0.027 seconds. Uptime 1 day 4 hours.
2013-05-29 18:33:22,751 | INFO  | FelixStartLevel  | Activator                        | apache.camel.impl.osgi.Activator   91 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Camel activator stopping
2013-05-29 18:33:22,755 | INFO  | FelixStartLevel  | Activator                        | apache.camel.impl.osgi.Activator   93 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Camel activator stopped
2013-05-29 18:33:22,759 | INFO  | FelixStartLevel  | ContextLoaderListener            | .activator.ContextLoaderListener  447 | 101 - org.springframework.osgi.extender - 1.2.1 | Stopping [org.springframework.osgi.extender] bundle v.[1.2.1]
2013-05-29 18:33:22,773 | INFO  | FelixStartLevel  | TimerTaskExecutor                | heduling.timer.TimerTaskExecutor  179 | 111 - org.springframework.context - 3.0.7.RELEASE | Cancelling Timer
2013-05-29 18:33:22,843 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=org.fusesource.fabric.jolokia.JolokiaSecureHttpContext@7b15c61c}
2013-05-29 18:33:22,843 | INFO  | b Runtime worker | HttpServiceFactoryImpl           | .internal.HttpServiceFactoryImpl   40 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Unbinding bundle: [org.fusesource.fabric.fabric-maven-proxy [72]]
2013-05-29 18:33:22,848 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=DefaultHttpContext{bundle=org.fusesource.fabric.fabric-maven-proxy [72]}}
2013-05-29 18:33:22,850 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=org.fusesource.fabric.maven.impl.SecureHttpContext@3597892}
2013-05-29 18:33:22,850 | INFO  | b Runtime worker | HttpServiceFactoryImpl           | .internal.HttpServiceFactoryImpl   40 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Unbinding bundle: [org.fusesource.fabric.fabric-jolokia [73]]
2013-05-29 18:33:22,913 | INFO  | FelixStartLevel  | Activator                        | x.web.service.internal.Activator  148 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Pax Web stopped
2013-05-29 18:33:22,997 | INFO  | FelixStartLevel  | DeploymentAgent                  | rce.fabric.agent.DeploymentAgent  216 | 67 - org.fusesource.fabric.fabric-agent - 7.1.0.fuse-047 | Stopping DeploymentAgent
2013-05-29 18:33:23,170 | INFO  | ispatcher thread | AbstractZKClient                 | ctZKClient$StateChangeDispatcher  361 | 59 - org.fusesource.fabric.fabric-zookeeper - 7.1.0.fuse-047 | StateChangeDispatcher terminated.
2013-05-29 18:33:23,181 | INFO  | ad-1-EventThread | ClientCnxn                       | zookeeper.ClientCnxn$EventThread  511 | 58 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | EventThread shut down
2013-05-29 18:33:23,182 | INFO  | FelixStartLevel  | ZooKeeper                        | org.apache.zookeeper.ZooKeeper    679 | 58 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | Session: 0x23ed3fe6231001d closed
2013-05-29 18:33:23,367 | INFO  | FelixStartLevel  | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Stopping JMX OSGi agent
2013-05-29 18:33:23,369 | INFO  | FelixStartLevel  | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering MBean with ObjectName [osgi.compendium:service=cm,version=1.3] for service with service.id [11]
2013-05-29 18:33:23,388 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.compendium:service=cm,version=1.3
2013-05-29 18:33:23,389 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.BundleStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=bundleState,version=1.5
2013-05-29 18:33:23,389 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.ServiceStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=serviceState,version=1.5
2013-05-29 18:33:23,390 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.FrameworkMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=framework,version=1.5
2013-05-29 18:33:23,390 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.PackageStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=packageState,version=1.5
2013-05-29 18:33:23,619 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   59 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling SLF4J API support.
2013-05-29 18:33:23,619 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   62 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Jakarta Commons Logging API support.
2013-05-29 18:33:23,620 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   65 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Log4J API support.
2013-05-29 18:33:23,621 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   68 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Avalon Logger API support.
2013-05-29 18:33:23,622 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   71 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling JULI Logger API support.
2013-05-29 18:33:24,852 | INFO  | -71-047-thread-2 | fileinstall                      | ?                                   ? | 6 - org.apache.felix.fileinstall - 3.3.11.fuse-71-047 | Installed /opt/FuseESBEnterprise-7.1.0/instances/esb-payroll-node02/etc/org.ops4j.pax.logging.cfg