Fabric child container restart when connection to dB lost when running active/passive mode with JDBC lock.

Solution Unverified - Updated -

Issue

We have Fuse ESB/MQ two node fabric cluster with multiple child containers running in active/passive mode. The failover id done via dB lock on Oracle 10g database, as per:

https://access.redhat.com/site/documentation/en-US/Fuse_ESB_Enterprise/7.1/html/Configuring_and_Running_Fuse_ESB_Enterprise/files/ESBRuntimeFailoverJdbc.html

Yesterday, we had an issue where the child containers restarted themselves on losing connection to the dB. The active/passive mode did not work as there was no connection for the lock to happen.
So, the question is, is it a feature of Fuse that the containers will restart on losing connection to the dB? or is it a bug?

Here is the log from the time it happened.

2013-05-29 18:33:22,677 | WARN  | Thread-2         | OracleJDBCLock                   | apache.karaf.main.OracleJDBCLock   97 |  -  -  | Failed to acquire database lock: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

2013-05-29 18:33:22,678 | INFO  | Thread-2         | Main                             | org.apache.karaf.main.Main       1392 |  -  -  | Lost the lock, reducing start level to 1
2013-05-29 18:33:22,690 | INFO  | FelixStartLevel  | BlueprintCamelContext            | e.camel.impl.DefaultCamelContext 1596 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Apache Camel 2.10.0.fuse-71-051 (CamelContext: payroll) is shutting down
2013-05-29 18:33:22,693 | INFO  | FelixStartLevel  | DefaultShutdownStrategy          | mel.impl.DefaultShutdownStrategy  165 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Starting to graceful shutdown 3 routes (timeout 300 seconds)
2013-05-29 18:33:22,708 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-processed-to-archive shutdown complete, was consuming from: Endpoint[file:///mnt/esb/prod/internal/payroll/internal/processed?delay=30000&delete=true&exclude=.*%7E%24&readLock=changed]
2013-05-29 18:33:22,709 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-internal-to-destination shutdown complete, was consuming from: Endpoint[file:///mnt/esb/prod/internal/payroll/internal/?delay=30000&exclude=.*%7E%24&move=processed&readLock=changed]
2013-05-29 18:33:22,711 | INFO  | 3 - ShutdownTask | DefaultShutdownStrategy          | ultShutdownStrategy$ShutdownTask  559 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Route: payroll-source-to-internal shutdown complete, was consuming from: Endpoint[file:///mnt/corpdfs/fps01/Payroll%20BPO/PAYROLLFINAL34T?delay=60000&delete=true&exclude=.*%7E%24&readLock=changed]
2013-05-29 18:33:22,711 | INFO  | FelixStartLevel  | DefaultShutdownStrategy          | mel.impl.DefaultShutdownStrategy  213 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Graceful shutdown of 3 routes completed in 0 seconds
2013-05-29 18:33:22,716 | INFO  | FelixStartLevel  | DefaultTypeConverter             | verter.BaseTypeConverterRegistry  522 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | TypeConverterRegistry utilization[attempts=14, hits=14, misses=0, failures=0] mappings[total=179, misses=0]
2013-05-29 18:33:22,718 | INFO  | FelixStartLevel  | BlueprintCamelContext            | e.camel.impl.DefaultCamelContext 1662 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Apache Camel 2.10.0.fuse-71-051 (CamelContext: payroll) is shutdown in 0.027 seconds. Uptime 1 day 4 hours.
2013-05-29 18:33:22,751 | INFO  | FelixStartLevel  | Activator                        | apache.camel.impl.osgi.Activator   91 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Camel activator stopping
2013-05-29 18:33:22,755 | INFO  | FelixStartLevel  | Activator                        | apache.camel.impl.osgi.Activator   93 | 106 - org.apache.camel.camel-core - 2.10.0.fuse-71-049 | Camel activator stopped
2013-05-29 18:33:22,759 | INFO  | FelixStartLevel  | ContextLoaderListener            | .activator.ContextLoaderListener  447 | 101 - org.springframework.osgi.extender - 1.2.1 | Stopping [org.springframework.osgi.extender] bundle v.[1.2.1]
2013-05-29 18:33:22,773 | INFO  | FelixStartLevel  | TimerTaskExecutor                | heduling.timer.TimerTaskExecutor  179 | 111 - org.springframework.context - 3.0.7.RELEASE | Cancelling Timer
2013-05-29 18:33:22,843 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=org.fusesource.fabric.jolokia.JolokiaSecureHttpContext@7b15c61c}
2013-05-29 18:33:22,843 | INFO  | b Runtime worker | HttpServiceFactoryImpl           | .internal.HttpServiceFactoryImpl   40 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Unbinding bundle: [org.fusesource.fabric.fabric-maven-proxy [72]]
2013-05-29 18:33:22,848 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=DefaultHttpContext{bundle=org.fusesource.fabric.fabric-maven-proxy [72]}}
2013-05-29 18:33:22,850 | INFO  | b Runtime worker | ContextHandler                   | lipse.jetty.util.log.JavaUtilLog   75 | 79 - org.eclipse.jetty.util - 7.6.7.v20120910 | stopped HttpServiceContext{httpContext=org.fusesource.fabric.maven.impl.SecureHttpContext@3597892}
2013-05-29 18:33:22,850 | INFO  | b Runtime worker | HttpServiceFactoryImpl           | .internal.HttpServiceFactoryImpl   40 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Unbinding bundle: [org.fusesource.fabric.fabric-jolokia [73]]
2013-05-29 18:33:22,913 | INFO  | FelixStartLevel  | Activator                        | x.web.service.internal.Activator  148 | 80 - org.ops4j.pax.web.pax-web-runtime - 1.1.11 | Pax Web stopped
2013-05-29 18:33:22,997 | INFO  | FelixStartLevel  | DeploymentAgent                  | rce.fabric.agent.DeploymentAgent  216 | 67 - org.fusesource.fabric.fabric-agent - 7.1.0.fuse-047 | Stopping DeploymentAgent
2013-05-29 18:33:23,170 | INFO  | ispatcher thread | AbstractZKClient                 | ctZKClient$StateChangeDispatcher  361 | 59 - org.fusesource.fabric.fabric-zookeeper - 7.1.0.fuse-047 | StateChangeDispatcher terminated.
2013-05-29 18:33:23,181 | INFO  | ad-1-EventThread | ClientCnxn                       | zookeeper.ClientCnxn$EventThread  511 | 58 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | EventThread shut down
2013-05-29 18:33:23,182 | INFO  | FelixStartLevel  | ZooKeeper                        | org.apache.zookeeper.ZooKeeper    679 | 58 - org.fusesource.fabric.fabric-linkedin-zookeeper - 7.1.0.fuse-047 | Session: 0x23ed3fe6231001d closed
2013-05-29 18:33:23,367 | INFO  | FelixStartLevel  | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Stopping JMX OSGi agent
2013-05-29 18:33:23,369 | INFO  | FelixStartLevel  | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering MBean with ObjectName [osgi.compendium:service=cm,version=1.3] for service with service.id [11]
2013-05-29 18:33:23,388 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.compendium:service=cm,version=1.3
2013-05-29 18:33:23,389 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.BundleStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=bundleState,version=1.5
2013-05-29 18:33:23,389 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.ServiceStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=serviceState,version=1.5
2013-05-29 18:33:23,390 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.FrameworkMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=framework,version=1.5
2013-05-29 18:33:23,390 | INFO  | JMX OSGi Agent   | core                             | ?                                   ? | 38 - org.apache.aries.jmx.core - 1.0.1.fuse-71-047 | Unregistering org.osgi.jmx.framework.PackageStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@16dd5a9d with name osgi.core:type=packageState,version=1.5
2013-05-29 18:33:23,619 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   59 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling SLF4J API support.
2013-05-29 18:33:23,619 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   62 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Jakarta Commons Logging API support.
2013-05-29 18:33:23,620 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   65 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Log4J API support.
2013-05-29 18:33:23,621 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   68 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling Avalon Logger API support.
2013-05-29 18:33:23,622 | INFO  | FelixStartLevel  | Activator                        | j.pax.logging.internal.Activator   71 | 4 - org.ops4j.pax.logging.pax-logging-api - 1.7.0 | Disabling JULI Logger API support.
2013-05-29 18:33:24,852 | INFO  | -71-047-thread-2 | fileinstall                      | ?                                   ? | 6 - org.apache.felix.fileinstall - 3.3.11.fuse-71-047 | Installed /opt/FuseESBEnterprise-7.1.0/instances/esb-payroll-node02/etc/org.ops4j.pax.logging.cfg

Environment

Fuse ESB Enterprise 7.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content