Chapter 14. Failover Deployments
Abstract
Red Hat Fuse provides failover capability using either a simple lock file system or a JDBC locking mechanism. In both cases, a container-level lock system allows bundles to be preloaded into a secondary kernel instance in order to provide faster failover performance.
14.1. Using a Simple Lock File System
Overview
When you first start Red Hat Fuse a lock file is created at the root of the installation directory. You can set up a primary/secondary system whereby if the primary instance fails, the lock is passed to a secondary instance that resides on the same host machine.
Configuring a lock file system
To configure a lock file failover deployment, edit the etc/system.properties file on both the primary and the secondary installation to include the properties in Example 14.1, “Lock File Failover Configuration”.
Example 14.1. Lock File Failover Configuration
karaf.lock=true
karaf.lock.class=org.apache.karaf.main.SimpleFileLock
karaf.lock.dir=PathToLockFileDirectory
karaf.lock.delay=10000-
karaf.lock—specifies whether the lock file is written. -
karaf.lock.class—specifies the Java class implementing the lock. For a simple file lock it should always beorg.apache.karaf.main.SimpleFileLock. -
karaf.lock.dir—specifies the directory into which the lock file is written. This must be the same for both the primary and the secondary installation. -
karaf.lock.delay—specifies, in milliseconds, the delay between attempts to reaquire the lock.
14.2. Using a JDBC Lock System
Overview
The JDBC locking mechanism is intended for failover deployments where Red Hat Fuse instances exist on separate machines.
In this scenario, the primary instance holds a lock on a locking table hosted on a database. If the primary instance loses the lock, a waiting secondary process gains access to the locking table and fully starts its container.
Adding the JDBC driver to the classpath
In a JDBC locking system, the JDBC driver needs to be on the classpath for each instance in the primary/secondary setup. Add the JDBC driver to the classpath as follows:
-
Copy the JDBC driver JAR file to the
ESBInstallDir/lib/extdirectory for each Red Hat Fuse instance. Modify the
bin/karafstart script so that it includes the JDBC driver JAR in itsCLASSPATHvariable.For example, given the JDBC JAR file,
JDBCJarFile.jar, you could modify the start script as follows (on a *NIX operating system):... # Add the jars in the lib dir for file in "$KARAF_HOME"/lib/karaf*.jar do if [ -z "$CLASSPATH" ]; then CLASSPATH="$file" else CLASSPATH="$CLASSPATH:$file" fi done CLASSPATH="$CLASSPATH:$KARAF_HOME/lib/JDBCJarFile.jar"NoteIf you are adding a MySQL driver JAR or a PostgreSQL driver JAR, you must rename the driver JAR by prefixing it with the
karaf-prefix. Otherwise, Apache Karaf will hang and the log will tell you that Apache Karaf was unable to find the driver.
Configuring a JDBC lock system
To configure a JDBC lock system, update the etc/system.properties file for each instance in the primary/secondary deployment as shown
Example 14.2. JDBC Lock File Configuration
karaf.lock=true karaf.lock.class=org.apache.karaf.main.lock.DefaultJDBCLock karaf.lock.level=50 karaf.lock.delay=10000 karaf.lock.jdbc.url=jdbc:derby://dbserver:1527/sample karaf.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver karaf.lock.jdbc.user=user karaf.lock.jdbc.password=password karaf.lock.jdbc.table=KARAF_LOCK karaf.lock.jdbc.clustername=karaf karaf.lock.jdbc.timeout=30
In the example, a database named sample will be created if it does not already exist. The first Red Hat Fuse instance to acquire the locking table is the primary instance. If the connection to the database is lost, the primary instance tries to gracefully shutdown, allowing a secondary instance to become the primary instance when the database service is restored. The former primary instance will require manual restart.
Configuring JDBC locking on Oracle
If you are using Oracle as your database in a JDBC locking scenario, the karaf.lock.class property in the etc/system.properties file must point to org.apache.karaf.main.lock.OracleJDBCLock.
Otherwise, configure the system.properties file as normal for your setup, as shown:
Example 14.3. JDBC Lock File Configuration for Oracle
karaf.lock=true karaf.lock.class=org.apache.karaf.main.lock.OracleJDBCLock karaf.lock.jdbc.url=jdbc:oracle:thin:@hostname:1521:XE karaf.lock.jdbc.driver=oracle.jdbc.OracleDriver karaf.lock.jdbc.user=user karaf.lock.jdbc.password=password karaf.lock.jdbc.table=KARAF_LOCK karaf.lock.jdbc.clustername=karaf karaf.lock.jdbc.timeout=30
The karaf.lock.jdbc.url requires an active Oracle system ID (SID). This means you must manually create a database instance before using this particular lock.
Configuring JDBC locking on Derby
If you are using Derby as your database in a JDBC locking scenario, the karaf.lock.class property in the etc/system.properties file should point to org.apache.karaf.main.lock.DerbyJDBCLock. For example, you could configure the system.properties file as shown:
Example 14.4. JDBC Lock File Configuration for Derby
karaf.lock=true karaf.lock.class=org.apache.karaf.main.lock.DerbyJDBCLock karaf.lock.jdbc.url=jdbc:derby://127.0.0.1:1527/dbname karaf.lock.jdbc.driver=org.apache.derby.jdbc.ClientDriver karaf.lock.jdbc.user=user karaf.lock.jdbc.password=password karaf.lock.jdbc.table=KARAF_LOCK karaf.lock.jdbc.clustername=karaf karaf.lock.jdbc.timeout=30
Configuring JDBC locking on MySQL
If you are using MySQL as your database in a JDBC locking scenario, the karaf.lock.class property in the etc/system.properties file must point to org.apache.karaf.main.lock.MySQLJDBCLock. For example, you could configure the system.properties file as shown:
Example 14.5. JDBC Lock File Configuration for MySQL
karaf.lock=true karaf.lock.class=org.apache.karaf.main.lock.MySQLJDBCLock karaf.lock.jdbc.url=jdbc:mysql://127.0.0.1:3306/dbname karaf.lock.jdbc.driver=com.mysql.jdbc.Driver karaf.lock.jdbc.user=user karaf.lock.jdbc.password=password karaf.lock.jdbc.table=KARAF_LOCK karaf.lock.jdbc.clustername=karaf karaf.lock.jdbc.timeout=30
Configuring JDBC locking on PostgreSQL
If you are using PostgreSQL as your database in a JDBC locking scenario, the karaf.lock.class property in the etc/system.properties file must point to org.apache.karaf.main.lock.PostgreSQLJDBCLock. For example, you could configure the system.properties file as shown:
Example 14.6. JDBC Lock File Configuration for PostgreSQL
karaf.lock=true karaf.lock.class=org.apache.karaf.main.lock.PostgreSQLJDBCLock karaf.lock.jdbc.url=jdbc:postgresql://127.0.0.1:5432/dbname karaf.lock.jdbc.driver=org.postgresql.Driver karaf.lock.jdbc.user=user karaf.lock.jdbc.password=password karaf.lock.jdbc.table=KARAF_LOCK karaf.lock.jdbc.clustername=karaf karaf.lock.jdbc.timeout=0
JDBC lock classes
The following JDBC lock classes are currently provided by Apache Karaf:
org.apache.karaf.main.lock.DefaultJDBCLock org.apache.karaf.main.lock.DerbyJDBCLock org.apache.karaf.main.lock.MySQLJDBCLock org.apache.karaf.main.lock.OracleJDBCLock org.apache.karaf.main.lock.PostgreSQLJDBCLock
14.3. Container-level Locking
Overview
Container-level locking allows bundles to be preloaded into the secondary kernel instance in order to provide faster failover performance. Container-level locking is supported in both the simple file and JDBC locking mechanisms.
Configuring container-level locking
To implement container-level locking, add the following to the etc/system.properties file on each system in the primary/secondary setup:
Example 14.7. Container-level Locking Configuration
karaf.lock=true karaf.lock.level=50 karaf.lock.delay=10000
The karaf.lock.level property tells the Red Hat Fuse instance how far up the boot process to bring the OSGi container. Bundles assigned the same start level or lower will then also be started in that Fuse instance.
Bundle start levels are specified in etc/startup.properties, in the format BundleName.jar=level. The core system bundles have levels below 50, where as user bundles have levels greater than 50.
Table 14.1. Bundle Start Levels
| Start Level | Behavior |
|---|---|
| 1 | A 'cold' standby instance. Core bundles are not loaded into container. Secondary instances will wait until lock acquired to start server. |
| <50 | A 'hot' standby instance. Core bundles are loaded into the container. Secondary instances will wait until lock acquired to start user level bundles. The console will be accessible for each secondary instance at this level. |
| >50 | This setting is not recommended as user bundles will be started. |
Avoiding port conflicts
When using a 'hot' spare on the same host you need to set the JMX remote port to a unique value to avoid bind conflicts. You can edit the fuse start script (or the karaf script on a child instance) to include the following:
DEFAULT_JAVA_OPTS="-server $DEFAULT_JAVA_OPTS -Dcom.sun.management.jmxremote.port=1100 -Dcom.sun.management.jmxremote.authenticate=false"
