ActiveMQ broker shuts down when connection between broker and Database is broken due to either database restart or network glitch

Issue

This article describes a scenario where the broker is connecting to a remote Database via the JDBCPersistenceAdapter, then failure of the Database or a network glitch between broker and the database causes the ActiveMQ broker to shut down.

Background:

In order to support master/slave deployments, the default behavior is for the broker to
acquire an exclusive database lock on its persistent store. On startup, master and slave will try to connect and acquire a lock on the same shared JDBC database. The broker instance that successfully acquires the lock, becomes the master. The secondary broker will wait for however long its takes for the lock to be released before completing its initialization.

Once the master has acquired a lock, it launches a DB locker keepAlive timer thread to periodically check that it maintains exclusive right to access/hold the lock. It accesses the database table "ACTIVEMQ_LOCK" and regularly updates a lock object within the table.

In cases where the connection is terminated, interrupted or torn down, the database lock will be released and the master broker automatically shuts down to give way to the slave broker to become the new master.

The reason the exclusive database lock is tied to the lifecycle of the connection is related to how the table implements its lock. The DB connection created for accessing the "ACTIVEMQ_LOCK" database table has auto-commit mode set to false [1]. This ensures that all SQL statements are grouped into a transaction that are terminated by a call to either commit() or rollback().

By keeping the connection open, the master broker can maintain the database lock on the table and all of its subsequent "keepAlive" SQL calls to "update" the lock. In effect the timer thread keeps a long running transaction to maintain its lock.