Newly elected master broker shuts down suddenly when previous master broker JVM process is finally shut down

Solution Verified - Updated -

Issue

The use case is an NFSv4 mounted shared file system Master / Slave configuration for ActiveMQ.
One broker becomes master, the other one slave.

Persistence store configuration explicitly configures a lockKeepAlivePeriod similar to

<kahaDB directory="/mnt/nfs/kahadb" concurrentStoreAndDispatchQueues="false" lockKeepAlivePeriod="5000">
  <locker>
    <shared-file-locker lockAcquireSleepInterval="10000"/>
  </locker>
</kahaDB>

If the master broker looses access to the NFS mount, it will shut down.
That shut down procedure can take a while depending on NFS timeout (a number of minutes at least).

In the meantime the slave broker (who does not suffer from loosing access to NFS mount) might become the new master broker even before the previous master broker has fully shut down its JVM.

If the NFS mount is restored on the original master before the original master process has fully shut down, it may cause a shut down of the new master broker as well.
The new master broker would log a line similar to

ERROR | localhost, no longer able to keep the exclusive lock so giving up being a master

prior to shutting down.

Environment

  • JBoss Fuse 6.1
  • NFSv4 based shared file system master / slave

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.