Perform graceful shutdown, restart, KahaDB gets damanged, rebuilding indeces

Issue

Note: This KCS article only applies to the non-Karaf version of ActiveMQ, i.e the distribution found in $AMQ_HOME/extras/apache-activemq-*.zip!

On one of our preproduction instances, we execute the ActiveMQ stop command and wait for the process to restart. We then invoke the start command. Everytime, ActiveMQ states that KahaDB is corrupt and the indexes need to be recovered.

The reason this is bad, is that it can take two hours to recover. During that time, ActiveMQ is inoperable. I cannot take any requests while it is recovering the DB. Second, it doesn't fix it itself. After recovering the first time, stop/start ActiveMQ and notice it recover again, and again. After doing it once, it shouldn't do it again.

Sample log entry.

2013-09-09 14:54:55,495 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:123287138, 5900000 entries recovered ..] 
2013-09-09 14:55:02,224 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:246168059, 6000000 entries recovered ..] 
2013-09-09 14:55:09,715 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:361043817, 6100000 entries recovered ..] 
2013-09-09 14:55:16,116 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:474014379, 6200000 entries recovered ..] 
2013-09-09 14:55:22,785 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:579379976, 6300000 entries recovered ..] 
2013-09-09 14:55:29,301 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:686637801, 6400000 entries recovered ..] 
2013-09-09 14:55:35,614 -0700 level=INFO  thread=WrapperSimpleAppMain class=MessageDatabase [@69:799921496, 6500000 entries recovered ..]