AMQ startup takes a long time with a large kahadb / index

Issue

• Upon startup with a large kahadb / index, we are seeing extended startup times (> 10 minutes), with no logging present in the activemq-related threads (BrokerService, ActiveMqServiceFactory, MessageDatabase) for several to many minutes between the Using Persistence Adapter: statement and the KahaDB is version statement. Looking at thread dumps, we see a long-running thread that appears to be in journal reading operations. It does not appear to be blocked - just busy (99% cpu):

• We are experiencing quite big delay during start of the broker (15 minutes) when the step "Amending user changes" occur.
  Can you describe what's the "Amending user changes" meant for and if it can potently cause such a delay?

  • Looking at the dump, the large majority of cpu time (99%) is being spent in this thread:

"AMQ-1-thread-1" #101 prio=5 os_prio=0 tid=0x00007ee4fc010800 nid=0x5d0 runnable [0x00007ee4988d6000]
   java.lang.Thread.State: RUNNABLE
    at java.io.RandomAccessFile.readBytes(Native Method)
    at java.io.RandomAccessFile.read(RandomAccessFile.java:377)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
    at org.apache.activemq.util.RecoverableRandomAccessFile.readFully(RecoverableRandomAccessFile.java:75)
    at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readFully(DataFileAccessor.java:97)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.checkBatchRecord(Journal.java:393)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.recoveryCheck(Journal.java:325)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.start(Journal.java:199)
    - locked <0x00007f28188dbfd8> (a org.apache.activemq.store.kahadb.disk.journal.Journal)
    at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:378)
    at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:418)
    at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:262)
    at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:205)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:657)
    at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:641)
    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:606)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.doStart(ActiveMQServiceFactory.java:506)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.access$400(ActiveMQServiceFactory.java:318)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$1.run(ActiveMQServiceFactory.java:449)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00007f2818099798> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    - <0x00007f28380d0310> (a java.util.concurrent.ThreadPoolExecutor$Worker)

• It takes a long time after start up before the following message is logged

INFO  | AMQ-1-thread-1   | ActiveMQServiceFactory           | mq.fabric.ActiveMQServiceFactory  102 | 143 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-621090 | Broker amqbroker has started.