AMQ startup takes a long time with a large kahadb / index

Solution In Progress - Updated -

Issue

  • Upon startup with a large kahadb / index, we are seeing extended startup times (> 10 minutes), with no logging present in the activemq-related threads (BrokerService, ActiveMqServiceFactory, MessageDatabase) for several to many minutes between the Using Persistence Adapter: statement and the KahaDB is version statement. Looking at thread dumps, we see a long-running thread that appears to be in journal reading operations. It does not appear to be blocked - just busy (99% cpu):

  • We are experiencing quite big delay during start of the broker (15 minutes) when the step "Amending user changes" occur.
    Can you describe what's the "Amending user changes" meant for and if it can potently cause such a delay?

    • Looking at the dump, the large majority of cpu time (99%) is being spent in this thread:
"AMQ-1-thread-1" #101 prio=5 os_prio=0 tid=0x00007ee4fc010800 nid=0x5d0 runnable [0x00007ee4988d6000]
   java.lang.Thread.State: RUNNABLE
    at java.io.RandomAccessFile.readBytes(Native Method)
    at java.io.RandomAccessFile.read(RandomAccessFile.java:377)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
    at org.apache.activemq.util.RecoverableRandomAccessFile.readFully(RecoverableRandomAccessFile.java:75)
    at org.apache.activemq.store.kahadb.disk.journal.DataFileAccessor.readFully(DataFileAccessor.java:97)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.checkBatchRecord(Journal.java:393)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.recoveryCheck(Journal.java:325)
    at org.apache.activemq.store.kahadb.disk.journal.Journal.start(Journal.java:199)
    - locked <0x00007f28188dbfd8> (a org.apache.activemq.store.kahadb.disk.journal.Journal)
    at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:378)
    at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:418)
    at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:262)
    at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:205)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:657)
    at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:641)
    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:606)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.doStart(ActiveMQServiceFactory.java:506)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration.access$400(ActiveMQServiceFactory.java:318)
    at io.fabric8.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$1.run(ActiveMQServiceFactory.java:449)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x00007f2818099798> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    - <0x00007f28380d0310> (a java.util.concurrent.ThreadPoolExecutor$Worker)
  • It takes a long time after start up before the following message is logged
INFO  | AMQ-1-thread-1   | ActiveMQServiceFactory           | mq.fabric.ActiveMQServiceFactory  102 | 143 - io.fabric8.mq.mq-fabric - 1.2.0.redhat-621090 | Broker amqbroker has started.

Environment

  • Red Hat JBoss AMQ 6.2.1
    • checksumJournalFiles="true" & checkForCorruptJournalFiles="true" set in activemq.xml
    • large index
    • large journal (kahadb) size

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In