How to recover the messages from the backup journal.

Solution Unverified - Updated -

Issue

  • We have a HornetQ cluster running inside JBoss EAP 6.2.2 with the following characteristics:
    Symmetrical
    Live/backup pairs
    Replication

    Given the following scenario, we end up with messages that are stuck in the backup journals:
    Start EAP 1 which includes a HornetQ live and backup server
    Start EAP 2 which includes a HornetQ live and backup server
    Start EAP 3 which consumes messages from the HornetQ cluster via an MDB
    Start EAP 4 which produces messages via a connection factory that balances the messages to the 2 live servers
    Messages are being consumed by EAP 3 from EAP 1 and EAP 2
    Stop EAP 1, the backup server on EAP 2 becomes active
    Messages are still being consumed by EAP 3 from EAP 2
    Stop EAP 3, messages are no longer being consumed
    Leave EAP 4 running for a little longer and then stop it, EAP 2 live/backup now contain unconsumed messages
    Stop EAP 2, no active HornetQ servers are left
    Start EAP 1
    Start EAP 2, journals from backup server on EAP 2 contains unconsumed messages but the journal is moved aside and the journal from EAP 1 is replicated
    to the backup on EAP 2.
    Start EAP 3 to consume messages. Notice that some of the messages are lost

    We understand that when EAP 2 starts, the journals from its backup HornetQ server are moved aside in favour of the new journal from the live server
    running on EAP 1. However given the fact that this is a situation that could occur in our production environment we are looking for a
    procedure/bestpractise to recover the messages from the backup journal.

    We know that we can manually move the journal files or change configurations to make a backup server active but preferably we are looking for an
    automated/scripted way to;
    check if recovery is necessary
    execute the recovery itself
    tools to look inside journals/move messages
    Can you provide us with information how to correctly implement this?

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content