How to recover the messages from the backup journal.
Issue
-
We have a HornetQ cluster running inside JBoss EAP 6.2.2 with the following characteristics:
Symmetrical
Live/backup pairs
ReplicationGiven the following scenario, we end up with messages that are stuck in the backup journals:
Start EAP 1 which includes a HornetQ live and backup server
Start EAP 2 which includes a HornetQ live and backup server
Start EAP 3 which consumes messages from the HornetQ cluster via an MDB
Start EAP 4 which produces messages via a connection factory that balances the messages to the 2 live servers
Messages are being consumed by EAP 3 from EAP 1 and EAP 2
Stop EAP 1, the backup server on EAP 2 becomes active
Messages are still being consumed by EAP 3 from EAP 2
Stop EAP 3, messages are no longer being consumed
Leave EAP 4 running for a little longer and then stop it, EAP 2 live/backup now contain unconsumed messages
Stop EAP 2, no active HornetQ servers are left
Start EAP 1
Start EAP 2, journals from backup server on EAP 2 contains unconsumed messages but the journal is moved aside and the journal from EAP 1 is replicated
to the backup on EAP 2.
Start EAP 3 to consume messages. Notice that some of the messages are lostWe understand that when EAP 2 starts, the journals from its backup HornetQ server are moved aside in favour of the new journal from the live server
running on EAP 1. However given the fact that this is a situation that could occur in our production environment we are looking for a
procedure/bestpractise to recover the messages from the backup journal.We know that we can manually move the journal files or change configurations to make a backup server active but preferably we are looking for an
automated/scripted way to;
check if recovery is necessary
execute the recovery itself
tools to look inside journals/move messages
Can you provide us with information how to correctly implement this?
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 6.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.