Issue

Over the last 24 hours we have found JBPM unable to consistently progress order processes. We've a feeling that this has something to do with HornetQ transactions no longer working, but the ESB logs are far from telling. From the first point we noticed order processes behaving erratically we noticed this (note it's only a WARN not an ERROR):

The warning:

WARN  [org.hornetq.core.client.impl.ClientSessionImpl] (pool-120-thread-1) Timed out waiting for response when sending packet 54
HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 54]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:302)
        at org.hornetq.core.client.impl.ClientSessionImpl.prepare(ClientSessionImpl.java:1478)
        at org.hornetq.ra.HornetQRAXAResource.prepare(HornetQRAXAResource.java:135)
        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:264)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2919)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2876)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2419)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1790)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94)
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:160)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1433)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:162)
        at org.jbpm.persistence.jta.JtaDbPersistenceService.commit(JtaDbPersistenceService.java:108)
        at org.jbpm.persistence.db.DbPersistenceService.endTransaction(DbPersistenceService.java:127)
        at org.jbpm.persistence.db.DbPersistenceService.close(DbPersistenceService.java:210)
        at org.jbpm.svc.Services.close(Services.java:244)
        at org.jbpm.JbpmContext.closeServices(JbpmContext.java:156)
        at org.jbpm.JbpmContext.close(JbpmContext.java:132)
        at org.jbpm.command.impl.CommandServiceImpl.execute(CommandServiceImpl.java:79)
        at org.jboss.soa.esb.services.jbpm.cmd.CommandExecutor.executeJbpmCommand(CommandExecutor.java:279)
        at org.jboss.soa.esb.services.jbpm.cmd.CommandExecutor.access$100(CommandExecutor.java:51)
        at org.jboss.soa.esb.services.jbpm.cmd.CommandExecutor$NewProcessInstancePerformer.perform(CommandExecutor.java:219)
        at org.jboss.soa.esb.services.jbpm.cmd.CommandExecutor$NewProcessInstancePerformer.execute(CommandExecutor.java:160)
        at org.jboss.soa.esb.services.jbpm.actions.BpmProcessor.process(BpmProcessor.java:90)

A restart seems to have restored new order processes in the system functioning correctly however we are now in a particularly bad situation with hundreds (if not a thousand) order processes "stuck" from the 24h period we think there was a problem for not progressing through automatic transitions for a reason we do not understand.