Why is a failed XA transaction added to the recovery list in a single resource transaction in EAP 6?

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 6

Issue

  • A single XA data source is enlisted in a transaction
  • The transaction fails silently (does not raise an exception to application code) during commit
  • A WARN message is logged such as the following:
... WARN  [com.arjuna.ats.jta] ... ARJUNA016039: onePhaseCommit on < ... > ... failed with exception XAException.XAER_RMFAIL
  • Subsequently, WARN messages are logged continuously (every few minutes - depending on the period of recovery runs)
... WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < ... >, heuristic: TwoPhaseOutcome.FINISH_OK, product: ...
... WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < ... >

Resolution

  • Unrecoverable transactions may require administrative action (if application "decisions" were made based on presumed commit success)
  • After any necessary administrative intervention has been performed, remove the transaction from the store
  • If the issue occurs frequently for different transactions, identify the cause of commit failures (see Diagnostic Steps)
  • Future EAP releases may provide configuration to raise an exception to application code on single-resource commit failure

Root Cause

  • The transaction failed with an error code indicating the commit outcome is in-doubt (e.g. XAException.XAER_RMFAIL)
    • The error code does not indicate if commit completed or if the request for commit reached the server
    • JBoss does not report the transaction as failed (as it may have completed or else may remain unterminated)
  • Because the actual outcome is unknown, JBoss includes the transaction in the recovery list
    • A subsequent commit attempt may be successful
    • Repeated recovery failures indicate that the resource manager (e.g. database) does not retain knowledge of the transaction
  • The root cause of commit failures for which recovery is not possible may vary
    • Connection termination prior to commit is often associated with such failures
      • This may be the result of network or database server issues or internally initiated by JBoss error handling
    • XAException.XAER_NOTA (the resource manager failed to retain memory of the Xid between a successful XAResource.end() and XAResource.commit()) has also been observed to produce a similar outcome

Diagnostic Steps

  • Enable JCA/Transaction trace
  • Search for warnings or exceptions prior to the initial commit and recovery failures involving:
    • The transaction identifier (Xid) mentioned in the recovery WARN messages
    • The thread on which the commit failed
    • The datasource / connection included in the failing transaction

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.