JON 3.2.0 fails with database transaction or row lock exceptions after bundle deployment is attempted

Issue

• Server not starting properly, ARJUNA016061 with no understandable cause
• Cannot store preferences
• Failed to load Resources with alerts or unavailability
• Can not load new dashboard
• Server returned FAILURE with no error message
• Can't seem to send commands the the agent
• During a bundle deployment some messages appeared about not finding deployment info
• User interface (UI) displays message Global uncaught exceptions
• Server will not restart
• Server throws ARJUNA errors then could not connect to back end database
• Bundle deploy to platform resource fails
• JON not working after update
• We are experiencing fatal errors after updating
• We cannot perform any operations on any resources without an error message in the web user interface (UI)
• In the server log we get a lot of Exceptions
• Operation does not appear in the operations history and no action on the resource is performed
• We get database timeout exceptions due to org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ORA-02049 on any operations
• Could not access to JON anymore
• I can't access to admin console anymore
• Cannot connect to the database

• Server log contains warnings similar to:

  WARN  [com.arjuna.ats.jta] (pool-7-thread-1) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff928b7363:190191d7:5304e076:46c, node_name=1, branch_uid=0:ffff928b7363:190191d7:5304e076:470, subordinatenodename=null, eis_name=java:jboss/datasources/RHQDS >: oracle.jdbc.xa.OracleXAException
      at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1110) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
      at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:240) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
      at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:259)
      at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.start(XAResourceWrapperImpl.java:203)
      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
  

• One of the follow errors appear in server.log:

  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@7453f2a5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d connection handles=0 lastUse=1389866415155 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@7f02f75b pool internal context=SemaphoreArrayListManagedConnectionPool@74a75430[pool=RHQDS] xaResource=XAResourceWrapperImpl@1bbdb882[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
  With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
  ERROR [org.jboss.as.ejb3] (EJB default - 2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-7-thread-1) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
  With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) ERROR: current transaction is aborted, commands ignored until end of transaction block
  ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component ServerManagerBean for method public abstract boolean org.rhq.enterprise.server.cloud.instance.ServerManagerLocal.getAndClearServerStatus(): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
  ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component StatusManagerBean for method public abstract java.util.List org.rhq.enterprise.server.cloud.StatusManagerLocal.getAndClearAgentsWithStatusForServer(java.lang.String): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
  

• Errors similar to the following in the log:

  ERROR [org.jboss.as.ejb3.invocation] (pool-7-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginsBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginsLocal.getServerPlugin(org.rhq.core.domain.plugin.PluginKey): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      ...
  Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
  With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
      ...
  Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff928b7363:190191d7:5304e076:46c status: ActionStatus.ABORT_ONLY >
      ...
  
  ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component TopologyManagerBean for method public abstract org.rhq.core.domain.cloud.Server org.rhq.enterprise.server.cloud.TopologyManagerLocal.getServerByName(java.lang.String): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:138) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:228) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:317) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
      ...
  Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
      ...
  
  ERROR [org.quartz.core.ErrorLogger] (RHQScheduler_QuartzSchedulerThread) An error occured while firing trigger 'org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob.org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob': org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ERROR: canceling statement due to statement timeout [See nested exception: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout]
      at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:112) [quartz-1.6.5.jar:1.6.5]
      at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:112) [quartz-1.6.5.jar:1.6.5]
      at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3753) [quartz-1.6.5.jar:1.6.5]
      ...
  Caused by: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout
      at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
      at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
      ...