JON 3.2.0 fails with database transaction or row lock exceptions after bundle deployment is attempted

Solution Unverified - Updated -

Issue

  • Server not starting properly, ARJUNA016061 with no understandable cause
  • Cannot store preferences
  • Failed to load Resources with alerts or unavailability
  • Can not load new dashboard
  • Server returned FAILURE with no error message
  • Can't seem to send commands the the agent
  • During a bundle deployment some messages appeared about not finding deployment info
  • User interface (UI) displays message Global uncaught exceptions
  • Server will not restart
  • Server throws ARJUNA errors then could not connect to back end database
  • Bundle deploy to platform resource fails
  • JON not working after update
  • We are experiencing fatal errors after updating
  • We cannot perform any operations on any resources without an error message in the web user interface (UI)
  • In the server log we get a lot of Exceptions
  • Operation does not appear in the operations history and no action on the resource is performed
  • We get database timeout exceptions due to org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ORA-02049 on any operations
  • Could not access to JON anymore
  • I can't access to admin console anymore
  • Cannot connect to the database
  • Server log contains warnings similar to:

    WARN  [com.arjuna.ats.jta] (pool-7-thread-1) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff928b7363:190191d7:5304e076:46c, node_name=1, branch_uid=0:ffff928b7363:190191d7:5304e076:470, subordinatenodename=null, eis_name=java:jboss/datasources/RHQDS >: oracle.jdbc.xa.OracleXAException
        at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1110) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
        at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:240) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0]
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:259)
        at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.start(XAResourceWrapperImpl.java:203)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4]
    
  • One of the follow errors appear in server.log:

    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@7453f2a5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d connection handles=0 lastUse=1389866415155 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@7f02f75b pool internal context=SemaphoreArrayListManagedConnectionPool@74a75430[pool=RHQDS] xaResource=XAResourceWrapperImpl@1bbdb882[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
    ERROR [org.jboss.as.ejb3] (EJB default - 2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-7-thread-1) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
    ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) ERROR: current transaction is aborted, commands ignored until end of transaction block
    ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component ServerManagerBean for method public abstract boolean org.rhq.enterprise.server.cloud.instance.ServerManagerLocal.getAndClearServerStatus(): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
    ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component StatusManagerBean for method public abstract java.util.List org.rhq.enterprise.server.cloud.StatusManagerLocal.getAndClearAgentsWithStatusForServer(java.lang.String): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet
    
  • Errors similar to the following in the log:

    ERROR [org.jboss.as.ejb3.invocation] (pool-7-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginsBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginsLocal.getServerPlugin(org.rhq.core.domain.plugin.PluginKey): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        ...
    Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null]
        ...
    Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff928b7363:190191d7:5304e076:46c status: ActionStatus.ABORT_ONLY >
        ...
    
    ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component TopologyManagerBean for method public abstract org.rhq.core.domain.cloud.Server org.rhq.enterprise.server.cloud.TopologyManagerLocal.getServerByName(java.lang.String): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:138) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:228) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:317) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]
        ...
    Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection
        ...
    
    ERROR [org.quartz.core.ErrorLogger] (RHQScheduler_QuartzSchedulerThread) An error occured while firing trigger 'org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob.org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob': org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ERROR: canceling statement due to statement timeout [See nested exception: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout]
        at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:112) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:112) [quartz-1.6.5.jar:1.6.5]
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3753) [quartz-1.6.5.jar:1.6.5]
        ...
    Caused by: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        ...
    

Environment

  • Red Hat JBoss Operations Network (ON) 3.2.0
  • Use of JBoss ON provisioning bundles

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In