JON 3.2.0 fails with database transaction or row lock exceptions after bundle deployment is attempted
Issue
- Server not starting properly, ARJUNA016061 with no understandable cause
- Cannot store preferences
- Failed to load Resources with alerts or unavailability
- Can not load new dashboard
- Server returned FAILURE with no error message
- Can't seem to send commands the the agent
- During a bundle deployment some messages appeared about not finding deployment info
- User interface (UI) displays message
Global uncaught exceptions - Server will not restart
- Server throws ARJUNA errors then
could not connectto back end database - Bundle deploy to platform resource fails
- JON not working after update
- We are experiencing fatal errors after updating
- We cannot perform any operations on any resources without an error message in the web user interface (UI)
- In the server log we get a lot of Exceptions
- Operation does not appear in the operations history and no action on the resource is performed
- We get database timeout exceptions due to
org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ORA-02049on any operations - Could not access to JON anymore
- I can't access to admin console anymore
- Cannot connect to the database
-
Server log contains warnings similar to:
WARN [com.arjuna.ats.jta] (pool-7-thread-1) ARJUNA016061: TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff928b7363:190191d7:5304e076:46c, node_name=1, branch_uid=0:ffff928b7363:190191d7:5304e076:470, subordinatenodename=null, eis_name=java:jboss/datasources/RHQDS >: oracle.jdbc.xa.OracleXAException at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1110) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0] at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:240) [ojdbc6-11.2.0.3.0.jar:11.2.0.3.0] at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:259) at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.start(XAResourceWrapperImpl.java:203) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636) [jbossjts-jacorb-4.17.7.Final-redhat-4.jar:4.17.7.Final-redhat-4] -
One of the follow errors appear in server.log:
ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@7453f2a5[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d connection handles=0 lastUse=1389866415155 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@7f02f75b pool internal context=SemaphoreArrayListManagedConnectionPool@74a75430[pool=RHQDS] xaResource=XAResourceWrapperImpl@1bbdb882[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@335f6a4d pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null] ERROR [org.jboss.as.ejb3] (EJB default - 2) javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (pool-7-thread-1) javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null] ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 2) ERROR: current transaction is aborted, commands ignored until end of transaction block ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component ServerManagerBean for method public abstract boolean org.rhq.enterprise.server.cloud.instance.ServerManagerLocal.getAndClearServerStatus(): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component StatusManagerBean for method public abstract java.util.List org.rhq.enterprise.server.cloud.StatusManagerLocal.getAndClearAgentsWithStatusForServer(java.lang.String): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not extract ResultSet -
Errors similar to the following in the log:
ERROR [org.jboss.as.ejb3.invocation] (pool-7-thread-1) JBAS014134: EJB Invocation failed on component ServerPluginsBean for method public abstract org.rhq.core.domain.plugin.ServerPlugin org.rhq.enterprise.server.plugin.ServerPluginsLocal.getServerPlugin(org.rhq.core.domain.plugin.PluginKey): javax.ejb.EJBException: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:250) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] ... Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked throwable in managedConnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@46c7a60[state=NORMAL managed connection=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 connection handles=0 lastUse=1392828664075 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject@2380d12b pool internal context=SemaphoreArrayListManagedConnectionPool@5197c8eb[pool=RHQDS] xaResource=XAResourceWrapperImpl@5e247042[xaResource=org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@6dd636e4 pad=false overrideRmValue=null productName=Oracle productVersion=Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options jndiName=java:jboss/datasources/RHQDS] txSync=null] ... Caused by: javax.transaction.SystemException: IJ000356: Failed to enlist: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 0:ffff928b7363:190191d7:5304e076:46c status: ActionStatus.ABORT_ONLY > ... ERROR [org.jboss.as.ejb3.invocation] (EJB default - 2) JBAS014134: EJB Invocation failed on component TopologyManagerBean for method public abstract org.rhq.core.domain.cloud.Server org.rhq.enterprise.server.cloud.TopologyManagerLocal.getServerByName(java.lang.String): javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:138) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:228) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:317) [jboss-as-ejb3-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10] ... Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection ... ERROR [org.quartz.core.ErrorLogger] (RHQScheduler_QuartzSchedulerThread) An error occured while firing trigger 'org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob.org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob': org.quartz.impl.jdbcjobstore.LockException: Failure obtaining db row lock: ERROR: canceling statement due to statement timeout [See nested exception: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout] at org.quartz.impl.jdbcjobstore.StdRowLockSemaphore.executeSQL(StdRowLockSemaphore.java:112) [quartz-1.6.5.jar:1.6.5] at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:112) [quartz-1.6.5.jar:1.6.5] at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3753) [quartz-1.6.5.jar:1.6.5] ... Caused by: org.postgresql.util.PSQLException: ERROR: canceling statement due to statement timeout at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) ...
Environment
- Red Hat JBoss Operations Network (ON) 3.2.0
- Use of JBoss ON provisioning bundles
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
