Content discovery and merge operations can fail when multiple content reports are being processed concurrently due to database table deadlocks
Issue
- Getting
Warning:JONUSR - 5 sessions are blocked by TM lock
alert periodically ORA-00060: deadlock detected while waiting for resource
- Content discovery and merge periodically fails
-
Server log contains the following errors:
WARN [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000 ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:87) at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43) ... at $Proxy435.mergeDiscoveredPackages(Unknown Source) at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:56) ... Caused by: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10345) at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230) ... WARN [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@3a573ecb javax.persistence.PersistenceException: org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629) at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:524) ... at $Proxy435.mergeDiscoveredPackages(Unknown Source) at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:56) ...
Environment
- Red Hat JBoss Operations Network (ON) 3.1
- JBoss ON server's
rhq.server.concurrency-limit.content-report
property is set to2
or more - Two or more content discovery or update reports are being sent by one or more agents concurrently
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.