Content discovery and merge operations can fail when multiple content reports are being processed concurrently due to database table deadlocks

Solution Verified - Updated -

Issue

  • Getting Warning:JONUSR - 5 sessions are blocked by TM lock alert periodically
  • ORA-00060: deadlock detected while waiting for resource
  • Content discovery and merge periodically fails
  • Server log contains the following errors:

    WARN  [org.hibernate.util.JDBCExceptionReporter] SQL Error: 60, SQLState: 61000
    ERROR [org.hibernate.util.JDBCExceptionReporter] ORA-00060: deadlock detected while waiting for resource
    ERROR [org.hibernate.event.def.AbstractFlushingEventListener] Could not synchronize database state with session
    org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
        at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:87)
        at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
        ...
        at $Proxy435.mergeDiscoveredPackages(Unknown Source)
        at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:56)
        ...
    Caused by: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
        at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10345)
        at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:230)
        ...
    WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_2] TwoPhaseCoordinator.beforeCompletion - failed for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@3a573ecb
    javax.persistence.PersistenceException: org.hibernate.exception.LockAcquisitionException: Could not execute JDBC batch update
        at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:629)
        at org.hibernate.ejb.AbstractEntityManagerImpl$1.beforeCompletion(AbstractEntityManagerImpl.java:524)
        ...
        at $Proxy435.mergeDiscoveredPackages(Unknown Source)
        at org.rhq.enterprise.server.content.ContentServerServiceImpl.mergeDiscoveredPackages(ContentServerServiceImpl.java:56)
        ...
    

Environment

  • Red Hat JBoss Operations Network (ON) 3.1
  • JBoss ON server's rhq.server.concurrency-limit.content-report property is set to 2 or more
  • Two or more content discovery or update reports are being sent by one or more agents concurrently

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content