Getting new database connections takes longer than expected in JBoss EAP 6.0/6.1

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.0.x
    • 6.1.x

Issue

  • Seeing high execution time for DataSource.getConnection()
  • A lot of threads are waiting to lock org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection or org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties

Resolution

Root Cause

There's a following know issue about concurrency bottleneck in JCA connection creation code:

Diagnostic Steps

  • Capture thread dumps at the time of issue

  • You will see many BLOCKED threads like the following:

   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:211)
    - waiting to lock <0x00000007020d97b8> (a org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:761)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:343)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:302)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
        ...(snip)...

   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:1027)
    - waiting to lock <0x00000007020d97b8> (a org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:490)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:282)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:302)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
        ...(snip)...

 java.lang.Thread.State: BLOCKED (on object monitor)
        at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:951)
        - waiting to lock <0x00000006e7208340> (a org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$MyXaMCF)
        at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:484)
        at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:283)
        at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:495)
        at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:374)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
        at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
        at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
        at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
        ...(snip)...

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments