Pool contention while "JCA PoolFiller" runs in JBoss EAP 5.2

Solution Verified - Updated -

Issue

  • JBoss update from EAP 5.1 CP2 to EAP 5.2 GA
  • A datasource pool has a high min-pool-size and filling the pool may take significant time due to external factors
  • A large number of (idle) connection timeouts occur after some period of low load
  • System performance/throughput is impacted during a period where the "JCA PoolFiller" thread can be seen (in thread dumps) to be re-filling the pool to min-pool-size connections
  • Many threads are attempting to obtain connections and repeated thread dumps show a high number of threads with the stack trace below

    java.lang.Thread.State: BLOCKED (on object monitor)
        at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:497)
        - waiting to lock <0x0000040024cb8028> (a org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory)
        at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:388)
        at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:225)
        at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:689)
        at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
        at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
        at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
        at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
        at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:111)
    
  • The "JCA PoolFiller" is seen to be holding a lock on the connection factory while attempting to establish a database connection (which may take some time depending on the performance of the network and/or the database server)

    "JCA PoolFiller" prio=5 tid=0x440008800000 nid=0xaf7e  [ JVM locked by VM running, polling bits: safep ]
       java.lang.Thread.State: RUNNABLE
            ... <STACK FRAMES SHOWING DATABASE DRIVER CONNECT> ...
            at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:207)
            at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:195)
            - locked <0x0000040024cb8028> (a org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory)
            at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:650)
            at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:558)
            at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
            at java.lang.Thread.run(Thread.java:748)
    
  • After some time the "JCA PoolFiller" thread completes and performance normalizes

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 5.2.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In