Pool contention while "JCA PoolFiller" runs in JBoss EAP 5.2

Issue

• JBoss update from EAP 5.1 CP2 to EAP 5.2 GA
• A datasource pool has a high min-pool-size and filling the pool may take significant time due to external factors
• A large number of (idle) connection timeouts occur after some period of low load
• System performance/throughput is impacted during a period where the "JCA PoolFiller" thread can be seen (in thread dumps) to be re-filling the pool to min-pool-size connections

• Many threads are attempting to obtain connections and repeated thread dumps show a high number of threads with the stack trace below

  java.lang.Thread.State: BLOCKED (on object monitor)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:497)
      - waiting to lock <0x0000040024cb8028> (a org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory)
      at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.matchManagedConnections(LocalManagedConnectionFactory.java:388)
      at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:225)
      at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:689)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
      at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:111)
  

• The "JCA PoolFiller" is seen to be holding a lock on the connection factory while attempting to establish a database connection (which may take some time depending on the performance of the network and/or the database server)

  "JCA PoolFiller" prio=5 tid=0x440008800000 nid=0xaf7e  [ JVM locked by VM running, polling bits: safep ]
     java.lang.Thread.State: RUNNABLE
          ... <STACK FRAMES SHOWING DATABASE DRIVER CONNECT> ...
          at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:207)
          at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:195)
          - locked <0x0000040024cb8028> (a org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory)
          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:650)
          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:558)
          at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
          at java.lang.Thread.run(Thread.java:748)
  

• After some time the "JCA PoolFiller" thread completes and performance normalizes