How to detect leaked datasource connections using the cached connection manager (CCM) debug facility in JBoss EAP
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 6
- 7
Issue
- How can I detect leaked datasource connections?
- How can I identify code which leaks datasource connections?
- "IJ000453: Unable to get managed connection for ..."
- "IJ000655: No managed connections available within configured blocking timeout ..."
- Application is not able to obtain a new connection from a connection pool but
INACTIVEsession count (reported by the database server) is increasing. - Does JBoss close a connection leak automatically?
Resolution
To enable the cached connection manager (CCM) to identify a connection leak:
-
Enable the CCM for the datasource. It defaults to
trueif it is not explicitly specified but you may setuse-ccm="true"explicitly.<subsystem xmlns="urn:jboss:domain:datasources:1.1"> <datasources> <datasource ... enabled="true" use-ccm="true"> ... </datasource> </datasources> </subsystem> -
Verify that
<cached-connection-manager>exists in the jca subsystem and setdebug="true".<subsystem xmlns="urn:jboss:domain:jca:1.1"> ... <cached-connection-manager debug="true" error="false"/> ... </subsystem>-
CLI commands
- NOTE : while CLI may be run to configure running servers, a restart should be performed for the server where debug is required to ensure the debug setting is applied to the connection pool
- Standalone Mode
/subsystem=jca/cached-connection-manager=cached-connection-manager/:write-attribute(name=debug,value=true)- Domain Mode
/profile=<your_profile_here>/subsystem=jca/cached-connection-manager=cached-connection-manager/:write-attribute(name=debug,value=true)` -
Setting
debug="true"will:- Log an
INFOmessage indicating that JBoss is "Closing a connection for you. Please close them yourself" - Generate a stacktrace for the code where the leaked connection was first opened.
- Close the leaked connection
- Log an
-
-
The additional property
errormay be used to raise aRuntimeExceptionand generate anERRORmessage in the log when it is set to true. - NOTE :
jtaMUST also be set totruefor the datasource in order for cached connection manager debug to work. - Restart JBoss to ensure that the changes are applied.
Examples
Example with debug="true" without error="true" (i.e. <cached-connection-manager debug="true" /> or <cached-connection-manager debug="true" error="false" />):
INFO [org.jboss.jca.core.api.connectionmanager.ccm.CachedConnectionManager] (http-/127.0.0.1:8080-1) IJ000100: Closing a connection for you. Please close them yourself: org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@6f1170a9: java.lang.Throwable: STACKTRACE
at org.jboss.jca.core.connectionmanager.ccm.CachedConnectionManagerImpl.registerConnection(CachedConnectionManagerImpl.java:269)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:495)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
...
Example with debug="true" and error="true" (i.e. <cached-connection-manager debug="true" error="true"/>):
INFO [org.jboss.jca.core.api.connectionmanager.ccm.CachedConnectionManager] (http-/127.0.0.1:8080-1) IJ000100: Closing a connection for you. Please close them yourself: org.jboss.jca.adapters.jdbc.jdk6.WrappedConnectionJDK6@27c94e11: java.lang.Throwable: STACKTRACE
at org.jboss.jca.core.connectionmanager.ccm.CachedConnectionManagerImpl.registerConnection(CachedConnectionManagerImpl.java:269)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:495)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
...
ERROR [org.apache.catalina.connector.CoyoteAdapter] (http-/127.0.0.1:8080-1) An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.RuntimeException: javax.resource.ResourceException: IJ000151: Some connections were not closed, see the log for the allocation stacktraces
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
...
Caused by: java.lang.RuntimeException: javax.resource.ResourceException: IJ000151: Some connections were not closed, see the log for the allocation stacktraces
at org.jboss.as.connector.deployers.ra.processors.CachedConnectionManagerSetupProcessor$CachedConnectionManagerSetupAction.teardown(CachedConnectionManagerSetupProcessor.java:85)
at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:61) [jboss-as-web-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
... 8 more
Caused by: javax.resource.ResourceException: IJ000151: Some connections were not closed, see the log for the allocation stacktraces
at org.jboss.jca.core.connectionmanager.ccm.CachedConnectionManagerImpl.popMetaAwareObject(CachedConnectionManagerImpl.java:249)
at org.jboss.as.connector.deployers.ra.processors.CachedConnectionManagerSetupProcessor$CachedConnectionManagerSetupAction.teardown(CachedConnectionManagerSetupProcessor.java:83)
... 9 more
Additional Notes
- If leaks are believed to be present but nothing is reported by the CCM verify the following:
- The datasource must not be configured with
use-ccm="false" - The datasource must not be configured with
jta="false".- See this article for details on appropriate
jtaconfiguration for your datasources. - In the EAP 6.4 release, a new connection pool implementation called
LeakDumperManagedConnectionPoolhas been added. See Using the LeakDumperManagedConnectionPool for additional details. This is especially useful if you must use datasource pools for which thejtaproperty cannot be set totrue.
- See this article for details on appropriate
- The minimum logging level must be
INFOfororg.jboss.jca
- The datasource must not be configured with
- Activating debug will have some impact on performance and log file size and it is, therefore, recommended that the debug configuration be used only during development or test cycles.
- See also What causes 'Closing a connection for you... in EAP 6 which demonstrates proper closure of connection resources.
- After verifying that no leaks exist/remain, restore the configuration by removing the
debug="true"setting or using<cached-connection-manager debug="false"/>(before deploying in production). - The debug feature is not intended to be used as a resolution for connections leaked by application code.
- To track the usage of datasource connections you can also enable datasource logging
- Setting
errortotrueis not recommended for production deployment. Where debug is required in production deployments, settingdebug="true"should provide adequate information to diagnose the issue. - In addition to providing support for debugging, the CCM is also used to support lazy enlistment
- The CCM (without debug enabled) is enabled by default and its use is normal for production systems
Diagnostic Steps
Byteman Based Tracking of Connection Usage
It is also possible to dynamically enable Byteman trace and deploy the two rules below to trace connection request and release (back to the pool). The internal pooledId can be correlated with org.jboss.jca tracing which reports on the population of in-use (reserved by application logic) and unused (but pooled) connections. The maxFrames parameter (i.e. the last parameter) in the traceStack(...) calls may be adjusted to increase context to include additional application and JBoss stack frames.
RULE ConnectionManagerRegisterAssociation
CLASS AbstractConnectionManager
METHOD registerAssociation
AT ENTRY
IF callerEquals("WrapperDataSource.getConnection", true, 3)
# Change the frame count (last parameter below) to refine the size of the stack trace - e.g. change 20 to 4
DO traceStack("CONNECTION TRACE - getConnection(pool=" + $1.getPool().getName()
+ ") => " + $2 + "(pooledId=" + Integer.toHexString(System.identityHashCode($1)) + ")\n", 20);
ENDRULE
RULE ConnectionManagerUnregisterAssociation
CLASS AbstractConnectionManager
METHOD unregisterAssociation
AT ENTRY
IF callerEquals("WrappedConnection.close", true, 5)
# Change the frame count (last parameter below) to refine the size of the stack trace - e.g. change 20 to 6
DO traceStack("CONNECTION TRACE - closeConnection(" + $2
+ "(pooledId=" + Integer.toHexString(System.identityHashCode($1)) + ")) => pool="
+ $1.getPool().getName() + "\n", 20);
ENDRULE
Heap Examination
In looking at heap dumps, use the following OQL to examine key details of connection pools
-- active shows the number of open/established connections (including inUse and idle)
-- checkedOut shows the reserved connections (owned by application components/clients of the pool)
select p.pool.poolName.toString() AS poolName,
p.checkedOut.size AS inUse,
p.cls.size AS active,
p.poolConfiguration.maxSize.value.toString() AS maxSize,
p.poolConfiguration.minSize.value.toString() AS minSize,
p.poolConfiguration.strictMin.value.toString() AS useStrictMin,
p.poolConfiguration.prefill.value.toString() AS prefill
from org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool p
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
