Tomcat comes to a halt with threads waiting on GenericObjectPool
Issue
Tomcat became unresponsive and thread dumps showed many threads waiting to lock a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool. The waiting threads look like:
"ajp-8009-1" daemon prio=10 tid=0x0896e400 nid=0x380e waiting for monitor entry [0x612b9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
- waiting to lock <0x75df1678> (a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
The thread holding the lock looks like:
"ajp-8009-2" daemon prio=10 tid=0x08dadc00 nid=0x3673 runnable [0x62bab000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at oracle.net.ns.Packet.receive(Packet.java:283)
at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)
at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:498)
- locked <0x7c45fc88> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3931)
- locked <0x7c45fc88> (a oracle.jdbc.driver.T4CConnection)
at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close(DelegatingConnection.java:159)
at org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose(PoolableConnection.java:95)
at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject(PoolableConnectionFactory.java:301)
at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:946)
at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:917)
- locked <0x75df1678> (a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
at org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:80)
- locked <0x7c45fc00> (a org.apache.tomcat.dbcp.dbcp.PoolableConnection)
at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)
- Issue about connection pool configuration. We have sometime issues on our JBoss web server based on tomcat 6 and JVM 1.6.0_45.Particularly we noticed lost of connection with the db (Oracle 11.2.04) with subsequent hang of Tomcat (JbossWS).
Environment
- JBoss Enterprise Web Server (EWS)
- Tomcat 6
- Oracle
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.