EAP 6 NIO deadlocks during blocking executor exhaustion
Issue
- We use an HTTP NIO connector on EAP 6 with a
blocking-bounded-queue-thread-pool. - JBoss became unresponsive and thread dumps showed most threads blocked like so in
ChannelProcessor.run:
"http-executor-threads - 4003" prio=10 tid=0x00007fb688bcd800 nid=0x6bd5 waiting for monitor entry [0x00007fb65ca74000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:939)
- waiting to lock <0x00000007970dd278> (a java.lang.Object)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
- The remaining threads and the lock holders looked like the following:
"http-executor-threads - 4004" prio=10 tid=0x00007fb6884fc000 nid=0x6bd6 waiting on condition [0x00007fb6802d1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078f1f6d88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at org.jboss.threads.QueueExecutor.execute(QueueExecutor.java:191)
at org.jboss.threads.DelegatingBlockingExecutorService.execute(DelegatingBlockingExecutorService.java:42)
at org.jboss.as.threads.ManagedExecutorService.execute(ManagedExecutorService.java:64)
at org.apache.tomcat.util.net.NioEndpoint.processChannel(NioEndpoint.java:468)
at org.apache.tomcat.util.net.NioEndpoint$EventPoller.add(NioEndpoint.java:1153)
at org.apache.tomcat.util.net.NioEndpoint.addEventChannel(NioEndpoint.java:411)
at org.apache.tomcat.util.net.NioEndpoint.addEventChannel(NioEndpoint.java:385)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.event(Http11NioProtocol.java:875)
- locked <0x0000000794a02610> (a org.apache.coyote.Request)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.event(Http11NioProtocol.java:872)
- locked <0x0000000794a02610> (a org.apache.coyote.Request)
at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:940)
- locked <0x0000000797097670> (a java.lang.Object)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:849)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Environment
- JBoss Enterprise Application Platform (EAP) 6.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
