JWS 5 Tomcat deadlocks across several request threads in java NIO calls
Issue
- Tomcat became unresponsive and thread dumps show many request threads blocked in
java.nio.channels.spi.AbstractSelector.cancel
from a deadlock like below:
Found one Java-level deadlock:
=============================
"http-nio-8080-exec-455":
waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
which is held by "http-nio-8080-ClientPoller"
"http-nio-8080-ClientPoller":
waiting to lock monitor 0x00007f59680af4d8 (object 0x00000006de810f10, a java.lang.Object),
which is held by "http-nio-8080-exec-215"
"http-nio-8080-exec-215":
waiting to lock monitor 0x00007f59c4057a98 (object 0x00000006c0b27028, a java.util.HashSet),
which is held by "http-nio-8080-BlockPoller"
"http-nio-8080-BlockPoller":
waiting to lock monitor 0x00007f5998946158 (object 0x00000006de8588c0, a java.lang.Object),
which is held by "http-nio-8080-exec-378"
"http-nio-8080-exec-378":
waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
which is held by "http-nio-8080-ClientPoller"
Java stack information for the threads listed above:
===================================================
"http-nio-8080-exec-455":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006de870a68> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006de870a90> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006de870a28> (a java.lang.Object)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
- locked <0x00000006c3ca9be8> (a org.apache.tomcat.util.net.NioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006de825cc0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-ClientPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006de810f10> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c0b4fe28> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c0b50068> (a sun.nio.ch.Util$3)
- locked <0x00000006c0b50058> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c0b4fdd0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:708)
at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-exec-215":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c0b27028> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006de810c70> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006de810f10> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006de810c30> (a java.lang.Object)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
- locked <0x00000006c2d7b8a0> (a org.apache.tomcat.util.net.NioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006de810c98> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-BlockPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006de8588c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c0b27028> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c0b27108> (a sun.nio.ch.Util$3)
- locked <0x00000006c0b270f8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c0b26fc0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"http-nio-8080-exec-378":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006de858648> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006de8588c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006de8585e0> (a java.lang.Object)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
- locked <0x00000006c3360628> (a org.apache.tomcat.util.net.NioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006de858670> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
Environment
- JBoss Web Server (JWS)
- 5.2.0
- 5.3.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.