JWS 5 Tomcat deadlocks across several request threads in java NIO calls

Solution Verified - Updated -

Issue

  • Tomcat became unresponsive and thread dumps show many request threads blocked in java.nio.channels.spi.AbstractSelector.cancel from a deadlock like below:
Found one Java-level deadlock:
=============================
"http-nio-8080-exec-455":
  waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
  which is held by "http-nio-8080-ClientPoller"
"http-nio-8080-ClientPoller":
  waiting to lock monitor 0x00007f59680af4d8 (object 0x00000006de810f10, a java.lang.Object),
  which is held by "http-nio-8080-exec-215"
"http-nio-8080-exec-215":
  waiting to lock monitor 0x00007f59c4057a98 (object 0x00000006c0b27028, a java.util.HashSet),
  which is held by "http-nio-8080-BlockPoller"
"http-nio-8080-BlockPoller":
  waiting to lock monitor 0x00007f5998946158 (object 0x00000006de8588c0, a java.lang.Object),
  which is held by "http-nio-8080-exec-378"
"http-nio-8080-exec-378":
  waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
  which is held by "http-nio-8080-ClientPoller"

Java stack information for the threads listed above:
===================================================
"http-nio-8080-exec-455":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de870a68> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de870a90> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de870a28> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c3ca9be8> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de825cc0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-ClientPoller":
    at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
    - waiting to lock <0x00000006de810f10> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
    at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
    at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
    - locked <0x00000006c0b4fe28> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c0b50068> (a sun.nio.ch.Util$3)
    - locked <0x00000006c0b50058> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c0b4fdd0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:708)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-exec-215":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b27028> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de810c70> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de810f10> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de810c30> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c2d7b8a0> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de810c98> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-BlockPoller":
    at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
    - waiting to lock <0x00000006de8588c0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
    at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
    at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
    - locked <0x00000006c0b27028> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c0b27108> (a sun.nio.ch.Util$3)
    - locked <0x00000006c0b270f8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c0b26fc0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"http-nio-8080-exec-378":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de858648> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de8588c0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de8585e0> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c3360628> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de858670> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Environment

  • JBoss Web Server (JWS)
    • 5.2.0
    • 5.3.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content