Deadlock between org.apache.coyote.http11.Http11NioProcessor.writeEvent and org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible in EAP 6

Solution Unverified - Updated -

Issue

Our WebSocket application faced the following deadlock between org.apache.coyote.http11.Http11NioProcessor#writeEvent() and org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer#onWritePossible() in EAP 6 :

Found one Java-level deadlock:
=============================
"WebSocketServer-/XXXXX-XX":
  waiting to lock monitor 0x00007fb26c1f9e68 (object 0x000000040631dd00, a org.apache.coyote.http11.Http11NioProcessor),
  which is held by "http-/0.0.0.0:8080-XXX"
"http-/0.0.0.0:8080-XXX":
  waiting to lock monitor 0x00007fb16819df98 (object 0x00000004127d7d00, a java.lang.Object),
  which is held by "WebSocketServer-/XXXXX-XX"

Java stack information for the threads listed above:
===================================================
"WebSocketServer-/XXXXX-XX":
    at org.apache.coyote.http11.Http11NioProcessor.writeEvent(Http11NioProcessor.java:760)
    - waiting to lock <0x000000040631dd00> (a org.apache.coyote.http11.Http11NioProcessor)
    at org.apache.coyote.http11.Http11NioProcessor.action(Http11NioProcessor.java:859)
    at org.apache.coyote.Response.action(Response.java:188)
    at org.apache.catalina.connector.OutputBuffer.lastWrite(OutputBuffer.java:392)
    at org.apache.catalina.connector.CoyoteOutputStream.isReady(CoyoteOutputStream.java:112)
    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:89)
    - locked <0x00000004127d7d00> (a java.lang.Object)
    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:75)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:445)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.startMessage(WsRemoteEndpointImplBase.java:335)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.write(WsRemoteEndpointImplBase.java:741)
    - locked <0x00000004127d7d00> (a java.lang.Object)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$TextMessageSendHandler.onResult(WsRemoteEndpointImplBase.java:757)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.endMessage(WsRemoteEndpointImplBase.java:371)
    at org.apache.tomcat.websocket.WsRemoteEndpointImplBase$EndMessageHandler.onResult(WsRemoteEndpointImplBase.java:483)
    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer$OnResultRunnable.run(WsRemoteEndpointImplServer.java:245)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
"http-/0.0.0.0:8080-XXX":
    at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.onWritePossible(WsRemoteEndpointImplServer.java:81)
    - waiting to lock <0x00000004127d7d00> (a java.lang.Object)
    at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsWriteListener.onWritePossible(WsHttpUpgradeHandler.java:243)
    at org.apache.catalina.core.StandardWrapperValve.async(StandardWrapperValve.java:605)
    at org.apache.catalina.core.StandardWrapperValve.event(StandardWrapperValve.java:350)
    at org.apache.catalina.core.StandardContextValve.event(StandardContextValve.java:171)
    at org.apache.catalina.valves.ValveBase.event(ValveBase.java:185)
    at org.apache.catalina.core.StandardHostValve.event(StandardHostValve.java:252)
    at org.apache.catalina.valves.ValveBase.event(ValveBase.java:185)
    at org.apache.catalina.core.StandardEngineValve.event(StandardEngineValve.java:121)
    at org.apache.catalina.connector.CoyoteAdapter.event(CoyoteAdapter.java:228)
    at org.apache.coyote.http11.Http11NioProcessor.event(Http11NioProcessor.java:232)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.event(Http11NioProtocol.java:819)
    - locked <0x000000040631dd00> (a org.apache.coyote.http11.Http11NioProcessor)
    at org.apache.tomcat.util.net.NioEndpoint$ChannelProcessor.run(NioEndpoint.java:940)
    - locked <0x00000004127c9560> (a java.lang.Object)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Found 1 deadlock.

We previously faced the different deadlock issue reported in this knowledge and upgraded to 6.4.7 onwards. However, we still face this new deadlock issue when load is high on WebSocket application.

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.4.14 or earlier

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.