The JGroups Flow Control (FC) Protocol and Deadlocks with Web Session Replication in JBoss
Issue
- Most or all JBoss http, https, or ajp threads appear to be stuck or blocked in the JGroups FC protocol:
"http-localhost-8080-19" daemon prio=1 tid=0x0000000043f460f0 nid=0x7d76 in Object.wait() [0x000000004aabd000..0x000000004aabec10] at java.lang.Object.wait(Native Method) at EDU.oswego.cs.dl.util.concurrent.CondVar.timedwait(CondVar.java:222) - locked <0x00002b5feef4a068> (a EDU.oswego.cs.dl.util.concurrent.CondVar) at org.jgroups.protocols.FC.handleDownMessage(FC.java:454) at org.jgroups.protocols.FC.down(FC.java:374) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499) at org.jgroups.protocols.FC.receiveDownEvent(FC.java:368) at org.jgroups.stack.Protocol.passDown(Protocol.java:533) at org.jgroups.protocols.FRAG2.down(FRAG2.java:167) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499) at org.jgroups.stack.Protocol.passDown(Protocol.java:533) at org.jgroups.protocols.pbcast.STATE_TRANSFER.down(STATE_TRANSFER.java:294) at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499) at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:390) at org.jgroups.JChannel.down(JChannel.java:1230) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:792) at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.passDown(MessageDispatcher.java:769) at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:304) at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:444) at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:193) at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:432) at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:192) at sun.reflect.GeneratedMethodAccessor151.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.cache.TreeCache.callRemoteMethodsViaReflection(TreeCache.java:4484) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4433) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4386) at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4504) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110) at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88) at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:119) at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:379) at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:174) at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68) at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167) at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5934) at org.jboss.cache.TreeCache.put(TreeCache.java:3788) at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy175.put(Unknown Source) at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:138) at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:325) at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:125) - locked <0x00002b60227853b8> (a org.jboss.web.tomcat.service.session.SessionBasedClusteredSession) at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1153) at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:702) - locked <0x00002b60227853b8> (a org.jboss.web.tomcat.service.session.SessionBasedClusteredSession) ...
-
The "IncomingPacketHandler (channel=Tomcat-CLUSTER-PROD)" thread is waiting to lock a org.jboss.web.tomcat.service.session.SessionBasedClusteredSession monitor held by an http, https, or ajp connector thread:
"IncomingPacketHandler (channel=Tomcat-CLUSTER-PROD)" daemon prio=1 tid=0x00002aaaad5b6e60 nid=0x73e6 waiting for monitor entry [0x000000004697b000..0x000000004697dc90] at org.jboss.web.tomcat.service.session.ClusteredSession.expire(ClusteredSession.java:818) - waiting to lock <0x00002b60227853b8> (a org.jboss.web.tomcat.service.session.SessionBasedClusteredSession) ... -
JBoss is unresponsive and "java.lang.OutOfMemoryError: Java heap space" appears in the JBoss
server.log. - The heap object histogram shows retention in org.jgroups.protocols.TP$IncomingQueueEntry.
- AJP Thread pool lockup, We have 2 servers in a cluster one of the servers has locked up the thread dump is below. The lock appears to be around the replication of HTTPSessions and in the FC protocol.
One server is logging repeatedly:
WARN [org.jgroups.protocols.FC] (IncomingPacketHandler (channel=Tomcat-PresentationPartition):) Received two credit requests from 10.252.1.10:34333 without any intervening messages; sending 1999936 credits
The other is logging repeatedly:
WARN [org.jboss.cache.TreeCache](ajp-sgz100-sap8%2F192.168.1.10-8009-6:) node /JSESSION/localhost/test?online/82ninyZauN+3eJieuTfVHQ** not found
Environment
- JBoss Enterprise Application Platform (EAP)
- 4.2
- 4.3
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
