Infinispan acquireSessionOwnership contention with distributed cache

Solution Unverified - Updated -

Issue

  • Many ajp threads waiting to acquire session ownership with a stack trace as follows:
  • Corresponding Server log will also show the Exceptions as below.
ajp-ABC/1.2.3.4:8009-123" daemon prio=10 tid=0x00007fdae6491800 nid=0x7a07 waiting on condition [0x00007fda7b937000]
       java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006a018c6a8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:894)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1221)
        at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
        at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:517)
        at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:488)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2638)
        at org.apache.catalina.connector.Request.getSession(Request.java:2382)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:490)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
        at java.lang.Thread.run(Thread.java:744)

mm/dd/yyyy hh:mm:ss,226 ERROR [org.apache.catalina.connector] (ajp-server-name/X.X.X.X:8170-25) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of KnxvBL8qOoKLtx0Ka9OF97dn
        at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:494)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:2616)
        at org.apache.catalina.connector.Request.getSession(Request.java:2361)
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336)
        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488)
        at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920)
        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010213: Cannot acquire lock default-host/ROOT/KnxvBL8qOoKLtx0Ka9OF97dn from cluster
        at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)
        at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:400)
        at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520)
        at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:492)
         12 more

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 6
    • 7
  • Infinispan distributed session cache
  • Load balancer that uses JSESSIONID to maintain session stickiness (mod_jk, mod_proxy, mod_cluster)

  • Load balancer cannot access all members of the session cluster

    • For example, a 2-stage load balancer is forwarding requests to two separate EAP groups, which are part of a single Infinispan cluster.

                    |--EAP1
              |
    |-httpd1--|
    |         |
    |         |--EAP2
    |
F5--|
    |         |--EAP3
    |         |
    |-httpd2--|
              |
              |--EAP4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content