JBoss EAP 7 does not process requests for a session in parallel

Solution Verified - Updated -

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 7.x

Issue

  • When a SOAP call is called while an async servlet call is hanging, the SOAP call fails to aquire a lock on the HTTP session.

  • Errors when a request times out waiting on a session lock:

    ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-79) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmn and requestor GlobalTransaction::45:local. Lock is held by GlobalTransaction::43:local

  • Getting exception while accessing session

    org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmn) and requestor GlobalTx:node1:1001. Lock is held by GlobalTx:node1:1000

  • JBoss EAP 7 only processes a single request for a given session at a time.

Resolution

You can allow concurrent access by modifying the web session replication cache configuration to remove <locking> and <transaction> completely. It is also necessary to remove the "mode" attribute to use the default setting of SYNC in EAP 7.1+.

  • For ha based configurations (standalone-ha.xml / standalone-full-ha.xml):

    <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
    <transport lock-timeout="60000"/>
    <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">  <!-- IN 7.1+ REMOVE mode="..." IF PRESENT -->
        <locking isolation="REPEATABLE_READ"/>                               <!-- REMOVE THIS LINE -->
        <transaction mode="BATCH"/>                                          <!-- REMOVE THIS LINE -->
        <file-store/>
    </distributed-cache>
</cache-container>

    You can execute the following CLI:

    /subsystem=infinispan/cache-container=web/distributed-cache=dist/component=locking:remove
/subsystem=infinispan/cache-container=web/distributed-cache=dist/component=transaction:remove

    If you are using a <replicated-cache> or <local-cache> instead of <distributed-cache> for web sessions, you'll still remove the same <locking> and <transaction> tags from the corresponding cache configuration, and in the CLI command change /distributed-cache=dist/ to the appropriate type and cache name.

NOTE : However there were bugs that prevent the above configuration from working in EAP 7.0.3 or before. It has been fixed in EAP 7.0.4 (JBEAP-6752) and 7.1.0 (JBEAP-4128) or later, so you can configure the above setting in the newer versions.

Root Cause

  • The servlet spec requires that requests for a session are only processed on a single node at once.
  • EAP 7.0 implements this by only processing a single request for a session at once cluster-wide.

Related article

How to configure and tune the session replication for EAP

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments