JBoss EAP 7 does not process requests for a session in parallel

Solution Verified - Updated -

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 7.x

Issue

  • When a SOAP call is called while an async servlet call is hanging, the SOAP call fails to aquire a lock on the HTTP session.
  • Errors when a request times out waiting on a session lock:

    ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-79) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmn and requestor GlobalTransaction::45:local. Lock is held by GlobalTransaction::43:local
    
  • JBoss EAP 7 only processes a single request for a given session at a time.

Resolution

You can allow concurrent access by modifying the web session replication cache configuration to remove <locking> and <transaction> completely. It is also necessary to remove the "mode" attribute to use the default setting of SYNC in EAP 7.1+.

  • For ha based configurations (standalone-ha.xml / standalone-full-ha.xml), change from:

        <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
            <transport lock-timeout="60000"/>
            <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">
                <locking isolation="REPEATABLE_READ"/>
                <transaction mode="BATCH"/>
                <file-store/>
            </distributed-cache>
        </cache-container>
    

    to:

        <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
            <transport lock-timeout="60000"/>
            <distributed-cache name="dist" l1-lifespan="0" owners="2">
                <file-store/>
            </distributed-cache>
        </cache-container>
    

    You can execute the following CLI:

    /subsystem=infinispan/cache-container=web/distributed-cache=dist/component=locking:remove
    /subsystem=infinispan/cache-container=web/distributed-cache=dist/component=transaction:remove
    

    If you are using other cache definition <replicated-cache> instead of <distributed-cache>, apply the above configuration change to your cache setting.

  • For local (non replicated) caches in non-ha based configurations (standalone.xml / standalone-full.xml) , change from:

        <cache-container name="web" default-cache="passivation" module="org.wildfly.clustering.web.infinispan">
            <local-cache name="passivation">
                <locking isolation="REPEATABLE_READ"/>
                <transaction mode="BATCH"/>
                <file-store passivation="true" purge="false"/>
            </local-cache>
            <local-cache name="persistent">
                <locking isolation="REPEATABLE_READ"/>
                <transaction mode="BATCH"/>
                <file-store passivation="false" purge="false"/>
            </local-cache>
        </cache-container>
    

    to:

        <cache-container name="web" default-cache="passivation" module="org.wildfly.clustering.web.infinispan">
            <local-cache name="passivation">
                <file-store passivation="true" purge="false"/>
            </local-cache>
            <local-cache name="persistent">
                <file-store passivation="false" purge="false"/>
            </local-cache>
        </cache-container>
    

    You can execute the following CLI:

    /subsystem=infinispan/cache-container=web/local-cache=passivation/component=locking:remove
    /subsystem=infinispan/cache-container=web/local-cache=passivation/component=transaction:remove
    /subsystem=infinispan/cache-container=web/local-cache=persistent/component=locking:remove
    /subsystem=infinispan/cache-container=web/local-cache=persistent/component=transaction:remove
    

NOTE : However there were bugs that prevent the above configuration from working in EAP 7.0.3 or before. It has been fixed in EAP 7.0.4 (JBEAP-6752) and 7.1.0 (JBEAP-4128) or later, so you can configure the above setting in the newer versions.

Root Cause

  • The servlet spec requires that requests for a session are only processed on a single node at once.
  • EAP 7.0 implements this by only processing a single request for a session at once cluster-wide.

Related article

How to configure and tune the session replication for EAP

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

1 Comments

Is this single threaded behavior true in all cases or just when using infinispan session replication?