Spring and jBPM6 integration causes deadlock on TaskTransactionInterceptor and SingleSessionCommandService instances during high concurrency situation

Issue

• The following two threads have been observed to be deadlocked involving jBPM APIs from BPMS 6.0.3 release, while the application code using those APIs also uses Spring framework as well. One thread is trying to complete a task which got the lock of TaskTransactionInterceptor, but also needed lock of SingleSessionCommandService . Another thread is trying to start a process and it got the lock of SingleSessionCommandService, but need lock of TaskTransactionInterceptor .

...
  Thread "WebContainer : 36" (0x0000000009442B00)
    is waiting for:
      sys_mon_t:0x00007F56A9E22520 infl_mon_t: 0x00007F56A9E22590:
      org/drools/persistence/SingleSessionCommandService@0x0000000111CB93E8/0x0000000111CB93F4: 
    which is owned by:
  Thread "WebContainer : 25" (0x0000000002022F00)
    which is waiting for:
      sys_mon_t:0x00007F56ED6BE150 infl_mon_t: 0x00007F56ED6BE1C0:
      org/jbpm/services/task/persistence/TaskTransactionInterceptor@0x000000011B911440/0x000000011B91144C: 
    which is owned by:
  Thread "WebContainer : 36" (0x0000000009442B00)
...

Investigating the thread dumps in detail reveal the following information as well. One of the threads is attempting to complete a task using CommandBasedTaskService API.

Thread Name   WebContainer : 36
State         Deadlock/Blocked
Monitor       Owns Monitor Lock on org/jbpm/services/task/persistence/TaskTransactionInterceptor@0x000000011B911440/0x000000011B91144C 
Waiting for Monitor Lock on org/drools/persistence/SingleSessionCommandService@0x0000000111CB93E8/0x0000000111CB93F4

Java Stack

at org/drools/core/command/impl/CommandBasedStatefulKnowledgeSession.getEnvironment(CommandBasedStatefulKnowledgeSession.java:494) 
at org/jbpm/services/task/wih/NonManagedTaskEventListener.processTaskState(NonManagedTaskEventListener.java:37) 
at org/jbpm/services/task/wih/NonManagedTaskEventListener.afterTaskCompletedEvent(NonManagedTaskEventListener.java:182) 
at org/jbpm/services/task/events/TaskEventSupport.fireAfterTaskCompleted(TaskEventSupport.java:189) 
at org/jbpm/services/task/internals/lifecycle/MVELLifeCycleManager.taskOperation(MVELLifeCycleManager.java:340) 
at org/jbpm/services/task/identity/UserGroupLifeCycleManagerDecorator.taskOperation(UserGroupLifeCycleManagerDecorator.java:46) 
at org/jbpm/services/task/impl/TaskInstanceServiceImpl.complete(TaskInstanceServiceImpl.java:142) 
at org/jbpm/services/task/commands/CompleteTaskCommand.execute(CompleteTaskCommand.java:70) 
at org/jbpm/services/task/commands/CompleteTaskCommand.execute(CompleteTaskCommand.java:37) 
at org/jbpm/services/task/commands/CompositeCommand.execute(CompositeCommand.java:38) 
at org/jbpm/services/task/commands/TaskCommandExecutorImpl$SelfExecutionCommandService.execute(TaskCommandExecutorImpl.java:65) 
at org/drools/core/command/impl/AbstractInterceptor.executeNext(AbstractInterceptor.java:41) 
at org/jbpm/services/task/persistence/TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:54) 
at org/jbpm/services/task/commands/TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) 
at org/jbpm/services/task/impl/command/CommandBasedTaskService.complete(CommandBasedTaskService.java:144) 
...

On the other hand the other thread attempts to execute and complete a domain specific task using "JPAWorkItemManager" API.

Thread Name    WebContainer : 25
State          Deadlock/Blocked
Monitor        Owns Monitor Lock on org/drools/persistence/SingleSessionCommandService@0x0000000111CB93E8/0x0000000111CB93F4 
Waiting for Monitor Lock on org/jbpm/services/task/persistence/TaskTransactionInterceptor@0x000000011B911440/0x000000011B91144C

Java Stack
at org/jbpm/services/task/commands/TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) 
at org/jbpm/services/task/impl/command/CommandBasedTaskService.addTask(CommandBasedTaskService.java:471) 
at org/jbpm/services/task/wih/NonManagedLocalHTWorkItemHandler.executeWorkItem(NonManagedLocalHTWorkItemHandler.java:99) 
at org/drools/persistence/jpa/processinstance/JPAWorkItemManager.internalExecuteWorkItem(JPAWorkItemManager.java:56) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.internalTrigger(WorkItemNodeInstance.java:124) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.trigger(NodeInstanceImpl.java:155) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.triggerNodeInstance(NodeInstanceImpl.java:337) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.triggerCompleted(NodeInstanceImpl.java:296) 
at org/jbpm/workflow/instance/impl/ExtendedNodeInstanceImpl.triggerCompleted(ExtendedNodeInstanceImpl.java:44) 
at org/jbpm/workflow/instance/node/StateBasedNodeInstance.triggerCompleted(StateBasedNodeInstance.java:286) 
at org/jbpm/workflow/instance/node/StateBasedNodeInstance.triggerCompleted(StateBasedNodeInstance.java:265) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.triggerCompleted(WorkItemNodeInstance.java:275) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.workItemCompleted(WorkItemNodeInstance.java:337) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.signalEvent(WorkItemNodeInstance.java:313) 
at org/jbpm/workflow/instance/impl/WorkflowProcessInstanceImpl.signalEvent(WorkflowProcessInstanceImpl.java:403) 
at org/drools/persistence/jpa/processinstance/JPAWorkItemManager.completeWorkItem(JPAWorkItemManager.java:134) 
...
at org/drools/persistence/jpa/processinstance/JPAWorkItemManager.internalExecuteWorkItem(JPAWorkItemManager.java:56) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.internalTrigger(WorkItemNodeInstance.java:124) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.trigger(NodeInstanceImpl.java:155) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.triggerNodeInstance(NodeInstanceImpl.java:337) 
at org/jbpm/workflow/instance/impl/NodeInstanceImpl.triggerCompleted(NodeInstanceImpl.java:296) 
at org/jbpm/workflow/instance/impl/ExtendedNodeInstanceImpl.triggerCompleted(ExtendedNodeInstanceImpl.java:44) 
at org/jbpm/workflow/instance/node/StateBasedNodeInstance.triggerCompleted(StateBasedNodeInstance.java:286) 
at org/jbpm/workflow/instance/node/StateBasedNodeInstance.triggerCompleted(StateBasedNodeInstance.java:265) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.triggerCompleted(WorkItemNodeInstance.java:275) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.workItemCompleted(WorkItemNodeInstance.java:337) 
at org/jbpm/workflow/instance/node/WorkItemNodeInstance.signalEvent(WorkItemNodeInstance.java:313) 
at org/jbpm/workflow/instance/impl/WorkflowProcessInstanceImpl.signalEvent(WorkflowProcessInstanceImpl.java:403) 
at org/drools/persistence/jpa/processinstance/JPAWorkItemManager.completeWorkItem(JPAWorkItemManager.java:134) 
...
at org/drools/persistence/jpa/processinstance/JPAWorkItemManager.internalExecuteWorkItem(JPAWorkItemManager.java:56)
...

• User is using Singleton mode of KieSession, which they configure it like this in their kjar.

...
        <kie:kmodule id="firstKieModule">
        <kie:kbase name="firstKieBase" packages="com.sample.jbpm.integration">

            <kie:ksession name="firstKieSession" type="stateful">
                <kie:processEventListener ref="first-audit-listener" ></kie:processEventListener>               
                <kie:processEventListener ref="firstProcessListener" ></kie:processEventListener>   
                <kie:configuration>
                    <kie:jpa-persistence>
                        <kie:transaction-manager ref="firstTransactionManager" />
                        <kie:entity-manager-factory ref="firstEntityManagerFactory" />
                    </kie:jpa-persistence>
                </kie:configuration>
            </kie:ksession>     
        </kie:kbase>
    </kie:kmodule>
...

Although users are using Spring to integrate jBPM inside the code, but they are not able to extend Spring transaction into jBPM transaction. Hence, in their project, the jBPM has its own transaction, and every command will commit the data when jBPM API is returned. The issue is more likely to happen in high concurrency situation, when users starting 10+ processes and completing tasks at the same time.

• What could have caused this kind of issue and how to fix it?