JBoss EAP start up deadlocks on Elytron ExpressionResolver

Issue

• We've configured JBoss to use an elytron expression resolver. After this, we randomly see startup hang until it times out and thread dumps show it has reached a deadlock between a RuntimeExpressionResolver HashSet lock and an ElytronDoohickey lock:

Found one Java-level deadlock:
=============================
"ServerService Thread Pool -- 63":
  waiting to lock monitor 0x00007fbf08013bf8 (object 0x00000006b32b4ca0, a java.util.HashSet),
  which is held by "ServerService Thread Pool -- 38"
"ServerService Thread Pool -- 38":
  waiting for ownable synchronizer 0x00000006b2f61d58, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "ServerService Thread Pool -- 42"
"ServerService Thread Pool -- 42":
  waiting to lock monitor 0x00007fbf08013bf8 (object 0x00000006b32b4ca0, a java.util.HashSet),
  which is held by "ServerService Thread Pool -- 38"

Java stack information for the threads listed above:
===================================================
"ServerService Thread Pool -- 63":
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:115)
        - waiting to lock <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.wildfly.extension.undertow.UndertowSubsystemAdd.performBoottime(UndertowSubsystemAdd.java:106)
        at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:119)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)
"ServerService Thread Pool -- 38":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006b2f61d58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at org.wildfly.extension.elytron.ElytronDoohickey.apply(ElytronDoohickey.java:80)
        at org.wildfly.extension.elytron.ElytronDoohickey.apply(ElytronDoohickey.java:49)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveCredentialStore(ElytronExpressionResolver.java:285)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveExpressionInternal(ElytronExpressionResolver.java:121)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveExpression(ElytronExpressionResolver.java:76)
        at org.jboss.as.controller.extension.ExtensionRegistry$FutureExpressionResolverExtension.resolveExpression(ExtensionRegistry.java:1288)
        at org.jboss.as.server.RuntimeExpressionResolver.resolveExpression(RuntimeExpressionResolver.java:152)
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:118)
        - locked <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.jboss.as.connector.subsystems.datasources.ConnectionPropertyAdd.performRuntime(ConnectionPropertyAdd.java:55)
        at org.jboss.as.controller.AbstractAddStepHandler.performRuntime(AbstractAddStepHandler.java:343)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)
"ServerService Thread Pool -- 42":
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:115)
        - waiting to lock <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.wildfly.extension.elytron.SecretKeyCredentialStoreDefinition$SecretKeyDoohickey.resolveRuntime(SecretKeyCredentialStoreDefinition.java:230)
        at org.wildfly.extension.elytron.ElytronDoohickey.resolveRuntime(ElytronDoohickey.java:150)
        at org.wildfly.extension.elytron.ElytronDoohickey.resolveRuntime(ElytronDoohickey.java:135)
        at org.wildfly.extension.elytron.DoohickeyAddHandler.performRuntime(DoohickeyAddHandler.java:74)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)

Found 1 deadlock.