EAP 7 start up with many applications using mod_cluster delayed in org.wildfly.extension.undertow.UndertowService.fireEvent

Issue

• We use mod_cluster and deploy a large amount of applications on EAP 7. Start up is very slow. Thread dumps show deployments waiting here:

"ServerService Thread Pool -- 368" #4342 prio=5 os_prio=0 tid=0x000055fc03b25800 nid=0xdc1c waiting for monitor entry [0x00007f17c63cf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:285)
    - waiting to lock <0x00000002cef91808> (a java.util.Collections$SynchronizedList)
    at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:268)

• The lock owner is blocked here:

"ServerService Thread Pool -- 168" #1860 prio=5 os_prio=0 tid=0x000055fc03708800 nid=0xd0d7 waiting for monitor entry [0x00007f17f0263000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:539)
    - waiting to lock <0x00000002c4b0ae38> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:373)
    at org.jboss.modcluster.ModClusterService.enable(ModClusterService.java:326)
    at org.jboss.modcluster.ModClusterService.start(ModClusterService.java:312)
    at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.onStart(UndertowEventHandlerAdapterService.java:129)
    - locked <0x00000002c2ce8108> (a org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService)
    at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.onDeploymentStart(UndertowEventHandlerAdapterService.java:145)
    at org.wildfly.extension.undertow.Host.lambda$registerDeployment$0(Host.java:268)
    at org.wildfly.extension.undertow.Host$$Lambda$1111/1883027328.invoke(Unknown Source)
    at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:286)
    - locked <0x00000002cef91808> (a java.util.Collections$SynchronizedList)
    at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:268)
    at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99)

• And that is blocked by this thread awaiting a mod_cluster STATUS update:

"UndertowEventHandlerAdapterService - 1" #106 prio=5 os_prio=0 tid=0x000055fbf83e1800 nid=0xc666 runnable [0x00007f1813e46000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
    at sun.security.ssl.InputRecord.read(InputRecord.java:503)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990)
    - locked <0x00000006ed984718> (a java.lang.Object)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
    - locked <0x00000006ed984948> (a sun.security.ssl.AppInputStream)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    - locked <0x00000006eda0bd30> (a java.io.InputStreamReader)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    - locked <0x00000006eda0bd30> (a java.io.InputStreamReader)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:468)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:552)
    - locked <0x00000002c4b0ae38> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:373)
    at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:398)
    at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.run(UndertowEventHandlerAdapterService.java:178)