EAP 6 shutdown stalls when using an Executor pool for JBossWeb

Issue

• EAP 6 fails to shutdown fully when using an Executor pool for a JBossWeb connector.
• Thread dumps show shutdown threads waiting like so:

"MSC Shutdown Thread" prio=6 tid=0x5552d000 nid=0x208c waiting on condition [0x55a3f000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x13bf0000> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
    at org.jboss.msc.service.ServiceContainerImpl$ShutdownHookHolder$1$1.run(ServiceContainerImpl.java:129)
    at java.lang.Thread.run(Thread.java:662)

"SIGINT handler" daemon prio=10 tid=0x5552c400 nid=0x6f8 in Object.wait() [0x5599f000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x181732f8> (a java.lang.Thread)
    at java.lang.Thread.join(Thread.java:1186)
    - locked <0x181732f8> (a java.lang.Thread)
    at java.lang.Thread.join(Thread.java:1239)
    at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
    at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
    at java.lang.Shutdown.runHooks(Shutdown.java:79)
    at java.lang.Shutdown.sequence(Shutdown.java:123)
    at java.lang.Shutdown.exit(Shutdown.java:168)
    - locked <0x42fe8158> (a java.lang.Class for java.lang.Shutdown)
    at java.lang.Terminator$1.handle(Terminator.java:35)
    at sun.misc.Signal$1.run(Signal.java:195)
    at java.lang.Thread.run(Thread.java:662)

• There are also many threads in a socket read during their keepalive state:

"http-executor-threads - 1" prio=6 tid=0x55515000 nid=0x20c0 runnable [0x5590f000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)
    at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)
    at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519)
    at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
    at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)
    at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
    at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)
    at java.lang.Thread.run(Thread.java:662)
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)

• When we shutdown the proxy/loadbalancer end for those AJP connections, those threads clear out and then shutdown completes.