JBoss shutdown hangs to be caused by deadlock between "JBoss Shutdown Hook" and JBoss Messaging thread

Issue

• In EAP 4.3, JBoss shutdown hangs to be caused by deadlock between "JBoss Shutdown Hook" and JBoss Messaging thread "WorkerThread":

  "JBoss Shutdown Hook" daemon prio=1 tid=0x0a023258 nid=0x2a38 in Object.wait() [0x826f8000..0x826fa130]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x91593088> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
      at java.lang.Object.wait(Object.java:474)
      at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
      - locked <0x91593088> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.setClosed(JMSServerInvocationHandler.java:68)
      at org.jboss.jms.server.ServerPeer.stopService(ServerPeer.java:349)
      - locked <0x914cd0a0> (a org.jboss.jms.server.ServerPeer)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
      at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)
      at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
      at $Proxy0.stop(Unknown Source)
      at org.jboss.system.ServiceController.stop(ServiceController.java:508)
      at org.jboss.system.ServiceController.stop(ServiceController.java:499)
      ...(snip)...
  
  ...(snip)...
  
  
  "WorkerThread#12[10.133.46.128:50365]" prio=1 tid=0x0badb610 nid=0x1077 waiting for monitor entry [0x85657000..0x85657eb0]
      at org.jboss.jms.server.ServerPeer.getDefaultDLQInstance(ServerPeer.java:1187)
      - waiting to lock <0x914cd0a0> (a org.jboss.jms.server.ServerPeer)
      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.<init>(ServerSessionEndpoint.java:220)
      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:249)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$createSessionDelegate$aop(ConnectionAdvised.java:75)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeNext(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.createSessionDelegate(ConnectionAdvised.java)
      at org.jboss.jms.wireformat.ConnectionCreateSessionDelegateRequest.serverInvoke(ConnectionCreateSessionDelegateRequest.java:87)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:866)
      at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
      at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)
      at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
  

• In EAP 5, JBoss shutdown hangs to be caused by deadlock between "JBoss Shutdown Hook" and JBoss Messaging thread "Thread-N":

  "JBoss Shutdown Hook" daemon prio=11 tid=0x00002aaac0143800 nid=0x7ef4 in Object.wait() [0x0000000044aa1000]
     java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x0000000795991950> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
      at java.lang.Object.wait(Object.java:485)
      at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
      - locked <0x0000000795991950> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.setClosed(JMSServerInvocationHandler.java:68)
      at org.jboss.jms.server.ServerPeer.stopService(ServerPeer.java:388)
      - locked <0x0000000794c09488> (a org.jboss.jms.server.ServerPeer)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:405)
      at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:281)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
      at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
      at $Proxy38.stop(Unknown Source)
      at org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:56)
      at org.jboss.system.microcontainer.StartStopLifecycleAction.uninstallAction(StartStopLifecycleAction.java:37)
      at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
      at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
      at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
      at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:354)
      at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1685)
      at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1295)
      at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1198)
      at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1254)
      at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1198)
      at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1254)
      at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:831)
      at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
      at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
      at org.jboss.system.ServiceController.stop(ServiceController.java:510)
      ...(snip)...
  
  ...(snip)...
  
  "Thread-20" prio=10 tid=0x00002aaabe4ca800 nid=0x49b9 waiting for monitor entry [0x0000000040f66000]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at org.jboss.jms.server.ServerPeer.getDefaultExpiryQueueInstance(ServerPeer.java:1572)
      - waiting to lock <0x0000000794c09488> (a org.jboss.jms.server.ServerPeer)
      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.<init>(ServerSessionEndpoint.java:230)
      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.createSessionDelegate(ServerConnectionEndpoint.java:249)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$createSessionDelegate$aop(ConnectionAdvised.java:75)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$createSessionDelegate_6052335267724906805.invokeTarget(ConnectionAdvised$createSessionDelegate_6052335267724906805.java)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
      at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.createSessionDelegate(ConnectionAdvised.java)
      at org.jboss.jms.wireformat.ConnectionCreateSessionDelegateRequest.serverInvoke(ConnectionCreateSessionDelegateRequest.java:87)
      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)
      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:930)
      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)
      at org.jboss.remoting.Client.invoke(Client.java:2034)
      at org.jboss.remoting.Client.invoke(Client.java:877)
      at org.jboss.remoting.Client.invoke(Client.java:865)
      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$createSessionDelegate$aop(ClientConnectionDelegate.java:190)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate$createSessionDelegate_6052335267724906805.invokeTarget(ClientConnectionDelegate$createSessionDelegate_6052335267724906805.java)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
      at org.jboss.jms.client.container.StateCreationAspect.handleCreateSessionDelegate(StateCreationAspect.java:115)
      at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateSessionDelegate_237427554.invoke(StateCreationAspect_z_handleCreateSessionDelegate_237427554.java)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      at org.jboss.jms.client.container.ConnectionAspect.handleCreateSessionDelegate(ConnectionAspect.java:162)
      at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      at org.jboss.jms.client.delegate.ClientConnectionDelegate.createSessionDelegate(ClientConnectionDelegate.java)
      at org.jboss.jms.client.JBossConnection.createSessionInternal(JBossConnection.java:269)
      at org.jboss.jms.client.JBossConnection.createSession(JBossConnection.java:91)
      at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.connect(MessagingXAResourceWrapper.java:300)
      - locked <0x0000000795408ed0> (a java.lang.Object)
      at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.getDelegate(MessagingXAResourceWrapper.java:260)
      at org.jboss.jms.server.recovery.MessagingXAResourceWrapper.recover(MessagingXAResourceWrapper.java:108)
      at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:1167)
      at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecovery(XARecoveryModule.java:665)
      at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:200)
      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)