Simultaneous shutdown of EAP instances fails due to JMS deadlock

Solution Verified - Updated -

Issue

  • While attempting to shutdown two nodes simultaneously, one node hangs while the other shuts down gracefully

  • With two nodes in a cluster running JBoss EAP 5.0, the command "kill <pid>" was issued to bring down the JBoss instance.  The command kill <pid1> <pid2> , which was supposed to trigger a shutdown of both JBoss  instances at the same time, causes the deadlock in one of the  instances. The instance did not stop.  Issued "kill -3 <pid>" to obtain stack trace and the thread dump detected a deadlock:

    Found one Java-level deadlock:
=============================
"JBoss Shutdown Hook":
  waiting to lock monitor 0x0906f79c (object 0xee4f26f8, a org.jgroups.Membership),
  which is held by "Incoming-6,10.160.15.235:50065"
"Incoming-6,10.160.15.235:50065":
  waiting to lock monitor 0xda4531e4 (object 0xee4e4370, a org.jboss.messaging.core.impl.postoffice.MessagingPostOffice),
  which is held by "JBoss Shutdown Hook"
  • We are using EPP 4.3 and are running two  nodes(Server1 and Server2) on the same machine and have written a start/stop script that would look for 'Started in' and 'Shutdown  complete' line in the server.log and then start/stop jboss server  respectively.  It normally works, but sometimes  Server1 shutdown is  successful but the Server2 does not stop because the server.log did  not write 'Shutdown complete' and it was waiting forever to stop the  server and it had to shutdown manually.  The javacore from the hung node contained the following
    1LKDEADLOCK    Deadlock detected !!! 
NULL           --------------------- 
NULL            
2LKDEADLOCKTHR  Thread "IncomingPacketHandler (channel=MessagingPostOffice-CTRL)" (0x000000011C16D600) 
3LKDEADLOCKWTR    is waiting for: 
4LKDEADLOCKMON      sys_mon_t:0x000000011BED6EF0 infl_mon_t: 0x000000011BED6F40: 
4LKDEADLOCKOBJ      org/jboss/messaging/core/impl/postoffice/MessagingPostOffice@070000000773AFD8/070000000773AFF0:  
3LKDEADLOCKOWN    which is owned by: 
2LKDEADLOCKTHR  Thread "JBoss Shutdown Hook" (0x000000011EDE1600) 
3LKDEADLOCKWTR    which is waiting for: 
4LKDEADLOCKMON      sys_mon_t:0x000000011BED5C60 infl_mon_t: 0x000000011BED5CB0: 
4LKDEADLOCKOBJ      org/jgroups/Membership@07000000084C15D8/07000000084C15F0:  
3LKDEADLOCKOWN    which is owned by: 
2LKDEADLOCKTHR  Thread "IncomingPacketHandler (channel=MessagingPostOffice-CTRL)" (0x000000011C16D600)

Environment

  • JBoss Enterprise Portal Platform(EPP) 4.3.0 CP05

  • JBoss Enterprise Application Platform(EAP)

    • 5.0.0

    • 4.3.0

  • JBoss Messaging(JMS)

    • 1.4.0.SP3.CP10 & earlier
    • 1.4.6.GA & earlier
  • AIX 5.3
  • Linux

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.