JacORB deadlock around GIOPConnection in JBoss EAP

Issue

• One of our production JBoss nodes has experienced the following problem. It makes CORBA calls to one of our "legacy" CORBA-based servers. It was noticed at some point that a lot of threads in the JBoss JVM got stuck in the following call (this the top of the thread stack):

    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at org.jacorb.orb.giop.GIOPConnection.getWriteLock(Unknown Source)

From the jstack output and the JacORB source (from a decompiler). It appears the culprit is the deadlock formed between the following two threads:

"WorkManager(2)-595" daemon prio=3 tid=0x0000000107ed7000 nid=0x865 in Object.wait() [0xfffffffd08bb9000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:485)
    at org.jacorb.orb.giop.GIOPConnection.getWriteLock(Unknown Source)
    - locked <0xfffffffd808859c8> (a java.lang.Object)
    at org.jacorb.orb.giop.ClientGIOPConnection.closeAllowReopen(Unknown Source)
    - locked <0xfffffffd808859b8> (a java.lang.Object)
    at org.jacorb.orb.giop.ClientGIOPConnection.streamClosed(Unknown Source)
    at org.jacorb.orb.giop.GIOPConnection.sendMessage(Unknown Source)
    at org.jacorb.orb.giop.GIOPConnection.sendRequest(Unknown Source)

"WorkManager(2)-407" daemon prio=3 tid=0x0000000107e38800 nid=0x689 waiting for monitor entry [0xfffffffd14ab9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.jacorb.orb.giop.GIOPConnection.sendMessage(Unknown Source)
    - waiting to lock <0xfffffffd808859b8> (a java.lang.Object)
    at org.jacorb.orb.giop.GIOPConnection.sendRequest(Unknown Source)

Looking at the source code of (Client)GIOPConnection, the 1st thread, WorkManager(2)-595, is holding the connect_sync lock while waiting to acquire writer_sync lock (in getWriteLock()). The 2nd thread, WorkManager(2)-407, is doing the opposite: it has acquired the write_async (getWriteLock() is called at the start of the sendMessage method) and is trying to grab connect_sync.

• We are seeing the following deadlock in our stacktrace. Is it possible to get a patch for this released to EAP 6.0.1 ?

default-threads - 366" prio=10 tid=0x00007f1a18025800 nid=0x6566 in Object.wait() [0x00007f17bda7d000]
 java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.jacorb.orb.giop.GIOPConnection.getWriteLock(Unknown Source)
    - locked <0x0000000747eaec98> (a java.lang.Object)
    at org.jacorb.orb.giop.ClientGIOPConnection.closeAllowReopen(Unknown Source)
    - locked <0x0000000747d60608> (a java.lang.Object)
    at org.jacorb.orb.giop.ClientGIOPConnection.streamClosed(Unknown Source)
    at org.jacorb.orb.giop.GIOPConnection.sendMessage(Unknown Source)
    at org.jacorb.orb.giop.GIOPConnection.sendRequest(Unknown Source)

"default-threads - 223" prio=10 tid=0x00007f1a4c028000 nid=0x6494 waiting for monitor entry [0x00007f17ca84a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
      at org.jacorb.orb.giop.ClientGIOPConnection.closeAllowReopen(Unknown Source)
      - waiting to lock <0x0000000747d60608> (a java.lang.Object)
      at org.jacorb.orb.giop.ClientGIOPConnection.streamClosed(Unknown Source)
      at org.jacorb.orb.giop.GIOPConnection.sendMessage(Unknown Source)
      at org.jacorb.orb.giop.GIOPConnection.sendRequest(Unknown Source)

• Seeing issue that looks like JacORB defect #759:
• http://www.jacorb.org/cgi-bin/bugzilla/show_bug.cgi?id=759