Diagnostic Steps

• Is SSL enabled for the agent that is managing the resource where the bundle is being deployed? This issue will only occur if SSL is enabled for server to agent communication.

• View the state of the JBoss ON server socket connection going to the affected agent. This can be done using netstat on the server and looking for the agent's IP address and port number in the output:

  netstat -nt | grep 16163
  

  This may reveal output similar to:

  tcp      112      0 192.168.1.50:42187       192.168.1.52:16163        CLOSE_WAIT  java
  

  In this case the server's address is 192.168.1.50 and it has a socket opened to the agent on 192.168.1.52 port 16163. The state of this socket is CLOSE_WAIT. The CLOSE_WAIT state indicates this is a stale connection. The next attempt to deploy a bundle to this agent will fail and this issue applies.

  Note that if the socket state is ESTABLISHED or there is no connection being returned for this agent, the connection is not stale and this issue does not apply.

• Use JBoss Byteman to capture details surrounding the failure.

  1. If necessary, download and install JBoss Byteman.

  2. Create the necessary Byteman rule:

     cat >/tmp/BZ1166383.btm <<EOF
     ########################################################################
     # To the extent possible under law, Red Hat, Inc. has dedicated all 
     # copyright to this software to the public domain worldwide, pursuant 
     # to the CC0 Public Domain Dedication. This software is distributed 
     # without any warranty.  
     # 
     # See <http://creativecommons.org/publicdomain/zero/1.0/>.
     #
     #
     # JBM-BZ1166383 Rule File
     # 
     # This JBoss Byteman rule set may be helpful in diagnosing JBoss Remoting 
     # errors.
     #
     # See Red Hat Bugzilla 1166383 <https://bugzilla.redhat.com/show_bug.cgi?id=1166383>.
     #
     RULE trace MicroSocketClientInvoker handleException
     CLASS ^org.jboss.remoting.transport.socket.MicroSocketClientInvoker
     METHOD handleException
     AT THROW ALL
     BIND socketInvoker:MicroSocketClientInvoker = \$0;
          generalizedExcept:boolean = socketInvoker.isGeneralizeSocketException();
          exception = \$^
     IF TRUE
     DO System.err.println("JBM-BZ1166383: generalizeSocketException = " + generalizedExcept + "; Exception: " + exception);
        exception.printStackTrace()
     ENDRULE
     EOF
     

  3. Update the following command with the appropriate values and then execute it:

     BYTEMAN_HOME=/usr/share/byteman RHQ_SERVER_ADDITIONAL_JAVA_OPTS="-javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:/tmp/BZ1166383.btm" "${RHQ_SERVER_HOME}"/bin/rhqctl restart --server
     

     You will need to change the value of BYTEMAN_HOME from /usr/share/byteman to the full path of where you have JBoss Byteman installed. Also, if you used a different location or file name to store the Byteman rule file, you will need to update the script value from /tmp/BZ1166383.btm to the complete path and name of your rule file.

     NOTE: If you do not set the environment variables BYTEMAN_HOME and RHQ_SERVER_ADDITIONAL_JAVA_OPTS as part of the command used to start or restart the JBoss ON server, you must remember to export them so they are available to the child process where JBoss ON server is created. For example:

     BYTEMAN_HOME=/usr/share/byteman 
     RHQ_SERVER_ADDITIONAL_JAVA_OPTS="-javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:/tmp/BZ1166383.btm" 
     export BYTEMAN_HOME
     export RHQ_SERVER_ADDITIONAL_JAVA_OPTS
     "${RHQ_SERVER_HOME}"/bin/rhqctl restart --server