JBoss ON agent fails to communicate with server and reports NoSuchMethodError: ConnectAgentRequest.<init>

Solution Unverified - Updated -

Environment

  • Red Hat JBoss Operations Network (ON) 3.2.1, 3.2.2, 3.2.3
  • JBoss ON agent was recently updated using the apply-updates utility
  • The directory or partition from where the JBoss ON update utility was executed had low or no available disk space or was read-only

Issue

  • Error from agent log communicating to server
  • Get a NoSuchMethodError from the agent log.

    ERROR [RHQ Server Polling Thread] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize callback has failed. It will be tried again. Cause: java.lang.NoSuchMethodError:org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V. Cause: java.lang.NoSuchMethodError: org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V
    ERROR [ClientCommandSenderTask Timer Thread #0] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize callback has failed. It will be tried again. Cause: java.lang.NoSuchMethodError:org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V. Cause: java.lang.NoSuchMethodError: org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V
    
  • Agent reports server offline:

    {ServerPollingThread.server-offline}The server has gone offline; client has been told to stop sending commands
    

Resolution

The agent will need to be re-installed. The existing configuration, data, and logs can be retained if desired:

  1. Shutdown the existing agent.
  2. Rename the existing agent's installation directory or move it to a backup location in case any of these steps fail. For example:

    mv rhq-agent rhq-agent.bad
    
  3. Install a new JBoss ON agent using the agent installer provided with your updated JBoss ON server installation. You must do this as the same user.

    java -jar rhq-enterprise-agent-4.9.0.JON320GA.jar -i
    
  4. If desired, copy the configuration, data, plugins, and logs from the previous agent to the new agent:

    # configuration
    cp -a rhq-agent.bad/conf rhq-agent/
    cp -a rhq-agent.bad/bin/rhq-agent-env.sh rhq-agent/bin/rhq-agent-env.sh
    cp -a rhq-agent.bad/bin/rhq-agent-env.sh.new rhq-agent/bin/rhq-agent-env.sh.new
    cp -a rhq-agent.bad/bin/rhq-agent-env.bat rhq-agent/bin/rhq-agent-env.bat
    cp -a rhq-agent.bad/bin/wrapper/rhq-agent-wrapper.conf rhq-agent/bin/wrapper/rhq-agent-wrapper.conf
    cp -a rhq-agent.bad/bin/wrapper/rhq-agent-wrapper.inc rhq-agent/bin/wrapper/rhq-agent-wrapper.inc
    
    # data
    cp -a rhq-agent.bad/data rhq-agent/
    
    # plug-ins
    cp -a rhq-agent.bad/plugins rhq-agent/
    
    # logs
    cp -a rhq-agent.bad/logs rhq-agent/
    
  5. Start the agent.

  6. Once you confirm the agent has properly started, you can delete the bad agent directory:

    rm rhq-agent.bad
    

Root Cause

A previous version of the JBoss ON client API library exists on the affected agent while the server has an updated version. This can happen when the apply-updates utility fails to properly backup the old library version when installing the new one. The failed backup results in the old file remaining in place in the agent installation. The backup failure was most likely due to insufficient disk space or write access on the disk or directory in where the apply-updates utility was located.

The update.log may include a message indicating the failure:

mv: closing `./.old-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar': No space left on device

This issue has been reported in Red Hat Bugzilla 1170405 and will be addressed in a future release.

Diagnostic Steps

  • Was the apply-updates utility used to update the affected agent from 3.2 to 3.2.1 or later? If not, this issue may still apply but would indicate that file system corruption or disk failure has occurred and a complete restore of all critical systems may be necessary.
  • Does the file rhq-core-client-api-4.9.0.JON320GA.jar exist in the affected agent's lib directory?

    [ -f rhq-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar ] && echo 'WARNING: File exists!'
    

    If the update was properly applied, this file will not exist.

  • Review the update.log file produced by the apply-updates command for a message indicated that the files was not properly backed up or other failures dealing with disk space or file permissions occurred:

        mv /opt/jboss/rhq-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar ./.old-agent/lib
    mv: closing `./.old-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar': No space left on device
    

    If there was insufficient disk space, update.log will be incomplete or truncated. This may mean that the error message regarding the failure may not be visible.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.