JBoss ON agent fails to communicate with server and reports NoSuchMethodError: ConnectAgentRequest.<init>
Environment
- Red Hat JBoss Operations Network (ON) 3.2.1, 3.2.2, 3.2.3
- JBoss ON agent was recently updated using the
apply-updatesutility - The directory or partition from where the JBoss ON update utility was executed had low or no available disk space or was read-only
Issue
- Error from agent log communicating to server
-
Get a
NoSuchMethodErrorfrom the agent log.ERROR [RHQ Server Polling Thread] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize callback has failed. It will be tried again. Cause: java.lang.NoSuchMethodError:org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V. Cause: java.lang.NoSuchMethodError: org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V ERROR [ClientCommandSenderTask Timer Thread #0] (enterprise.communications.command.client.JBossRemotingRemoteCommunicator)- {JBossRemotingRemoteCommunicator.init-callback-failed}The initialize callback has failed. It will be tried again. Cause: java.lang.NoSuchMethodError:org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V. Cause: java.lang.NoSuchMethodError: org.rhq.core.clientapi.server.core.ConnectAgentRequest.<init>(Ljava/lang/String;Lorg/rhq/core/clientapi/server/core/AgentVersion;Z)V -
Agent reports server offline:
{ServerPollingThread.server-offline}The server has gone offline; client has been told to stop sending commands
Resolution
The agent will need to be re-installed. The existing configuration, data, and logs can be retained if desired:
- Shutdown the existing agent.
-
Rename the existing agent's installation directory or move it to a backup location in case any of these steps fail. For example:
mv rhq-agent rhq-agent.bad -
Install a new JBoss ON agent using the agent installer provided with your updated JBoss ON server installation. You must do this as the same user.
java -jar rhq-enterprise-agent-4.9.0.JON320GA.jar -i -
If desired, copy the configuration, data, plugins, and logs from the previous agent to the new agent:
# configuration cp -a rhq-agent.bad/conf rhq-agent/ cp -a rhq-agent.bad/bin/rhq-agent-env.sh rhq-agent/bin/rhq-agent-env.sh cp -a rhq-agent.bad/bin/rhq-agent-env.sh.new rhq-agent/bin/rhq-agent-env.sh.new cp -a rhq-agent.bad/bin/rhq-agent-env.bat rhq-agent/bin/rhq-agent-env.bat cp -a rhq-agent.bad/bin/wrapper/rhq-agent-wrapper.conf rhq-agent/bin/wrapper/rhq-agent-wrapper.conf cp -a rhq-agent.bad/bin/wrapper/rhq-agent-wrapper.inc rhq-agent/bin/wrapper/rhq-agent-wrapper.inc # data cp -a rhq-agent.bad/data rhq-agent/ # plug-ins cp -a rhq-agent.bad/plugins rhq-agent/ # logs cp -a rhq-agent.bad/logs rhq-agent/ -
Start the agent.
-
Once you confirm the agent has properly started, you can delete the bad agent directory:
rm rhq-agent.bad
Root Cause
A previous version of the JBoss ON client API library exists on the affected agent while the server has an updated version. This can happen when the apply-updates utility fails to properly backup the old library version when installing the new one. The failed backup results in the old file remaining in place in the agent installation. The backup failure was most likely due to insufficient disk space or write access on the disk or directory in where the apply-updates utility was located.
The update.log may include a message indicating the failure:
mv: closing `./.old-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar': No space left on device
This issue has been reported in Red Hat Bugzilla 1170405 and will be addressed in a future release.
Diagnostic Steps
- Was the
apply-updatesutility used to update the affected agent from 3.2 to 3.2.1 or later? If not, this issue may still apply but would indicate that file system corruption or disk failure has occurred and a complete restore of all critical systems may be necessary. -
Does the file
rhq-core-client-api-4.9.0.JON320GA.jarexist in the affected agent'slibdirectory?[ -f rhq-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar ] && echo 'WARNING: File exists!'If the update was properly applied, this file will not exist.
-
Review the
update.logfile produced by theapply-updatescommand for a message indicated that the files was not properly backed up or other failures dealing with disk space or file permissions occurred:mv /opt/jboss/rhq-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar ./.old-agent/lib mv: closing `./.old-agent/lib/rhq-core-client-api-4.9.0.JON320GA.jar': No space left on deviceIf there was insufficient disk space,
update.logwill be incomplete or truncated. This may mean that the error message regarding the failure may not be visible.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
