Red Hat Training

A Red Hat training course is available for Red Hat JBoss Operations Network

5. Agent

Q: I have a physical machine hosting multiple virtual machines with shared disk resources. How can I run an agent on each virtual instance?
Q: How do I get debug messages from the JBoss ON agent?
Q: How do I restrict which agents are allowed to connect to the server?
Q: Do I have to run the agent as root?
Q: How do I clean start the JBoss ON agent, as if newly installed?
Q: How can I do a "clean config" for an agent running as a background Windows service?
Q: How can I update the plug-ins on all my agents?
Q: How can I change the agent name after it has already been registered?
Q: I want to run agents on all my machines, but only one starts OK. The rest fail due to binding to a wrong address.
Q: When starting the agent via a Windows service, the agent fails to start, and I see the error "java.lang.IllegalStateException: The name of this agent is not defined - you cannot start the agent until you give it a valid name" in the agent wrapper log file. What does this mean?
Q: My agent setup is correct but my agent is getting the following message: "Cause: org.jboss.remoting.CannotConnectException: Can not connect http client invoker."
Q: My agent machine does not have a writable directory called /var/run. How can I get my rhq-agent-wrapper.sh script to successfully write out its pid file?
Q: How often does the agent scan for resources?
Q: How can I view the agent's persisted configuration?
Q: How can I find out what environment variables and Java system properties are set in my agent JVM process?
Q: How can I get a dump of inventory information from an agent running on another machine?
Q: I need to change the IP address of my agent machine. How do I keep my server and agent up to date with that change?
Q: How can I stop my agent from thinking the server keeps going up and down when the server has remained running the whole time?
Q:
I have a physical machine hosting multiple virtual machines with shared disk resources. How can I run an agent on each virtual instance?
A:
You can run multiple agents on the same box, but the agents must run out of two different agent installation directories.
Q:
How do I get debug messages from the JBoss ON agent?
A:
The easiest and quickest way to get your agent to start logging debug messages is, before starting your JBoss ON agent, to set the environment variable RHQ_AGENT_DEBUG to any value. When you start the agent, both the launcher scripts and the agent itself will output debug messages. When you use this environment variable, the agent will use an internal log4j configuration file.
For more fine-grained control over what log4j categories have DEBUG priority, edit the conf/log4j.xml file and restart the agent to load the changes. Do not set RHQ_AGENT_DEBUG if you want the agent to use the log4j.xml file; setting that environment variable causes the agent to override this log4j.xml with an internally configured log4j configuration.
The log messages can be found in the log files located in the agentRoot/rhq-agent/logs directory.
If you are launching the JBoss ON agent on Microsoft Windows using the service wrapper, you must set RHQ_AGENT_DEBUG and then install the service: 
rhq-agent-wrapper.bat install
Q:
How do I restrict which agents are allowed to connect to the server?
A:
Change the server's JBoss/Remoting servlet configuration to restrict the IPs agent requests can come from. If the agents are on a specific subnet, then connections can be restricted to only that subnet.
  1. Create a file for the restriction rule, with this name and location: 
    vim serverInstallDir/jbossas/server/default/deploy/rhq.ear/jboss-remoting-servlet-invoker-2x.r3040.jon.war/WEB-INF/context.xml
  2. Add this content to the file: 
    <?xml version="1.0" encoding="UTF-8"?> 
<Context> 
<Valve className="org.apache.catalina.valves.RemoteAddrValve" 
allow="192.168.*,142.104.128.*,10.224.27.182"/> 
</Context>
    The allow= attribute lists the IPs that are allowed to connect to the server. All other IPs are blocked.
Q:
Do I have to run the agent as root?
A:
You do not have to run the agent as root.
However, that said, some resource types have very strict limits on what users can access their configuration files and processes, and running the agent as root may be the only way to manage or monitor those resources.
For example, a PostgreSQL plug-in lets the agent probe the PostgreSQL configuration file, postgres.conf. Running the agent as a non-root user without PostgreSQL privileges means that the agent cannot read and manage the file. (And there will be log messages in the agent log saying so.) There are other resources that have similarly privileged files, like iptables and even some JBoss servers.
Running the agent as root gives the agent privileges to manage all those things. Without that privilege level, the agent has restricted views of managed resources.
Q:
How do I clean start the JBoss ON agent, as if newly installed?
A:
There are three points of configuration for the agent: the agent's (local) persisted configuration, the agent inventory (and associated resource data), and the platform entry in the server inventory.
To clean the agent configuration and restart the agent as if it were new, then there are two steps to take:
  1. Remove the platform entry from the JBoss ON server inventory. Since the platform entry is representative of the agent entry, this effectively removes the agent from the JBoss ON topology.
  2. Stop the agent and then restart it using the --fullcleanconfig (-L) command-line option. 
    agentRoot/rhq-agent/bin/rhq-agent.sh --fullcleanconfig
    The --fullcleanconfig option removes all of the local inventory for the agent, reloads its configuration fresh from the agent-configuration.xml file, and re-registers the agent with the server.
    Optionally, pass the --config argument to have it start up with a user-specified configuration file. Otherwise, the default conf/agent-configuration.xml file is used. If no directory is given, then the command looks for the configuration file in the agent's conf/ directory. 
    agentRoot/rhq-agent/bin/rhq-agent.sh --fullcleanconfig -c my-agent-configuration.xml
Q:
How can I do a "clean config" for an agent running as a background Windows service?
A:
The JBoss ON agent Windows service (like all Windows services) runs as a specific user. Go to that user's Windows registry and delete the agent's configuration node. The RHQ Agent uses the standard Java Preferences API, so the agent's configuration is stored as a node under the normal Java Preferences location in the Windows registry, such as HKEY_CURRENT_USER\Software\JavaSoft\Prefs\rhq-agent\default. ("default" is the name of the preferences node.)
Deleting the node only removes the previous configuration; the agent has to be reconfigured before it can be started again. The simplest path is to start the agent in the foreground and go through the interactive agent configuration, then stop that session and start the agent as a service.
Alternatively, the agent can be forced to read its configuration anew from the agent-configuration.xml file. To force the agent to re-read its configuration from file, you won't be able to start it in the foreground, which makes re-configuring it a little bit more difficult.
If the agent has been added as a JBoss ON resource, you can invoke the "Execute Command Prompt" operation and run the config --import agent-configuration.xml command.
Alternatively, you can edit the rhq-agent-wrapper.conf file and add a line for a third parameter: 
wrapper.app.parameter.3=--cleanconfig
This forces the agent to re-read its configuration from the agent-configuration.xml every time it is started as a service. In this case, the agent-configuration.xml must be preconfigured with all of the required (and optional) settings for the agent, so that it restarts with the correct configuration.
Q:
How can I update the plug-ins on all my agents?
A:
When you add a new plug-in to your system or upgrade an existing plug-in, you can instruct your agents to update their existing plug-ins with the new plug-in versions.
You can do this individually by executing the prompt command plugins update at any agent prompt or through the UI with the Update All Plugins task in the agent's OPERATION tab.
To update all of agents with the latest plug-ins, launch a group operation on the JBoss ON agents autogroup. First, create an autogroup by navigating to the Browse Resources page. Then click Group Definitions > New Definition button. The autogroup definition should have the expression: 
resource.resourceType.pluginName = RHQAgent
resource.resourceType.typeName = RHQ Agent
This creates a compatible group that dynamically adds all JBoss ON agents as members to that group.

Note

If you already have a compatible group with your agents as members, you can skip this group creation step.
Next, navigate to the agent group. In the OPERATIONS tab, invoke the Update All Plugins operation. This tells all of your agents in that group to update their plug-ins. Once that group operation is completed, all of your agents will have up-to-date versions of all plug-ins.
Q:
How can I change the agent name after it has already been registered?
A:
When you start the agent for the first time, you are asked in the setup screen for an agent name. This name must be unique across all agents in your environment. Once it is registered you cannot change this name. If you attempt to re-register this agent, you must re-register it with the same name that it was registered with before.
The agent name is not the same as the JBoss ON agent resource name in the UI. If you import an JBoss ON agent resource into inventory, that resource's name is agent_name JON agent. This JBoss ON agent resource name can be changed by editing its value within the INVENTORY tab. Changing this name does not change the name that the agent is registered under. Your agent is still registered under its original agent name.
Q:
I want to run agents on all my machines, but only one starts OK. The rest fail due to binding to a wrong address.
A:
There are a couple of things that can cause this error. 
FATAL [main] (org.jboss.on.agent.AgentMain)-
{AgentMain.startup-error}The agent encountered an error during startup
and must abort java.net.BindException: Cannot assign requested address
First, did you change the agent-configuration.xml manually (to change IP addresses, for example) after setting up the agent? The agent's configuration XML file is not referenced after the agent is setup because its configuration is persisted using Java Preferences. Persisting the configuration allows the agent to be updated or re-installed without losing its configuration. To change the agent's configuration file and have those changes picked up, restart the agent and pass the --config command line option (or -c which is shorthand for --config). This tells the agent to re-read the configuration file and override any old configuration it persisted before.
If your home directory is stored on NFS, then you are probably picking up the same Java preferences across all machines ($HOME/.java on Red Hat Enterprise Linux). This is usually not an issue on Windows since the Java preferences are stored in the registry. If you are running the agents as the same user and your user's home directory is shared, then have the agents use different Java preferences names. Edit your agents' agent-configuration.xml files and change their Java preferences node names from default to something that makes them unique across all agents. For example: 
<node name="NewName">
Since you overrode the default location, every time you start your agent you need to tell the agent where it can find its preferences. You tell the agent the new preference name using the --pref option. Since you changed the configuration file, restart the agent with the -c to specify the configuration file. 
agentRoot/rhq-agent/bin/rhq-agent.sh --pref NewName -c agent-configuration.xml
Subsequently, always restart the agent with the --pref option to pass in the node name.
Alternatively, define the system property java.util.prefs.userRoot to point to another, unique, location for preference. When the agent starts, Java will use the value of that system property as the location where it will store its Java Preferences. You can set this system property on the agent by setting the RHQ_AGENT_ADDITIONAL_JAVA_OPTS environment variable. When you set that environment variable, rhq-agent.sh will add its value to the default set of Java options when passing in options to the agent's Java VM: 
set RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Djava.util.prefs.userRoot=/etc/rhq-agent-prefs"
agentRoot/rhq-agent/bin/rhq-agent.sh
Q:
When starting the agent via a Windows service, the agent fails to start, and I see the error "java.lang.IllegalStateException: The name of this agent is not defined - you cannot start the agent until you give it a valid name" in the agent wrapper log file. What does this mean?
A:
The agent cannot ask for its initial setup configuration when installing as a Windows service because there is no console for the user to see and answer the prompts. This means that you need to either pass the information through a preconfigured file or run the agent in standard, non-service, mode once as the user that should run the service to configure it before installing it as a service.
Q:
My agent setup is correct but my agent is getting the following message: "Cause: org.jboss.remoting.CannotConnectException: Can not connect http client invoker."
A:
This error is typically seen when the server's endpoint address is not set to something that can be resolved by the agent. The public endpoint address set for each server must be resolvable by every JBoss ON agent because of the high availability cloud configuration of JBoss ON servers.
Check your server endpoint information in the high availability pages in the GUI and change the settings if necessary. After the update, restart the agent.
Q:
My agent machine does not have a writable directory called /var/run. How can I get my rhq-agent-wrapper.sh script to successfully write out its pid file?
A:
Set the environment variable RHQ_AGENT_PIDFILE_DIR to the full path of the directory where you want the pid file to be stored. When you run the script, that variable's value will override the default location. If you have an older script (2.1 or older), directly edit rhq-agent-wrapper.sh and change /var/run to the desired directory.
Q:
How often does the agent scan for resources?
A:
When an agent is installed, it scans the platform, and all applications on it, for any servers, services, or other items which can be included into the inventory. The process of finding potential resources is discovery.
There are different scans for each type of resource: platform, server, and service. High level scans for servers and platforms are initiated by the agent every 15 minutes. A service scan detects lower-level services that are running in servers that have already been imported into the inventory. These scans run by default every 24 hours. Both of these intervals are configurable.

Note

A server must be imported into the inventory before any of its child processes, servers, or services can be detected by the discovery scan.
Q:
How can I view the agent's persisted configuration?
A:
The agent's configuration is initially read from agent-configuration.xml and overlaid with the value given at the agent setup. After the agent is initially configured, it persists that configuration and never refers back to agent-configuration.xml, unless you clear the configuration.
There are several ways to view the agent's persisted configuration:
  1. If the agent is in the JBoss ON inventory, simply go to your agent's Configuration tab to view its live configuration. This is the same configuration that is persisted.
  2. If the agent is currently running in non-daemon mode (i.e. you have the agent prompt on your console), you can use the getconfig or config prompt commands to view the live configuration. Type help getconfig or help config for more information.
  3. If the agent is in the JBoss ON inventory, run the Execute Prompt Command operation and invoke the getconfig prompt command.
  4. Because the agent configuration is stored in the standard Java Preferences API backing store, you can use any tool that can examine Java preferences, such as Google's Java Preferences Tool. This is a GUI tool that can give you a file system-like view into your Java preferences. The agent preferences are stored in the User preferences node under the node name rhq-agent. Depending on the -p option that is passed to the agent for its node name when it is started, the actual configuration settings are found under a sub-node under rhq-agent. The default preferences node is called default, so typically your agent's persisted configuration is found in the user preferences under rhq-agent/default.

Warning

Do not attempt to change the values of the preferences using third-party tools without knowing what you are doing. This could disable the agent if you change the wrong preference to the wrong value. Use this mechanism only to view your agent's configuration.
Q:
How can I find out what environment variables and Java system properties are set in my agent JVM process?
A:
The version agent prompt command shows a list of the agent process' environment variables and system properties. version --sysprops provides a list of all the system properties, and version --env provides a list of all the environment variables. (At the agent prompt, run help version for the syntax of that command.)
Q:
How can I get a dump of inventory information from an agent running on another machine?
A:
If the agent inventory becomes corrupted, dumping the agent's inventory can help debug the problem.
To get this information, get the agent's data/inventory.dat file. Copy that file to the local machine. Then, run an agent on the local machine, with the same plug-ins as the other agent. The agent doesn't necessarily have to be connected to a server, but the plug-in container must be started, so the agent has to have been registered. Then, export the information from the imported DAT file.: 
inventory --xml --export=/bad-inventory.xml /the/bad/inventory.dat
If you do not specify the --export option, the XML will simply be dumped to the stdout console window.
Now you have an XML file that describes what the customer's agent thinks is its inventory.
Q:
I need to change the IP address of my agent machine. How do I keep my server and agent up to date with that change?
A:
The agent has a configuration preference named rhq.communications.connector.bind-address which sets the value of the IP address the agent binds to when it starts its server socket (the thing it listens to for incoming messages from the server).
If you change the agent's IP address (and invalidate the old agent IP address), you have to do a couple things:
  1. Change the agent's configuration so that preference value is the same as the new IP address. Issue a setconfig prompt command on the agent prompt: 
    setconfig rhq.communications.connector.bind-address=IP_address
    Do not change agent-configuration.xml; the changes will not take effect.
    If the agent is running in the background as a daemon process, shut it down with the script (rhq-agent-wrapper.sh|bat) and restart it.
  2. Restart the agent after editing its configuration.
Once the agent is restarted, it will use that new IP address.
Q:
How can I stop my agent from thinking the server keeps going up and down when the server has remained running the whole time?
A:
There can be errors like the following appears in the agent logs: 
INFO (org.rhq.enterprise.agent.AgentAutoDiscoveryListener)- {AgentAutoDiscoveryListener.server-offline}
The Agent has auto-detected the Server going offline [InvokerLocator
[servlet://server:7080/jboss-remoting-servlet-invoker
/ServerInvokerServlet?rhq.communications.connector.rhqtype=server]] -
the agent will stop sending new messages 
...
INFO (org.rhq.enterprise.agent.AgentAutoDiscoveryListener)- {AgentAutoDiscoveryListener.server-online}
The Agent has auto-detected the Server coming online [InvokerLocator
[servlet://server:7080/jboss-remoting-servlet-invoker
/ServerInvokerServlet?rhq.communications.connector.rhqtype=server]] -
the agent will be able to start sending messages now
This means the agent has auto-detected, through the multicast detector, the server going down and then back up. This is different from the detection-via-polling, which is the second way the agent attempts to detect the server's status.
If the agent is erroneously detecting the server going up or down, it is possible that either the network does not support multicast traffic or the multicast network is acting abnormally. In either case, disable the agent multicast detector and have the agent instead rely on polling to detect changes in the server status.
To turn off the multicast detection, set the following agent preferences to false:
  • rhq.agent.server-auto-detection
  • rhq.communications.multicast-detector.enabled
Since you are disabling multicast detection, ensure that the polling detection feature is enabled, meaning the rhq.agent.client.server-polling-interval-msecs value is larger than 0, typically 60000. Otherwise, the agent will never be able to know when the server goes down.
Once you reconfigure the agent, restart the agent so the communications subsystem can detect the changes.