-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for Red Hat JBoss Operations Network
Configuring JBoss ON Servers, Agents, and Storage Nodes
tuning server and agent configuration
Abstract
1. About JBoss Operations Network
1.1. About JBoss ON Agents
1.2. About JBoss ON Servers
- Stores the configuration for both resources and resource groups.
- Organizes and responds to the data collect by the agents.
- The inventory of resources
- The configured groups
- Monitoring data
- Configuration data
- Content available to resources
- User and access control information
1.3. About Metrics Storage Nodes
Note
- Dedicated CPU
- More available physical memory
- Faster disks
- More disk space
rhqctl install --storage
command. The storage node always requires a companion agent.
2. General Management
2.1. JBoss ON File Locations
2.1.1. Server File Locations
serverRoot | jon | ---------------------------------------------------------- | | | | | | | | alert-scripts/ bin/ etc/ EULA jbossas/ LICENSE logs/ plugins/
Table 1. Server Directories and Files
Configuration Area | Directory or File Location | Description |
---|---|---|
Configuration directory | serverRoot/bin/ | Contains the server start scripts, PID files, and configuration file. |
Start scripts | serverRoot/bin/rhqctl | The management script which is used to start and stop, install, and upgrade the server, storage nodes, and local agents. |
Server configuration file | serverRoot/bin/rhq-server.properties | The configuration file for all server settings that are not stored in the JBoss ON database. |
Storage node configuration file | serverRoot/bin/rhq-storage.properties | The configuration file for all storage node settings that are not stored in the JBoss ON database. |
Password hash script | serverRoot/bin/rhq-encode-password.{sh|bat} | For a migrated server, it generates an encoded form of the database password to use in the rhq-server.properties file. |
SNMP files | serverRoot/etc/RHQ-mib.txt | The JBoss ON MIB file to use for setting SNMP traps. |
Log files | serverRoot/logs/ | The JBoss ON server log files are automatically created in this directory. The current log is named rhq-server-log4j.log . Older log files are named rhq-server-log4j.log .#, and the higher the number, the older the log file. |
Custom plug-in deployment directory | serverRoot/plugins/ | The directory where custom plug-in files can be dropped for them to be automatically detected and polled by the JBoss ON server. |
JBoss AS directory | serverRoot/jbossas/ | Contains all of the required JBoss AS client and server libraries.[a] |
Server JAR files | serverRoot/jbossas/default/deploy/rhq.ear/ | Contains all of the JAR files used by JBoss ON servers, web interface, and clients. |
Server-side plug-ins directory | serverRoot/jbossas/default/deploy/rhq.ear/rhq-serverplugins/ | Contains all of the JAR files for the default JBoss ON server-side plug-ins. |
Agent plug-ins directory | serverRoot/jbossas/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/ | Contains all of the JAR files for the default JBoss ON agent plug-ins. |
Server-side plug-ins directory | serverRoot/jbossas/default/deploy/rhq.ear/rhq-serverplugins/ | Contains all of the JAR files for the default JBoss ON server-side plug-ins. |
Agent package directory | serverRoot/jbossas/default/deploy/rhq.ear/rhq-downloads/rhq-agent/ | Contains the snapshot packages for the JBoss ON agent. |
Web interface directory | serverRoot/jbossas/default/work/jboss.web/localhost/ | Contains the directories that hold the files for rendering the web interface. |
[a]
Most of the libraries and files in this directory don't relate directly to JBoss ON.
|
2.1.2. Agent File Locations
rhq-agent/
directory in that root directory.
serverRoot | rhq-agent/ | ------------------------------------ | | | | | | bin/ conf/ data/ lib/ logs/ plugins/
Table 2. Agent Directories and Files
Configuration Area | Directory or File Location | Description |
---|---|---|
Start scripts | serverRoot/rhq-agent/bin/ | Contains the agent start scripts. |
Configuration file | serverRoot/rhq-agent/conf/agent-configuration.xml | The configuration file for basic agent settings. |
Library files | serverRoot/rhq-agent/lib/ | Contains the libraries used by the agent to monitor resources. |
Start scripts | serverRoot/rhq-agent/logs/ | The JBoss ON agent log files are automatically created in this directory. The current log is named agent.log . Older log files are named agent.log .#, and the higher the number, the older the log file. |
Plug-ins directory | serverRoot/rhq-agent/plugins/ | Contains the plug-ins used by the agent for managing resources (like editing resource configuration). |
2.1.3. Storage Node File Locations
/opt/jon/jon-server-3.2.GA
.
serverRoot | jon-server-3.2.GA/ | ----------------------------------------- | | | | | | bin/ conf/ interface/ lib/ pylib/ tools/
/opt/jon | rhq-data/ | --------------------- | | | commit_log/ data/ saved_caches/
Table 3. Storage Directories and Files
Configuration Area | Directory or File Location | Description |
---|---|---|
Distributed database scripts | serverRoot/rhq-storage/bin/ | Contains the scripts used to manage the distributed database nodes. These are invoked most frequently by the JBoss ON agent which manages the storage node, not the administrator. |
PID file | serverRoot/rhq-storage/bin/cassandra.pid | The PID file for the node process. |
Configuration file | serverRoot/rhq-storage/conf/rhq-storage-auth.conf | The configuration file for basic storage node settings. |
Logging configuration file | serverRoot/rhq-storage/conf/log4j-server.properties | The configuration file for log4j for the storage node. |
Library files |
| Contains the libraries used by the storage node. |
Data files | serverRoot/rhq-data/ | The parent directory for the data, cache files, and backups for the distributed storage database notes. |
Data files | serverRoot/rhq-data/commit_log/ | Contains binary commit logs which store write operations before they are written to the storage database. |
Data files | serverRoot/rhq-data/saved_caches/ | Contains database cache files for each database in the data/ subdirectories. |
Data files | serverRoot/rhq-data/data/ | Contains data files and snapshots for the node, organized by the keyspace and the table. There are four keyspaces: rhq , system , system_auth , and system_traces . |
Data files | serverRoot/rhq-data/data/rhq/ | Contains data for all the metric tables. This is all of the new (raw) metrics and tables for each level of aggregation (one hour, six hour, and 24 hour). This also contains schema and index information for the version of metrics being collected. |
Data files | serverRoot/rhq-data/data/system/ | Contains information for the distributed database nodes. |
Data files | serverRoot/rhq-data/data/system_auth/ | Contains information for the credentials, users, and permissions configured for the distributed database. |
Data files | serverRoot/rhq-data/data/system_traces/ | Contains information on database sessions and events. |
2.2. Default Server, Agent, and Storage Node Ports
- Server to database communication
- The server has to be able to connect to its database. The database port number depends on both the type of database and the specific configuration for the database.
- Server to agent communication
- The server connects to an agent over a single port configured for the agent. This port is used for both standard and SSL communications between the server and agent.
- Agent to server communication
- An agent can talk to multiple JBoss ON servers, even if they use the same port (since each server is on a different host.) The agent will use either a standard port or an SSL port to connect to the JBoss ON server, depending on the connection (transport) method that is configured. The agent will only attempt to use a single port.
- Server to storage node communication
- The JBoss ON server receives the monitoring data from an agent, and it then forwards that information to an available storage node. THe server connects to the storage node over its client (CQL) port.
- Storage node to storage node communication
- For data integrity and availability, the monitoring data are regularly synchronized between the storage nodes. The nodes send data to one another over a communal cluster port, called a gossip port.
Note
Table 4. Default Ports
Connection Type | Port Number |
---|---|
Server to agent | 16163 |
Agent to server (standard) | 7080 |
Agent to server (secure) | 7443 |
Server to database | 5432 (default PostgreSQL port) |
Server to storage node | 9142 |
Storage node to storage node | 7100 |
2.3. The rhqctl Control Script
rhqctl
script has subcommands and options:
rhqctl [command] [[options]
--server
, --agent
, or --storage
.
Example 1. Installing Specific Services
install
command configures the JBoss ON server, storage node, and agent all at the same time.
install
command has options for each service. If that option is used, the only that service is installed; the other services are excluded.
[root@server bin]# ./rhqctl install --storage --start [root@server bin]# ./rhqctl install --server --start [root@server bin]# ./rhqctl install --agent --start
rhqctl
command is used to start and stop the instance, so start/stop commands are the most common ones to use. Running a command with no options runs that command for all JBoss ON instances on the system.
[root@server bin]# ./rhqctl start
[root@server bin]# ./rhqctl start --server --agent
Table 5. rhqctl Command
Command | Description |
---|---|
console | Starts a server or storage node in the foreground rather than as a service. |
install | Installs a server, storage node, or agent instance. |
restart | Restarts a JBoss ON service. If the type of service is not specified, then all local services are restarted. |
start | Starts a JBoss ON service. If the type of service is not specified, then all local services are started. |
status | Shows the status of a JBoss ON service. If the type of service is not specified, then the status of all local services is displayed. |
stop | Stops a JBoss ON service. If the type of service is not specified, then all local services are stopped. |
upgrade | Upgrades all JBoss ON services. This can be rerun to migrate historical metric data after a server migration. |
Table 6. rhqctl Options
Option | Description | Commands That Allow It |
---|---|---|
--start | Starts all services. |
|
--server | Runs the given command for the local server. If this is specified, then the command only is run for the server, and the other components are ignored (unless they are explicitly mentioned). |
|
--agent | Runs the given command for the agent. If this is specified, then the command only is run for the agent, and the other components are ignored (unless they are explicitly mentioned). |
|
--storage | Runs the given command for the storage database node. If this is specified, then the command only is run for the storage database node, and the other components are ignored (unless they are explicitly mentioned). |
|
--storage-data-root-dir directory | Changes the directory where the storage data are stored. By default, the storage node directory is serverRoot/jon-server-3.2.GA/rhq-data/ . |
|
--from-server-dir directory | Gives the directory path to the server to be upgraded. |
|
--from-agent-dir directory | Gives the directory path to the agent to be upgraded. The upgrade script assumes that the agent is installed in the same directory as the server, in a subdirectory named rhq-agent/ . If the agent is in a different location, then this option is required. |
|
--run-data-migrator | Migrates the data from an existing JBoss ON SQL database into a new monitoring storage database. The upgrade command creates the monitoring database. If this option is not used, than any previous monitoring data are lost. |
|
2.4. Starting the JBoss ON Server
2.4.1. Starting the JBoss ON Server (Basic)
rhqctl
script in the serverRoot/bin/
directory.
Important
start
command. This starts the server process and then exits from the script.
serverRoot/bin/rhqctl start --server Trying to start the RHQ Server... RHQ Server (pid 27547) is starting
rhqctl
script looks for specific environment variables during its execution, especially related to the JVM to use with the JBoss EAP server instance. A complete list of environment variables is given in the script itself; defaults based on the installation information are used, so most environment variables don't need to be reset.
Note
RHQ_SERVER_JAVA_HOME
environment variable must be set on Red Hat Enterprise Linux systems for the server to start. This can be set to a general value like /usr/
.
[root@server ~]# ./rhqctl console --server 20:28:44,120 INFO [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1 Starting RHQ Server in console... JAVA_OPTS already set in environment; overriding default settings with values: - ....
2.4.2. Running the JBoss ON Server as a Service
- Create a initalization scipt
/etc/init.d/jboss-on
:#!/bin/sh # # Example Red Hat JBoss Operations Network init script # # chkconfig: - 95 20 # description: Red Hat JBoss Operations Network rhqctl services # processname: standalone.sh # # # This is an example script that can be used with a SysV style # init service which starts a JBoss ON server, storage node, and # agent using the rhqctl script provided by the JBoss ON server # installation. prog="`basename "$0"`" # If available, the following files will be sourced, in order, for # configuration. This means that if the same value is defined in a later # file, it will override the one specified in an earlier one: # # /etc/jboss-on.conf # /etc/jboss-on/${prog}.conf # # The following environment variables can be defined: # # RHQ_HOME -- Defaults to /opt/jboss/jboss-on # RHQ_SERVER_HOME -- Defaults to RHQ_HOME/jon-server-* # RHQ_AGENT_HOME -- Defaults to RHQ_HOME/rhq-agent # RHQ_JAVA_HOME -- Defaults to /usr # RHQ_USER -- Defaults to jonadmin # # If available, source environment variables for the service # Global defaults/settings [ -r "/etc/jboss-on.conf" ] && . "/etc/jboss-on.conf" # Settings specific to this service [ -r "/etc/jboss-on/${prog}.conf" ] && . "/etc/jboss-on/${prog}.conf" # If not yet set, use some reasonable defaults: [ -z "${RHQ_HOME}" ] && RHQ_HOME='/opt/jboss/jboss-on' [ -z "${RHQ_SERVER_HOME}" ] && RHQ_SERVER_HOME="`eval echo "${RHQ_HOME}/jon-server-"*`" [ -z "${RHQ_AGENT_HOME}" ] && RHQ_AGENT_HOME="${RHQ_HOME}"'/rhq-agent' [ -z "${RHQ_JAVA_HOME}" ] && RHQ_JAVA_HOME='/usr' [ -z "${RHQ_USER}" ] && RHQ_USER=jonadmin # We have to export these variables so that they are available to child # processes. export RHQ_SERVER_HOME export RHQ_AGENT_HOME export RHQ_JAVA_HOME # This is a convenient function that invokes rhqctl rhqctl() { RHQCTL_SCRIPT="${RHQ_SERVER_HOME}/bin/rhqctl" [ ! -x "${RHQCTL_SCRIPT}" ] && { echo >&2 "${RHQCTL_SCRIPT}"' was not found or is not executable.' exit 2 } if [ -n "${RHQ_USER}" -a "$(whoami)" != "${RHQ_USER}" ]; then CMD="su -m ${RHQ_USER} -c '\"${RHQCTL_SCRIPT}\" $@'" else CMD="\"${RHQCTL_SCRIPT}\" $@" fi eval ${CMD} } # We invoke rhqctl differently if start or stop is used so we need this # case statement to determine what should happen. case "$1" in start) # Because rhqctl is so chatty/verbose we redirect standard output to # null so we do not see it during system start and stop. # Additionally, we invoke it in the background as it can block for # some time during start. rhqctl "$@" >/dev/null & ;; stop) # Because rhqctl is so chatty/verbose we redirect standard output to # null so we do not see it during system start and stop. rhqctl "$@" >/dev/null ;; *) # Assuming we are not starting or stopping, we will display complete # output. rhqctl "$@" ;; esac
- Change the permissions on the file so it can be executed:
chmod +x /etc/init.d/jboss-on
- Add the initalization script to chkconfig:
sudo chkconfig --add /etc/init.d/jboss-on
- To indicate that you intend for the script to run system startup and stop during shutdown:
sudo chkconfig jboss-on on
- To override the default values for
RHQ_HOME
,RHQ_SERVER_HOME
,RHQ_AGENT_HOME
,RHQ_JAVA_HOME
andRHQ_USER
, create the service configuration file/etc/jboss-on.conf
or/etc/jboss-on/<SERVICE NAME>.conf
with your specific values:RHQ_HOME=/usr/share/jboss-on RHQ_USER=jbosson
2.5. Starting the JBoss ON Agent
Important
2.5.1. Starting the JBoss ON Agent on a Managed System
bin/
directory. Unlike the server start script, which starts the server process and then exits the script, the agent script remains open, with a prompt to accept further input if necessary. (Usually, the script can simply be started and left to run in the background.)
/opt/rhq-agent/bin/rhq-agent.sh RHQ 3.2.0-SNAPSHOT [cda7569] (Tue Apr 13 13:39:16 EDT 2013) >
rhq-agent.sh
script are listed in Table 15, “Options for the rhq-agent.sh Script”. Additional configuration options can be set by editing the rhq-agent-env.sh
script file.
Note
--cleanconfig
to wipe the previous agent configuration and start fresh.
2.5.2. Starting an Agent Installed on a Server Machine
rhqctl
) that the server and storage node are. To start an agent, use the --agent
option with the script.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --agent
rhq-agent-env.bat
script, as described in Section 2.5.3, “Configuring the Agent as a Windows Service”. The agent on the server machine is still started and stopped using the rhqctl
command.
2.5.3. Configuring the Agent as a Windows Service
- Edit the
rhq-agent-env.bat
script and set the environment variable to define the system user as whom the init script will run. There are two options:RHQ_AGENT_RUN_AS
explicitly sets the user account name. This must match the format of a Windows user account name, DOMAIN\username.RHQ_AGENT_RUN_AS_ME
forces the agent to run as whoever the current user is; this uses the format . \ %USERNAME %. If both environment variables are defined, this variable overridesRHQ_AGENT_RUN_AS
.
Note
Before settingRHQ_AGENT_RUN_AS_ME
orRHQ_AGENT_RUN_AS
, make sure that the given user actually has permission to start services. If necessary, assign the user the appropriate rights. Assigning rights is covered in the Windows documentation.If neither variable is set, the agent init script runs as the System user.Other available environment variables are listed and defined in the comments in therhq-agent-wrapper.bat
script. - Run the
rhq-agent-wrapper.bat
script to install the init script as a service. Use theinstall
command to install the init script. - When prompted, fill in the password for the system user as whom the service will run.
rhq-agent-wrapper.bat
script using the start
and stop
commands. The status
command shows whether the agent init script is installed as a service and whether it is running. The remove
command removes the agent init script as a service.
\bin\wrapper\rhq-agent-wrapper.conf
, contains the service configuration properties. These are standard wrapper service properties; more information is available at http://wrapper.tanukisoftware.org/doc/english/properties.html.
wrapper.app.parameter.#
set command-line options to pass to the agent. These are the same options listed in Section 7.3, “Working with the Agent Command Prompt”. Each option requires its own configuration property. Properties must be placed in numeric order and the first two properties (wrapper.app.parameter.1
andwrapper.app.parameter.2
) are reserved. Start withwrapper.app.parameter.3
.wrapper.java.additional.#
set additional JVM options that are passed directly to the VM, corresponding to the-D
and-X
options. These also must be incremented numerically.wrapper.java.additional.1
always specifies the log configuration file.wrapper.ntservice.starttype
sets when to start the service. The default isAUTO_START
, which starts the service when the system boots. To start the service manually, the value isDEMAND_START
.
2.5.4. Running the Agent as a Daemon or init.d Service
Important
rhq-agent.sh
script starts the agent and opens the command console. The rhq-agent-wrapper.sh
script simply starts the agent daemon and exits. Both methods can have additional environment variables configured through the rhq-agent-env.sh
script file.
/etc/init.d
and then installing it using chkconfig
. For Solaris and other Unix systems, this is done by configuring /etc/init.d
and then using other system tools to set up the service.
- Make sure the agent is fully set up.
- Open the
rhq-agent-env.sh
file. - Uncomment and configure the required environment variables for the agent's
bin
directory, the JDK directory, and the PID directory (which must be writable by the agent user).RHQ_AGENT_HOME=agentRoot/rhq-agent/bin/ export RHQ_AGENT_JAVA_HOME=/usr PIDFILEDIR=/var/run
Note
When setting thePIDFILEDIR
on Red Hat Enterprise Linux, edit thepidfile
setting in therhq-agent-wrapper.sh
script file. The wrapper script value is used bychkconfig
. - Set any of the optional environment variables.
RHQ_AGENT_DEBUG
enables debug logging.RHQ_AGENT_JAVA_EXE_FILE_PATH
specifies a Java executable.RHQ_AGENT_JAVA_OPTS
passes settings to the agent JVM.RHQ_AGENT_ADDITIONAL_JAVA_OPTS
passes additional Java options to the JVM.
- Log into the system as root.
Important
The rest of this procedure describes how to configure the agent init script as a service on Red Hat Enterprise Linux. For other Unix systems, follow a similar procedure that corresponds to the specific platform. - Make sure the wrapper script is executable.
[root@server rhq-agent]# chmod a+x agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh
- Symlink the
rhq-agent-wrapper.sh
file to/etc/init.d/
. For example:# ln -s agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh /etc/init.d/rhq-agent-wrapper.sh
Important
On Solaris, symlinking the agent script file requires invokingreadlink
inrhq-agent-wrapper.sh
.readlink
is not supplied by default in some Solaris installations. Solaris users must downloadreadlink
from a source such as Sunfreeware. - Register
rhq-agent-wrapper.sh
withchkconfig
.# /sbin/chkconfig --add rhq-agent-wrapper.sh
- Enable the agent service to run at boot time and have it stop gracefully at when the system shuts down.
# /sbin/chkconfig rhq-agent-wrapper.sh on
chkconfig
:
# /sbin/chkconfig rhq-agent-wrapper.sh off
2.5.5. Starting an Agent with a Custom Command
su
or sudo
, allowing the agent to run as a different user.
rhq-agent-env.sh
file. There are two parts to the configuration: the start command itself and then a setting to enable a password prompt.
RHQ_AGENT_START_COMMAND="su -m test -c '${RHQ_AGENT_HOME}/bin/rhq-agent.sh'" RHQ_AGENT_PASSWORD_PROMPT=true
sudo
and agent user configuration. If it is required, then the password prompt should be enabled so that the user can enter the password or a password should be set in the RHQ_AGENT_PASSWORD
parameter; otherwise, the start process will appear to hang.
2.5.6. Restarting the Agent and the JVM
- Select the Resources menu in the top navigation bar, and select the Servers menu item.
- Click the agent resource in the list.
- Click the Operations tab.
- Select and launch the Restart task.
- Select the Resources menu in the top navigation bar, and select the Servers menu item.
- Click the agent resource in the list.
- Navigate to the launcher script child resource beneath the agent.
- Click the Operations tab for the launcher script resource.
- Select and launch the Restart task.
2.6. Starting the Storage Node
rhqctl
script in the serverRoot/bin/
directory for the JBoss ON server.
start
command. This starts the node process and then exits from the script.
[root@server bin]# ./rhqctl start storage 22:03:01,362 INFO [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1 INFO 22:03:03,138 Logging initialized INFO 22:03:03,177 JVM vendor/version: OpenJDK 64-Bit Server VM/1.6.0_24 ... INFO 22:03:51,215 Node /10.16.46.34 state jump to normal INFO 22:03:51,223 Startup completed! Now serving reads.
rhqctl
script looks for specific environment variables during its execution, especially related to the JVM for the storage node. A complete list of Java settings is given in the /opt/jon/jon-server-3.2.GA/rhq-storage/conf/cassandra-jvm.properties
file.
Note
RHQ_SERVER_JAVA_HOME
environment variable must be set on Red Hat Enterprise Linux systems for the server to start. This can be set to a general value like /usr/
.
[root@server ~]# ./rhqctl console --server 20:28:44,120 INFO [org.jboss.modules] JBoss Modules version 1.2.2.Final-redhat-1 Starting RHQ Storage in console... JAVA_OPTS already set in environment; overriding default settings with values: - ....
3. Tuning JBoss ON for Better Performance
Note
3.1. Inventory Baselines
Important
A server could typically upgrade with 100 or more agents before encountering out of memory errors. If those errors are encountered, the server settings can be tuned as described in Section 3.6, “Server Tuning for Large Numbers of Agents” to increase the thread pool and concurrency limits to allow more requests to be processed.
The resources on a system can be arranged in two slightly different ways. One way is a flat hierarchy, where there are fewer levels in the hierarchy and each level is very broad. Essentially, this is few parent resources with lots of child resources.
platform | ------------------------------------------------ | | | | | | server1 server2 server3 server4 server5 ... | ----------------------------------------------------- | | | | | | service1 service2 service3 service4 service5 ...
platform | --------------------- | | | server1 server2 server3 | ------------- | | service1 service2 | ------------ | | serviceA serviceB | ------------ | | serviceI serviceII | ------------ | | serviceX serviceY | --- | service...
Table 7. Agent Import Performance
Resource Hierarchy | Tested Import Time | Notes |
---|---|---|
Deep hierarchy: 10 top-level servers, 10 mid-level servers, 750 child services (10x10x750)
(75,000 total resources)
| 2 hours 46 minutes | Increasing the agent heap size to 2GB moderately decreased the import time, to 2 hours 34 minutes. It also reduced the risk of out-of-memory errors, which could occur when importing a large number of resources with the normal agent memory settings. |
Flat hierarchy: 100 top-level servers, 750 child services (10x10x750)
(75,000 total resources)
| 2 hours 14 minutes |
A single agent can manage 40 to 50 JBoss Enterprise Application Platform 6 instances.
Note
3.2. Monitoring and Alerting Baselines
- Database performance, which is the primary factor in most environments
- Network bandwidth
- Up to 30,000 metrics can be collected per minute
- Up to 100,000 alerts can be fired per day (roughly 70 per minute)
Note
3.3. Tuning the Agent JVM Memory Size
RHQ_AGENT_JAVA_OPTS
in the rhq-agent-env.sh
file to set the appropriate JVM settings.
- Stop the agent. For example, if the agent is running as a service:
[root@server ~]# service rhq-agent-wrapper.sh stop
- Open the
rhq-agent-env.sh
file to set the required environment variables that the agent will use.[root@server ~]# vim agentRoot/rhq-agent/bin/rhq-agent-env.sh
- Set the
RHQ_AGENT_JAVA_OPTS
value with the-Xms
and-Xmx
parameters for the minimum and maximum bounds of the heap size for the agent JVM. For example:RHQ_AGENT_JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=256M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true"
- Optionally, use
-XX:PermSize
and-XX:MaxPermSize
to set the perm gen size. - Restart the agent process to load the new configuration. For example, if the agent is running as a service:
[root@server ~]# service rhq-agent-wrapper.sh start
3.4. Changing the Agent JVM Health Check Settings
Note
- The scan interval,
rhq.agent.vm-health-check.interval-msecs
- The heap threshold,
rhq.agent.vm-health-check.low-heap-mem-threshold
- The non-heap memory threshold,
rhq.agent.vm-health-check.low-nonheap-mem-threshold
- Open the agent prompt. Use the
-n
option to open the prompt without[root@server ~]# agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
with the name of the preference to edit and its new value.> setconfig rhq.agent.vm-health-check.interval-msecs=3000 Set preference: rhq.agent.vm-health-check.interval-msecs=3000 > setconfig rhq.agent.vm-health-check.low-heap-mem-threshold=1.1 Set preference: rhq.agent.vm-health-check.low-heap-mem-threshold=1.1 > setconfig rhq.agent.vm-health-check.low-nonheap-mem-threshold=1.1 Set preference: rhq.agent.vm-health-check.low-nonheap-mem-threshold=1.1
- Restart the agent process to load the new configuration.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --agent
3.5. Tuning the Server JVM
- As root, increase the user thread limit for the system.
[root@server ~]# ulimit -u 4096
- Open the
rhq-server.sh
file to set the new JVM settings.[root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-server.sh
- Change the Java options to increase the heap size,
-Xms
for the minimum heap and-Xmx
for the maximum. For example:RHQ_CONTROL_JAVA_OPTS="
-Xms512M -Xmx1024M
-XX:MaxPermSize=128M -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true" - Restart the server to load the new heap settings.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --server
3.6. Server Tuning for Large Numbers of Agents
- Increase the default size of the storage node memory usage. This is only necessary for a large number of nodes, around 1,000 or more.
- Open the
rhq-storage.properties
file.[root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-storage.properties
- Uncomment the
heap-size
parameter and set the heap size to 5GB.rhq.storage.heap-size=5120
- Increase the EJB pool.
- Open the server's
standalone-full.xml
profile.[root@server ~]# vim /opt/jon/jon-server-3.2.GA/jbossas/standalone/configuration/standalone-full.xml
- Change the
strict-max-pool
key to increase the pool size. The default is 20. For example:<strict-max-pool name="slsb-strict-max-pool" max-pool-size="2000" instance-acquisition-timeout="1" instance-acquisition-timeout-unit="MINUTES"/>
- Increase the concurrency limits to increase how many agents can communicate with the server simultaneously.
- Open the
rhq-server.properties
file.[root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-server.properties
- There is a block of communication-related parameters. Concurrency limits are set in the
concurrency-limit
parameters and therhq.communications.global-concurrency-limit
parameter. There are other communication limits for web UI connections and downloads. The different communication parameters are covered in Section 6.3.3, “Setting Concurrency Limits”.For example:rhq.server.startup.web.max-connections=1000 rhq.server.agent-downloads-limit=45 rhq.server.client-downloads-limit=5 rhq.communications.global-concurrency-limit=200 rhq.server.concurrency-limit.inventory-report=25 rhq.server.concurrency-limit.availability-report=25 rhq.server.concurrency-limit.inventory-sync=25 rhq.server.concurrency-limit.content-report=25 rhq.server.concurrency-limit.content-download=25 rhq.server.concurrency-limit.measurement-report=25 rhq.server.concurrency-limit.measurement-schedule-request=25 rhq.server.concurrency-limit.configuration-update=25
- Restart the server to load the new settings.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl restart --server
Note
3.7. Database and Disk Space
4. Configuring SSL Connections for Server-Agent Communication
- Encryption specially encodes the data sent between agents and servers during a session.
- Authentication uses SSL server and client certificates to verify the identity of an agent before it connects to a server, and vice versa.
Note
4.1. Setting up Encryption
sslservlet
and sslsocket
.
- First, enable SSL encryption on the JBoss ON server.
- Shut down the JBoss ON server.
serverRoot/jon-server-3.2.GA/bin/rhqctl.sh stop
- Open the
serverRoot/jon-server-3.2.GA/bin/rhq-server.properties
file for the JBoss ON server. - Edit the
rhq.communications.connector.*
settings to use SSL. To use thesslsocket
transport method, which is recommended for authentication, update therhq.communications.connector.transport
method, set the port number to use for the socket, and remove the servlet specified in the transport parameters setting.rhq.communications.connector.transport=sslsocket
rhq.communications.connector.bind-address=rhq.communications.connector.bind-port=55555
rhq.communications.connector.transport-params=To use thesslservlet
transport method, all that's necessary is to change therhq.communications.connector.transport
method.rhq.communications.connector.transport=sslservlet
rhq.communications.connector.bind-address= rhq.communications.connector.bind-port= rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet - For setting encryption alone, make sure that certificate-based authentication is disabled:
rhq.server.tomcat.security.client-auth-mode=false rhq.server.client.security.server-auth-mode-enabled=false
- Optionally, define the secure protocol to use. The default is TLS (which is usually fine), but you can set it to SSL.
rhq.server.tomcat.security.secure-socket-protocol=TLS rhq.server.client.security.secure-socket-protocol=TLS
- Save the changes, and restart the JBoss ON server.
serverRoot/jon-server-3.2.GA/bin/rhqctl start
- Verify that the end point address and port number given in the configuration are actually the settings set for the server in JBoss ON.
- Click the Administration tab in the top menu.
- In the Topology box on the left, select the Servers item.
- Check the port number in the Secure Port column.
- If the value is wrong, click the name of the server to open the edit page.
- Click the Edit under the server information, and reset the end point address or port as necessary.
- Then, enable SSL encryption in the agent.
Note
This shows how to edit the agent configuration by editing the agent configuration file. The agent configuration can also be edited by going through the advanced setup mode in the agent start script:agentRoot/rhq-agent/bin/rhq-agent.sh --cleanconfig --setup --advanced
- Open the agent configuration file:
vim agentRoot/rhq-agent/conf/agent-configuration.xml
- Change the transport protocol to
sslsocket
.<entry key="rhq.communications.connector.transport"
value="sslsocket"
/> - Set the server connection information so that it matches the configuration for the server. The bind address for the server is commented out by default, and the other parameters are set to the JBoss ON server defaults, including using
sslservlet
for the server's transport protocol.<entry key="rhq.communications.connector.transport"
value="sslsocket"
/> <entry key="rhq.agent.server.transport"value="sslservlet"
/> <entry key="rhq.agent.server.bind-port"value="55555"
/> <entry key="rhq.agent.server.bind-address"value="server.example.com"
/> <entry key="rhq.agent.server.transport-params"value=""
/> - For setting encryption alone, make sure that certificate-based authentication is disabled. These parameters can be left commented out or can be explicitly set to turn off authentication.
<entry key="rhq.communications.connector.security.client-auth-mode"
value="none"
/> <entry key="rhq.agent.client.security.server-auth-mode-enabled"value="false"
/> - Optionally, define additional protocol settings for the agent. This is necessary if the server is configured to use transport protocols other than TLS.
<entry key="rhq.communications.connector.security.secure-socket-protocol" value="TLS" /> <entry key="rhq.agent.client.security.secure-socket-protocol" value="TLS" />
- Exit the agent and restart it, using the
--cleanconfig
option to load the new configuration.agentRoot/rhq-agent/bin/rhq-agent.sh --cleanconfig
4.2. Setting up Client Authentication Between Servers and Agents
Note
sslservlet
and sslsocket
.
sslsocket
, which allows the default given port to be used for GUI connections while a special port is used for server-agent SSL connections.
JBOSS_HOME/bin/jboss-cli.\[sh,bat]
.
Procedure 1.
jboss-cli.sh --connect --command='/socket-binding-group=standard-sockets/socket-binding=httpsbrowser/:add(port=9443)'
jboss-cli.sh --connect --command='/subsystem=web/connector=httpsbrowser/:add(socket-binding=httpsbrowser,scheme=https,protocol=HTTP/1.1,secure=true,enabled=true)'
jboss-cli.sh --connect --command='/subsystem=web/connector=httpsbrowser/ssl=configuration:add(name=ssl,verify-client=false,key-alias=JON,password=JONManagement,certificate-key-file=${jboss.server.config.dir}/rhq.keystore,certificate-file=${jboss.server.config.dir}/rhq.keystore'
jboss-cli.sh --connect --command='/:reload'
- Enable encryption, as in Section 4.1, “Setting up Encryption”, only make sure that client authentication is not disabled.
- SSL socket connections will occur over a user-defined port. If necessary, open the firewall or VPN to allow access to that port.
- Generate SSL certificates for each JBoss ON server and agent. For example:
keytool -genkey -dname "CN=server1.example.com" -keystore server1-keystore.dat -validity 3650 -alias server1 -keyalg DSA -storetype JKS -keypass secret -storepass secret
This creates a self-signed certificate with the following characteristics:- A common name (CN) value that is the same as the server hostname,
server1.example.com
. The-dname
value must be the same as the hostname because during the initial steps of the SSL connection (the SSL handshake), the client will verify that the same identity which was issued the certificate is the same as the one presenting it. Meaning, it will match the hostname in the CN against the hostname of the server or agent presenting the certificate. - A keystore file called
server1-keystore.dat
- A validity period of 3650 days
- An alias of
server1
- A key algorithm of DSA
- Stored in the JKS format in the keystore
- Key and storage passwords of
secret
Your organization may have a method already for generating or obtaining certificates. This example useskeytool
; other utilities, likecertutil
, can be used as well. Thekeytool
documentation is available through the Oracle-Sun site at http://java.sun.com/javase/6/docs/technotes/tools/windows/keytool.html. - Put each self-signed certificate in a single truststore file.
- Export the self-signed certificate from each keystore:
keytool
-export
-keystore server1-keystore.dat -alias server1 -storetype JKS -storepass secret-file server1-cert
- Import every certificate into a single truststore file:
keytool
-import
-keystore truststore.dat -alias server1 -storetype JKS-file server1-cert
-noprompt -keypass secret -storepass secret-alias
is the name to give to the imported certificate in the truststore. For convenience, this is the same as the alias of the original keystore file.Important
Import every exported server and agent certificate into the same truststore file. - Verify that all the certificates were successfully imported by using the
keytool
to list the certificates:keytool -list -keystore truststore.dat -storepass secret -storetype JKS Keystore type: JKS Keystore provider: SUN Your keystore contains 2 entries server2, Feb 25, 2013, trustedCertEntry, Certificate fingerprint (MD5): 24:D9:8A:50:BA:1B:26:08:DC:44:A8:2A:9E:8A:43:D9 server, Feb 25, 2013, trustedCertEntry, Certificate fingerprint (MD5): 91:F8:78:15:21:E8:0C:73:EC:B6:3B:1D:5A:EC:2B:01
- Distribute both the keystore and the truststore files to all the JBoss ON and server and agent machines. Be sure to distribute the keystores only to the machines which match the hostname in the CN of the certificate; putting the keystore on the wrong machine will cause SSL connections to fail.
- For the server, copy the keystore into the
serverRoot/jon-server-3.2.GA/jbossas/server/default/conf/
directory of the JBoss AS server embedded in the JBoss Operations Network server. Make sure this file is namedkeystore.dat
. - For the server, copy the truststore into the
serverRoot/jon-server-3.2.GA/jbossas/server/default/conf/
directory of the embedded JBoss AS server. Make sure this file is namedtruststore.dat
. - For the agent, copy the keystore into the
agentRoot/rhq-agent/conf
directory. Any certificate file in theagentRoot/rhq-agent/conf
directory is retained even after an automatic update.
- Shut down the JBoss ON server.
serverRoot/jon-server-3.2.GA/bin/rhqctl.sh stop
- Open the
rhq-server.properties
file for the JBoss ON server.vim serverRoot/jon-server-3.2.GA/bin/rhq-server.properties
- Enable client authentication by setting the
rhq.communications.connector.security.client-auth-mode
parameter toneed
and therhq.server.client.security.server-auth-mode-enabled
parameter totrue
.Set the information about the keystore and truststore files.All of the configuration for incoming messages (agent-to-server communications) is set inrhq.communications.connector.security.*
parameters. The configuration for outgoing messages is set inrhq.server.client.security.*
parameters.# Server-side SSL Security Configuration (for incoming messages from agents) # These are used when secure transports other than sslservlet are used rhq.communications.connector.security.secure-socket-protocol=TLS rhq.communications.connector.security.keystore.file=${jboss.server.home.dir}/conf/keystore.dat rhq.communications.connector.security.keystore.algorithm=SunX509 rhq.communications.connector.security.keystore.type=JKS rhq.communications.connector.security.keystore.password=secret rhq.communications.connector.security.keystore.key-password=secret rhq.communications.connector.security.keystore.alias=server1 rhq.communications.connector.security.truststore.file=${jboss.server.home.dir}/conf/truststore.dat rhq.communications.connector.security.truststore.algorithm=SunX509 rhq.communications.connector.security.truststore.type=JKS rhq.communications.connector.security.truststore.password=secret rhq.communications.connector.security.client-auth-mode=need ... # Client-side SSL Security Configuration (for outgoing messages to agents) rhq.server.client.security.secure-socket-protocol=TLS rhq.server.client.security.keystore.file=${jboss.server.home.dir}/conf/keystore.dat rhq.server.client.security.keystore.algorithm=SunX509 rhq.server.client.security.keystore.type=JKS rhq.server.client.security.keystore.password=secret rhq.server.client.security.keystore.key-password=secret rhq.server.client.security.keystore.alias=myhost rhq.server.client.security.truststore.file=${jboss.server.home.dir}/conf/truststore.dat rhq.server.client.security.truststore.algorithm=SunX509 rhq.server.client.security.truststore.type=JKS rhq.server.client.security.truststore.password=secret rhq.server.client.security.server-auth-mode-enabled=true
- Save the file and restart the server.
serverRoot/jon-server-3.2.GA/bin/rhqctl start
- In the agent configuration file, uncomment the lines related to secure connections. These parameters begin with
rhq.communications.connector.security.*
andrhq.agent.client.security.*
for agent-to-server communications and server-to-agent connections, respectively.Fill in the appropriate values.<entry key="rhq.communications.connector.security.secure-socket-protocol" value="TLS" /> <entry key="rhq.communications.connector.security.keystore.file" value="conf/keystore.dat" /> <entry key="rhq.communications.connector.security.keystore.algorithm" value="SunX509" /> <entry key="rhq.communications.connector.security.keystore.type" value="JKS" /> <entry key="rhq.communications.connector.security.keystore.password" value="rhqpwd" /> <entry key="rhq.communications.connector.security.keystore.key-password" value="rhqpwd" /> <entry key="rhq.communications.connector.security.keystore.alias" value="rhq" /> <entry key="rhq.communications.connector.security.truststore.file" value="conf/truststore.dat" /> <entry key="rhq.communications.connector.security.truststore.algorithm" value="SunX509" /> <entry key="rhq.communications.connector.security.truststore.type" value="JKS" /> <entry key="rhq.communications.connector.security.truststore.password" value="" /> <entry key="rhq.communications.connector.security.client-auth-mode" value="none" /> <entry key="rhq.agent.client.security.secure-socket-protocol" value="TLS" /> <entry key="rhq.agent.client.security.keystore.file" value="conf/keystore.dat" /> <entry key="rhq.agent.client.security.keystore.algorithm" value="SunX509" /> <entry key="rhq.agent.client.security.keystore.type" value="JKS" /> <entry key="rhq.agent.client.security.keystore.password" value="rhqpwd" /> <entry key="rhq.agent.client.security.keystore.key-password" value="rhqpwd" /> <entry key="rhq.agent.client.security.keystore.alias" value="rhq" /> <entry key="rhq.agent.client.security.truststore.file" value="conf/truststore.dat" /> <entry key="rhq.agent.client.security.truststore.algorithm" value="SunX509" /> <entry key="rhq.agent.client.security.truststore.type" value="JKS" /> <entry key="rhq.agent.client.security.truststore.password" value="" /> <entry key="rhq.agent.client.security.server-auth-mode-enabled" value="false" />
Note
This shows how to edit the agent configuration by editing the agent configuration file. The agent configuration can also be edited by going through the advanced setup mode in the agent start script:agentRoot/rhq-agent/bin/rhq-agent.sh --cleanconfig --setup --advanced
4.3. Troubleshooting SSL Problems
4.3.1. Common SSL Connection Issues
- Make sure that both the agent and the server hostnames are resolvable to the hostnames in their server certificates.
- Make sure that port number given for the server's secure port is actually the port number configured for the server. Check the Administration > High Availability > Servers page and verify that the public endpoint address and port are correct. Edit the server definition in the UI so they are the same as the SSL configuration.
Figure 1. Server Hostname and Port Configuration
If these values do not match the same values configured for the SSL connection, the agent will not be able to talk to the server. - Make sure that both the agent and the server hostnames are resolvable to the hostnames in their server certificates.
- Make sure that every certificate that is used for agent-server communication is stored in the requisite keystores with the proper aliases.
- Check that the password is properly set to access the keystore.
- Make sure that the communication is set to use TLS.
- Validate the server and agent configuration, especially the assigned transport (socket or servlet) options. There are examples of configuration in Section 4.3.3, “Example SSL Configuration”.
- If client authentication is required and the server is using the
sslservlet
transport option, make sure that every user who connects to the JBoss ON UI has an installed user certificate so that they can connect to the server UI using client authentication. As with the agent certificate, the user certificates must be stored in the server's keystore, Section 4.2, “Setting up Client Authentication Between Servers and Agents”.If users are unable to connect using client authentication, then change the server to usesslsocket
instead ofsslservlet
.
4.3.2. Enabling SSL Debugging
- Open the agent environment variable file. This defines some settings for the JVM which the agent runs in, including debug log settings.
vim agentRoot/rhq-agent/bin/rhq-agent-env.sh
- Add a
RHQ_AGENT_ADDITIONAL_JAVA_OPTS
line to set a debug environment variable.RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Djavax.net.debug=all"
- Restart the agent.
agentRoot/rhq-agent/bin/rhq-agent.sh
4.3.3. Example SSL Configuration
Example 2. Encryption Only: Server (sslservlet) and Agent (sslsocket)
Server Configuration | Agent Configuration |
---|---|
rhq.communications.connector.transport=sslservlet rhq.communications.connector.bind-address= rhq.communications.connector.bind-port= rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet rhq.server.tomcat.security.client-auth-mode=false rhq.server.client.security.server-auth-mode-enabled=false |
<entry key="rhq.communications.connector.transport" value="sslsocket" /> <entry key="rhq.agent.server.transport" value="sslservlet" /> <entry key="rhq.agent.server.bind-port" value="7443" /> |
sslservlet
or sslsocket
. The agent can only receive incoming messages over sslsocket
.
Example 3. Encryption Only: Server (sslsocket) and Agent (sslsocket)
Server Configuration | Agent Configuration |
---|---|
rhq.communications.connector.transport=sslsocket rhq.communications.connector.bind-address= rhq.communications.connector.bind-port=7800 rhq.communications.connector.transport-params= rhq.server.tomcat.security.client-auth-mode=false rhq.server.client.security.server-auth-mode-enabled=false |
<entry key="rhq.agent.server.transport" value="sslsocket" /> <entry key="rhq.agent.server.bind-port" value="7800" /> <entry key="rhq.agent.server.transport-params" value="" /> |
rhq-server.properties
file.
Example 4. Encryption and Client Authentication: Server (sslservlet) and Agent (sslsocket)
Server Configuration | Agent Configuration |
---|---|
rhq.communications.connector.transport=sslservlet rhq.communications.connector.bind-address= rhq.communications.connector.bind-port= rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet rhq.server.tomcat.security.client-auth-mode=true rhq.server.client.security.server-auth-mode-enabled=true |
<entry key="rhq.communications.connector.transport" value="sslsocket" /> <entry key="rhq.agent.server.transport" value="sslservlet" /> <entry key="rhq.agent.server.bind-port" value="7443" /> |
Example 5. Encryption and Client Authentication: Server (sslsocket) and Agent (sslsocket)
Server Configuration | Agent Configuration |
---|---|
rhq.communications.connector.transport=sslsocket rhq.communications.connector.bind-address= rhq.communications.connector.bind-port=55555 rhq.communications.connector.transport-params= rhq.communications.connector.security.client-auth-mode=true rhq.server.client.security.server-auth-mode-enabled=true |
<entry key="rhq.agent.server.transport" value="sslsocket" /> <entry key="rhq.agent.server.bind-port" value="55555" /> <entry key="rhq.agent.server.transport-params" value="" /> |
5. Using High Availability and Agent Load Balancing
5.1. About Agent-Server Communication and Server Availability
5.1.1. Agents and Server Communication
Note
5.1.2. Server Availability: Multiple Servers in a Single Cloud
- There are problems processing the agent load, which can impact evaluating metrics, generating alerts or events, or reporting resource availability. This is not necessarily because of the number of agents; it could be related to network quality or other factors.
- You have a geographically distributed environment with multiple data centers or logical grouping of agents to servers.
Important
- All servers must be running the same version of JBoss ON.
- All servers must be uniquely named. This string is defined during server installation.
- Each server must define a unique endpoint (hostname or IP address) that is resolvable by all JBoss ON agents running against the high availability server cloud.
- Optional. Adjust the concurrency limits on the servers to prevent creating too much load on the database and damaging performance.
5.1.3. Agents and Server Partitions: Distributing Agent Load
Figure 2. All on a Single Server
A1: S1, S2, S3 A2: S2, S3, S1 A3: S3, S1, S2
Figure 3. Partitions: Agent Load Distributed Among Multiple Servers
5.1.4. Agents and Preferred Servers: Affinity and Load Balancing
Figure 4. Affinity Preferences for Agent Load
Note
Figure 5. Failover with Affinity
- Physical or network efficiency. Generally, if certain agent-server connections clearly run more efficiently than others, then defining affinity to prefer those connections makes sense. This could include servers and agents co-located in the same data center, geographic grouping, or network topology.
- Logical organization. There may be organizational reasons, apart from operating efficiency, to group specific agents and servers together, such as administrative responsibilities or business unit assignments.
- Warm backup. It may be the case that certain machines should not be assigned agent load unless specifically needed for failover purposes. In this case, all agents should be assigned affinity to a subset of available servers, leaving some servers without any associated agents in normal operation.
5.1.5. Agents and Server Failover
5.2. Creating Affinity Groups
Important
Note
Note
- Click the Administration tab in the top menu.
- In the Topology menu table on the left, select the Affinity Groups item.
- Click the CREATE NEW button.
- Enter a name for the new affinity group, and click OK.
- In the new affinity group's details page, click the EDIT GROUP AGENTS button.
- In the lower section, Agents not part of an affinity group, click the checkboxes by the agent names to add to the group, and click ADD TO GROUP.
- Click the Return to Affinity Group Link.
- As with the agents, click the EDIT GROUP SERVERS button to open the server lists and look at the list in the lower section of servers which do not currently belong to the affinity group. Click the checkboxes by the server names to add to the group, and click ADD TO GROUP.
5.3. Putting Servers in Maintenance Mode
- Click the Administration tab in the top menu.
- In the Topology menu table, select the Servers item.
- Select the check box next to the name of the JBoss ON server to put into maintenance mode.
- Click the SET MAINTENANCE button.
5.4. Removing Servers from the High Availability Cloud
- Click the Administration tab in the top menu.
- In the Topology menu table, select the Servers item.
- Select the check box next to the name of the JBoss ON server to remove from the cloud, and click SET MAINTENANCE.
- When the screen refreshes, select the check box next to the name of the JBoss ON server again, and click the REMOVE SELECTED button.
5.5. Managing Partition Events
5.5.1. Viewing Partition Events
- Click the Administration tab in the top menu.
- In the Topology menu table on the left, select the Partition Events item.
- The partition events page lists all of the events that have been recorded. (Table 9, “Partition Events Entries” describes the different fields.) Click the type name of any partition event opens up that record with more information about how the partition assignments were affected.The partition events log can be filtered to display entries which match the type, status, or details in the event record.
- Affinity group changes
- Agent events
- Server events
- Partition changes
Table 8. Types of Partition Events
Partition Event | Description |
---|---|
Affinity Group Changes | |
AFFINITY_GROUP_CHANGE | Registers a change in the agent or server assignments for an affinity group or that a group was added. |
AFFINITY_GROUP_DELETE | Registers an affinity group was deleted from the JBoss ON configuration. |
AGENT_AFFINITY_GROUP_ASSIGN | Registers that an agent was added to an affinity group. |
AGENT_AFFINITY_GROUP_REMOVE | Registers that an agent was removed from an affinity group. |
SERVER_AFFINITY_GROUP_ASSIGN | Registers that a server was added to an affinity group. |
SERVER_AFFINITY_GROUP_REMOVE | Registers that a server was removed from an affinity group. |
Agent Events | |
AGENT_CONNECT | Shows that a JBoss ON agent was started. |
AGENT_SHUTDOWN | Shows that a JBoss ON agent was stopped. |
AGENT_LEAVE | Shows that a JBoss ON agent was permanently removed from the JBoss ON configuration. |
AGENT_REGISTRATION | Shows that a new JBoss ON agent was added to the JBoss ON configuration. |
Server Events | |
SERVER_DELETION | Shows that a JBoss ON server was permanently removed from the JBoss ON configuration. |
SERVER_COMPUTE_POWER_CHANGE | |
OPERATION_MODE_CHANGE | Shows that a server went stopped, was started, or was newly installed. The type also shows how the mode transitioned (such as server.example.com: DOWN --> NORMAL ). |
Partition Changes | |
SYSTEM_INITIATED_PARTITION | Shows that JBoss ON initiated a repartition of the servers. |
ADMIN_INITIATED_PARTITION | Shows that a JBoss ON user initiated a repartition of the servers. |
Table 9. Partition Events Entries
Field | Description |
---|---|
Execution Time | The time of the partition event. |
Type | Shows the partition event type. This indicates what happened in the system affecting agent partition. |
Details | Gives detailed information about the partition event; the type of information given varies based on the partition event type. |
Initiated By | Shows the name of the user who initiated the action generating the partition event. Events initiated by the JBoss ON server itself show they were initiated by admin . |
Execution Status | Shows low for status descriptions. There are four different status settings:
|
6. Configuring Servers
- In the JBoss ON GUI
Note
Settings that can be edited in the JBoss ON UI must be edited in the JBoss ON UI. - In the
rhq-server.properties
configuration file
6.1. Enabling Debug Logging for the JBoss ON Server
serverRoot/jon-server-3.2.GA/logs/rhq-server-log4j.log
.
6.1.1. Setting the Debug Environment Variable
RHQ_SERVER_DEBUG
to any value. After setting this variable when you start the launcher, scripts will output debug messages.
RHQ_SERVER_DEBUG
environment variable to any value before starting the server.
6.1.2. Dumping Current Server State to the Logs
- Click the Administration tab in the top menu.
- In the Configuration menu table on the left, select the System Settings item.
- In the main window, scroll to the bottom of the server configuration, and click the Dump system info button.
- All of the current server settings and details are printed to the server log.
2012-05-14 19:44:28,587 INFO [SystemInfoManager] SystemInformation: ******** CAM_LDAP_BIND_PW: [- non null -] AlertDefinitionCount: [2] CAM_LDAP_BASE_DN: [o=JBoss,c=US] AVAILABILITY_PURGE: [31536000000] CAM_JAAS_PROVIDER: [false] BuildNumber: [ca099bc:3a46aff] ServerCount: [27] DATABASE_DRIVER_NAME: [PostgreSQL Native Driver] RESOURCE_GENERIC_PROPERTIES_UPGRADE: [false] ... 8< ...
6.2. Changing the JBoss ON Server URL
- Connecting to the GUI
- Details on alerts, contained in email notifications of alerts
- Click the Administration tab in the top menu.
- In the Configuration menu table on the left, select the System Settings item.
- Scroll to the JON General Configuration Properties section in the main work area.
- Change the hostname or IP address in the GUI Console URL field.
- Click Save.
6.3. Editing JBoss ON Server Configuration in rhq-server.properties
rhq-server.properties
configuration file in the serverRoot/jon-server-3.2.GA/bin
directory or in the JBoss ON database. The configuration file contains most of the basic information about the JBoss ON server, such as the TCP/IP ports it listens on and its hostname or IP address.
rhq-server.properties
file when the server starts. The initial configuration is defined by the installer when the JBoss ON program is set up.
Note
rhq-server.properties
when the JBoss ON server starts up, you have to restart the JBoss ON server after making any changes to that configuration file so the new settings are loaded.
6.3.1. Properties Set at Installation
This sets the type or vendor of the database that is used by the JBoss ON server. This is either PostgreSQL or Oracle.
This gives the JDBC URL that the JBoss ON server uses when connecting to the database, such as jdbc:postgresql://localhost:5432/rhq or jdbc:oracle:oci:@localhost:1521:orcl.
This gives the fully qualified class name of the JDBC driver that the JBoss ON server uses to communicate with the database, such as oracle.jdbc.driver.OracleDriver.
This gives the fully qualified class name of the JDBC driver that the JBoss ON server uses to communicate with the database, such as org.postgresql.xa.PGXADataSource or oracle.jdbc.xa.client.OracleXADatasource.
This gives the name of the user that the JBoss ON server uses when logging into the database. This user must already exist in the database; according to the basic installation instructions, this is a specially-created rhqadmin
user (not related to the super user in JBoss ON).
This gives the password of the database user that is used by the JBoss ON server when logging into the database. This password is stored in a hash in the rhq-server.properties
file. When the password is changed in the database, then the password must be manually hashed and copied into the rhq-server.properties
file. This is described in Section 8.2, “Changing Database Passwords”.
This sets a unique name for the JBoss ON server. The default is the system's hostname, but it can be any string, as long as it is unique within the JBoss ON server cloud.
Note
rhq-server.properties
file.
This gives the public IP address to use for the server. This is the address that must be recognized by all agents needing access to this server. By default, this is the value of the localhost's public IP address. The public IP address is used with the HTTP/HTTPS ports to provide a high availability endpoint for agents.
Note
rhq-server.properties
file.
This sets the port that the JBoss ON GUI listens to for unsecured HTTP requests. This is the port number in the JBoss ON GUI URL, such as the 7080 in http://localhost:7080. This is also the unsecured port used as the endpoint in high availability.
This sets the port that the JBoss ON GUI listens to for secured HTTPS requests. This is the port number used in the JBoss ON GUI URL, such as the 7443 in https://localhost:7443. This is also the secure port used as the endpoint in high availability.
This gives the IP address for the JBoss ON GUI console, among other services, to bind to. This is the host in the JBoss ON GUI URL, such as server.example.com
in http://server.example.com:7080
.
This sets the hostname of the SMTP server used by the JBoss ON server. Emails are sent primarily for alert notifications.
This sets the address to use for the From header of all emails sent by the JBoss ON server.
6.3.2. Configuring Communication Settings
servlet
(HTTP) and sslservlet
(HTTPS) transports route traffic through the Tomcat server embedded in the JBoss ON server.
Important
servlet
or sslservlet
, the agent communication is over the Tomcat connector. This means rhq.communications.connector.bind-port
is ignored and the Tomcat connector port is used to send messages from agent to server. By default, the Tomcat connector port is 7080 (servlet) or 7443 (sslservlet).
Note
# General Properties rhq.server.startup.web.http.port=7080 rhq.server.startup.web.https.port=7443
rhq.server.tomcat.security.client-auth-mode=want rhq.server.tomcat.security.secure-socket-protocol=TLS rhq.server.tomcat.security.algorithm=SunX509 rhq.server.tomcat.security.keystore.alias=RHQ rhq.server.tomcat.security.keystore.file=conf/rhq.keystore rhq.server.tomcat.security.keystore.password=RHQManagement rhq.server.tomcat.security.keystore.type=JKS rhq.server.tomcat.security.truststore.file=conf/rhq.truststore rhq.server.tomcat.security.truststore.password=RHQManagement rhq.server.tomcat.security.truststore.type=JKS
protocol://server:port/transportParm1=value1&transportParam2=value2
Important
rhq.communications.connector.transport=servlet rhq.communications.connector.bind-address=192.168.1.2 rhq.communications.connector.bind-port=16163 rhq.communications.connector.transport-params=/jboss-remoting-servlet-invoker/ServerInvokerServlet
servlet://192.168.1.2:16163/jboss-remoting-servlet-invoker/ServerInvokerServlet
RHQ Server IP Address=192.168.1.2 RHQ Server Port=16163 RHQ Server Transport Protocol=servlet RHQ Server Transport Parameters=/jboss-remoting-servlet-invoker/ServerInvokerServlet
Example 6. Basic Server-Agent Transport Example
192.168.0.10
will connect to agents over the standard agent port of 16163. The server configuration has the following configuration to define the server address (rhq.communications.connector.bind-address
), the agent communications port (rhq.communications.connector.bind-port
), and the connection protocol (rhq.communications.connector.transport
):
rhq.communications.connector.transport=servlet rhq.communications.connector.bind-address=192.168.0.10 rhq.communications.connector.bind-port=16163 rhq.communications.connector.transport-params=enableTcpNoDelay=true&backlog=200
servlet://192.169.0.10:16163/enableTcpNoDelay=true&backlog=200
RHQ Server IP Address=192.168.0.10 RHQ Server Port=16163 RHQ Server Transport Protocol=socket RHQ Server Transport Parameters=enableTcpNoDelay=true&backlog=200
enableTcpNoDelay
(client) and backlog
(server). When the JBoss ON server is receiving messages — when it function as a communications server — it uses the backlog
parameter and ignore enableTcpNoDelay
because enableTcpNoDelay
is only for outgoing (client) messages.
Table 10. rhq-server.properties Parameters for Server Connections
Parameter | Description |
---|---|
General Connection Parameters | |
jboss.bind.address[a][b] | Gives the IP address for the JBoss ON GUI console, among other services, to bind to. This is the host in the JBoss ON GUI URL, such as server.example.com in http://server.example.com:7080 . |
rhq.server.startup.web.http.port[a][b] | Gives the port that the JBoss ON GUI listens to for unsecured HTTP requests. This is the port number in the JBoss ON GUI URL, such as the 7080 in http://localhost:7080. This is also the unsecured port used as the endpoint in high availability. |
rhq.server.startup.web.https.port[a][b] | Gives the port that the JBoss ON GUI listens to for secured HTTPS requests. This is the port number in the JBoss ON GUI URL, such as the 7443 in https://localhost:7443. This is also the secure port used as the endpoint in high availability. |
rhq.server.startup.keystore.filename[b] | The JBoss ON GUI can accept browser requests over HTTPS. If you want to authenticate the JBoss ON GUI to remote browsers, you need to put an SSL certificate in a keystore file. This setting points to the location of the keystore file. Note that this file name must be a relative file path relative to the <JBoss ON server Install Dir>/jbossas/server/default/conf directory. The JBoss ON server ships with a self-signed certificate in a default keystore file. |
rhq.server.startup.keystore.password[b] | The password of the keystore file. This is so the JBoss ON GUI can access the keystore file in order to be able to serve the certificate to browser clients. |
rhq.server.startup.keystore.sslprotocol[b] | The protocol that browser clients should use to communicate with the JBoss ON GUI. |
rhq.server.maintenance-mode-at-start | Sets whether to start the server in maintenance mode (true) or whether to start the server in whatever mode it was in when it shut down (false). The default is false. |
| Ports used by internal services. |
SSL Connection Parameters | |
| The secure protocol that agents must use when communicating with this JBoss ON server. |
| The keystore file that contains a certificate that authenticates the JBoss ON server to the agents. |
| |
| The file format of the keystore. |
| The password that is used to gain access to the keystore file. |
| The password that is used to gain access to the key inside the keystore. |
| The alias that identifies the JBoss ON server's key within its keystore. |
| The truststore file that contains certificates that this JBoss ON server trusts. If you need the JBoss ON server to authenticate JBoss ON agents, you must set this; otherwise it is not needed. This truststore contains certificates for all JBoss ON agents that need to communicate with this JBoss ON server. Refer to the Incoming Client Authentication Mode. |
| |
| The file format of the truststore file. |
| The password that is used to gain access to the truststore file. |
| Indicates if the JBoss ON server must authenticate the JBoss ON agents that are sending it messages. If the server is using secure connections, but does not have trusted certificates for all of the JBoss ON agents in a truststore, set this to none. The valid values are none, want, or need. |
Transport Connection Parameters | |
rhq.communications.connector.transport | Defines how the JBoss ON agents need to transport messages to the JBoss ON server. The allowed values are either servlet or sslservlet. The agent requests go through the JBoss ON server web application layer (i.e. the secure Tomcat Connector). With sslservlet, not only do agent requests route through the web application layer, but they are also secured through the secure Tomcat Connector. The keystore used for incoming agent message authentication is the same as that configured in rhq.communications.connector.security.keystore.file .
Note
This transport setting does not restrict agents from only going over that particular connection method. By default, the JBoss ON server always deploys the communications connector that allows for both servlet and sslservlet traffic. This setting tells the agent to decide what transport is used when it sends messages to the server. If the server has its transport set to servlet, but the agent is configured to talk to the server via sslservlet, the messages the agent sends will be via sslservlet.
|
rhq.communications.connector.bind-address | This is the address that is placed in the server's JBoss/Remoting locator URL. This defines the endpoint that the JBoss ON server will bind its connector to. It also represents the public endpoint address that all agents can use to connect to the server. |
rhq.communications.connector.bind-port | Defines the endpoint that the JBoss ON server binds to, as well as the public address that all agents can use to connect to the server. This is hidden from view in the installer, although it still appears in the rhq-server.properties file. This value can be blank; the server sets this to either the HTTP or HTTPS port, depending on the transport configured for the server. |
rhq.communications.connector.transport-params | Defines additional transport parameters the JBoss ON server will set on its connector that will accept incoming messages from the JBoss ON agents. All of the possible transport parameters are listed in Table 11, “Transport Parameters”. |
rhq.communications.multicast-detector.enabled | If true, the JBoss ON server will attempt to auto-detect JBoss ON agents coming online and going offline using multicast detection. Your network must support multicast traffic for this to work. |
rhq.communications.multicast-detector.bind-address | The address that the multicast detector directly binds to. This is not used, or needed, if you have not enabled multicast detection. |
rhq.communications.multicast-detector.multicast-address | The address that the multicast detector will broadcast messages to. This is not used, or needed, if you have not enabled multicast detection. |
rhq.communications.multicast-detector.port | The port that the multicast detector will broadcast messages to. This is not used, or needed, if you have not enabled multicast detection. |
[a]
These settings configure specific IP addresses and ports for the JBoss ON server instance. If there are firewall issues the require different settings, then these parameters can be changed.
[b]
The JBoss ON server has to be restarted for any changes to this value to take effect.
|
Table 11. Transport Parameters
Transport Parameter | Description | For Incoming Messages or for Outgoing Messages |
---|---|---|
serverBindAddress | The address on which the server socket binds to listen for requests. The default is an empty value which indicates the server socket should be bound to the host provided by the InvokerLocator URL (the host). | Incoming |
serverBindPort | The port to listen for requests on. | Incoming |
timeout | The socket timeout value. The default on the server side is 60000 (one minute). If the timeout parameter is set, its value will also be passed to the client-side (see below). | Incoming |
backlog | The preferred number of unaccepted incoming connections allowed at a given time. The actual number may be greater than the specified backlog. When the queue is full, further connection requests are rejected. Must be a positive value greater than 0. If the value passed if equal or less than 0, then the default value will be assumed. The default value is 200. | Incoming |
numAcceptThreads | The number of threads that exist for accepting client connections. The default is 1. | Incoming |
maxPoolSize | The number of server threads for processing client requests. The default is 300. | Incoming |
socket.check_connection | Indicates if the invoker should try to check the connection before re-using it by sending a single byte ping from the client to the server and then back from the server. This configuration needs to be set on both the client and server to work. The default value is false. | Incoming |
clientConnectAddress | The IP address or hostname the client will use to connect to the server-side socket. This would be needed in the case that the client will be going through a router that forwards requests made externally to a different IP address or hostname internally. If no clientConnectAddress or serverBindAddress is specified, the local host's address is used. | Outgoing |
clientConnectPort | The port the client will use to connect to the server-side socket. This would be needed in the case that the client will be going through a router that forwards requests made externally to a different port internally. | Outgoing |
timeout | The socket timeout value. The default on the client side is 1800000 (or 30 minutes). | Outgoing |
enableTcpNoDelay | Indicates if the client socket should have TCP_NODELAY turned on or off. TCP_NODELAY is for a specific purpose; to disable the Nagle buffering algorithm. It should only be set for applications that send frequent small bursts of information without getting an immediate response. The default is false. | Outgoing |
clientMaxPoolSize | The client-side maximum number of active socket connections. This basically equates to the maximum number of concurrent client calls that can be made from the socket client invoker. The default is 50. | Outgoing |
numberOfRetries | The number of retries to get a socket from the pool. This basically equates to the number of seconds the client will wait to get a client socket connection from the pool before timing out. If the max retries is reached, a CannotConnectException will be thrown. The default is 30. | Outgoing |
numberOfCallRetries | The number of retries for making the invocation. This is unrelated to numberOfRetries in that when this comes into play is after it has already received a client socket connection from the pool. However, it is possible that the socket connection timed out while waiting within the pool. Since a connection check is not done by default, the connection is thrown away and an attempt to get a new one will be made. This will happen for however many numberOfCallRetries is (which defaults to 3). However, when (numberOfCallsRetries - 2) is reached, the entire connection pool is flushed under the assumption that all connections in the pool have timed out and are invalid and will start over by creating a new connection. If this still fails, a MarshalException is thrown. | Outgoing |
socket.check_connection | Indicates if the invoker should try to check the connection before re-using it by sending a single byte ping from the client to the server and then back from the server. This configuration needs to be set on both client and server to work. This if false by default. | Outgoing |
6.3.3. Setting Concurrency Limits
- A global limit on the total number of incoming messages to the server (
rhq.communications.global-concurrency-limit
).This is the total number of allowed agent connections. There are other concurrency limits for specific message types which can help tune performance for content downloads, inventory synchronization, and other resource-intensive or recurring agent operations. Those concurrency limits apply only to those specific message types, and those limits are evaluated independently of each other. The global concurrency limit is the total cap for all agent connections. This is the effective concurrency limit, even if the sum of the other concurrency limits is higher. - A limit on the total number of concurrent web connections allowed (
rhq.server.startup.web.max-connections
).This counts any client connection which connects to the JBoss ON server over an HTTP or HTTPS connection. This includes web GUI connections, of course, but it also includes all agent connections which use the (default)servlet
orssslservlet
transports.The limit on web connections is the same for both non-secured HTTP requests and HTTPS requests, but the limit is additive so HTTP and HTTPS connections count against different pools. The total maximum connections allowed is actually twice whatever therhq.server.startup.web.max-connections
value is. For example, if the setting is 300, then 300 HTTP requests are allowed and 300 HTTPS requests are allowed, for total of 600 concurrent web connections. - Limits on the number of downloads from agents (
rhq.server.agent-downloads-limit
) and from other clients (rhq.server.client-downloads-limit
).
Example 7. Concurrency Limits
rhq.server.startup.web.max-connections=200 rhq.server.agent-downloads-limit=45 rhq.server.client-downloads-limit=5 rhq.communications.global-concurrency-limit=30
Table 12. rhq-server.properties Parameters for Concurrency Limits
Parameter | Description |
---|---|
rhq.server.startup.web.max-connections | Sets a limit on the number of web connections that can be concurrently created, including both connections to the GUI and connections by agents.
Note
If agent requests are routed over web connections, make sure that the rhq.communications.global-concurrency-limit value is slightly lower than the web connections limit. Otherwise, GUI users could be blocked from accessing the JBoss ON UI whenever there is a high agent load.
|
rhq.communications.global-concurrency-limit |
Sets the total number of agent messages that come into the server. This only affected incoming agent messages, not GUI requests. If this global concurrency limit is set to 300, no more than 300 total agent messages can be processed at any one time, regardless of what kinds of messages are coming in.
Even if the sum of the other concurrency limits are higher than this global limit, they are capped at this global limit since there can never be more messages processed than the global limit.
This value should be slightly lower than the number of allowed web connections so that web connections to the GUI are not blocked when there is a high agent load.
|
rhq.server.concurrency-limit.inventory-report | Inventory reports are sent from the agent when the agent starts up, and periodically thereafter. Inventory reports can be large, depending on the number of resources on the agent machine. |
rhq.server.concurrency-limit.availability-report | Availability reports are regularly sent from the agent, typically every 60 seconds. Availability reports are usually very small, but occur in large numbers due to the high frequency of their transmission. |
rhq.server.concurrency-limit.inventory-sync | Inventory synchronizations occur when the agent needs to synchronize its inventory with that of the server. Agents typically synchronize at startup. Traffic that flows as part of inventory synchronizations is usually large, depending upon the number of resources managed by the agent. |
rhq.server.concurrency-limit.content-report | Content reports are similar to inventory reports except they contain information about discovered content (i.e., installed packages of software). These reports can be large depending on the number of installed software the agent has discovered and is managing. |
rhq.server.concurrency-limit.content-download | Content downloads occur when a resource on an agent needs to ask for the content of a package version, usually for the purpose of installing the package. |
rhq.server.concurrency-limit.measurement-report | Measurement reports are periodically sent to the server whenever the agent completes measurement collections. The number and size of measurement reports can vary, depending on the number and frequency of measurements scheduled to be collected. The greater the number of schedule measurements the agent needs to collect, the more frequently measurement reports are sent, and the larger the reports will be. |
rhq.server.concurrency-limit.measurement-schedule-request | Similar to inventory synchronization, measurement schedule requests are sent to the agent asking the server for an up-to-date set of measurement schedules that have to be collected. |
6.3.4. Configuring the SMTP Server for Email Notifications
rhq-server.properties
file. The default configuration points to the local JBoss ON server hosts.
# Email rhq.server.email.smtp-host=localhost rhq.server.email.smtp-port=25 rhq.server.email.from-address=rhqadmin@localhost
Note
http://server/portal/admin/test/email.jsp
.
Table 13. rhq-server.properties Parameters for SMTP
Parameter | Description |
---|---|
rhq.server.email.smtp-host | Sets the hostname of the SMTP server used by the JBoss ON server. |
rhq.server.email.smtp-port | Sets the port of the SMTP server used by the JBoss ON server. |
rhq.server.email.from-address | Sets the address to use for the From header of all emails sent by the JBoss ON server. |
6.3.5. Installing a Server Silently
rhq-server.properties
file tell the installation process to load the server configuration from the file rather than the from the web-based installer.
# Auto-Install Pre-Configuration Settings rhq.autoinstall.enabled=true rhq.autoinstall.database=auto rhq.autoinstall.public-endpoint-address=
Important
Table 14. rhq-server.properties Parameters for Silent Installation
Parameter | Description |
---|---|
rhq.autoinstall.enabled | Tells the installation process whether to load the configuration from the rhq-server.properties file (true ) or from the web-based installer (false ). |
rhq.autoinstall.database | Tells the install process how to load or add database schema. There are three options:
|
rhq.autoinstall.public-endpoint-address | Sets the IP address or hostname to use for the server. If no value is given, then the server detects and sets its own value when it starts. |
6.4. Synchronizing Server Configuration
- System settings, which include how long alerts, events, and monitoring metrics are stored; the baseline calculation schedule; and the LDAP server configuration.
- Metric collection settings for each resource types.
Note
6.4.1. Exporting a Server's Configuration
- Log into the JBoss ON CLI.
[root@server bin]# installDir/bin/rhq-cli.sh -u rhqadmin -p rhqadmin
- Export the data to a database object:
rhqadmin@localhost:7080$ var ex = SynchronizationManager.exportAllSubsystems();
- Convert that object into an export file. The file extension should be
.xml.gz
because the export format is a GZIP'ed XML file.rhqadmin@localhost:7080$ saveBytesToFile(ex.exportFile, 'export.xml.gz');
Note
6.4.2. Importing a Server's Configuration
- The configuration data are imported directly into the server, using all of the default settings.
- The XML file can be edited so that the configuration values are adapted to the target JBoss ON servers.
- The synchronizer behavior is changed, which changes what data elements are imported.
6.4.2.1. Editing the XML Import File
<entities>
elements.
<entity>
elements, with the metric itself identified by its name, resource type, and plug-in as arguments for the element. The entity ID identifies the template in the JBoss ON database, but is ignored during import because the IDs do not need to match between servers.
<entities id="org.rhq.enterprise.server.sync.MetricTemplateSynchronizer"> <entity> <data> <metricTemplate enabled="false" defaultInterval="300000" perMinute="false" metricName="trap_count" resourceTypePlugin="snmptrapd" resourceTypeName="SnmpTrapd" referencedEntityId="10001"> </metricTemplate> </data> </entity> .....
<entity>
element, and each configuration parameter is given as a key on the entry. Not all of these keys are imported into the target server; the keys which are imported depend on the synchronizer configuration.
<entities id="org.rhq.enterprise.server.sync.SystemSettingsSynchronizer"> <entity> <data> <systemSettings referencedEntityId="0"> <entry key="CAM_BASE_URL">http://10.16.65.121:7080/</entry> <entry key="CAM_DATA_PURGE_6H">2678400000</entry> <entry key="CAM_LDAP_BIND_DN"></entry> ..... </systemSettings> </data> </entity> </entities>
6.4.2.2. Changing the Synchronizer Configuration
Note
SynchronizationManager.getImportConfigurationDefinition()
. For example:
rhqadmin@localhost:7080$ var configDef = SynchronizationManager.getImportConfigurationDefinition('org.rhq.enterprise.server.sync.SystemSettingsSynchronizer')
rhqadmin@localhost:7080$ var configDefs = SynchronizationManager.importConfigurationDefinitionOfAllSynchronizers rhqadmin@localhost:7080$ configDef = configDefs.get(0) rhqadmin@localhost:7080$ pretty.print(configDef.configurationDefinition.defaultTemplate.configuration)
6.4.2.2.1. Changing the Synchronizer Settings in the XML File
<default-configuration>
, and the configuration settings are listed as properties within that element.
<ci:simple-property>
element, and the list of settings to import is given in the value=
flag on the <ci:simple-property>
element.
<default-configuration> <ci:simple-property value="AGENT_MAX_QUIET_TIME_ALLOWED, ENABLE_AGENT_AUTO_UPDATE, ENABLE_DEBUG_MODE, ENABLE_EXPERIMENTAL_FEATURES, CAM_DATA_PURGE_1H, CAM_DATA_PURGE_6H, CAM_DATA_PURGE_1D, CAM_DATA_MAINTENANCE, DATA_REINDEX_NIGHTLY, RT_DATA_PURGE, ALERT_PURGE, EVENT_PURGE, TRAIT_PURGE, AVAILABILITY_PURGE, CAM_BASELINE_FREQUENCY, CAM_BASELINE_DATASET" type="string" name="propertiesToImport"> <c:description>The names of the properties that should be imported. Note that these are the INTERNAL names as used in the RHQ database</c:description> </ci:simple-property> </default-configuration>
Note
- A simple list, which has a
<ci:list-property>
list members defined by a property (<ci:simple-property>
) and a list of values<default-configuration> <ci:list-property name="my-list"> <c:simple-property name="element" type="string"/> <ci:values> <ci:simple-value value="a"/> <ci:simple-value value="b"/> <ci:simple-value value="c"/> </ci:values> </ci:list-property> </default-configuration>
- A map of values, which is very similar to a simple list in that it uses a list of properties (
<ci:simple-property>
) and a corresponding list of values (<ci:simple-value>
), except that each value corresponds to a single, specified property based on the name<default-configuration> <ci:map-property name="my-map"> <c:simple-property name="prop1" type="integer"/> <c:simple-property name="prop2" type="string"/> <c:simple-property name="prop3" type"boolean"/> <ci:values> <ci:simple-value property-name="prop1" value="1"/> <ci:simple-value property-name="prop2" value="abc"/> <ci:simple-value property-name="prop3" value="true"/> </ci:values> </ci:map-property> </default-configuration>
- A table, which is a list of maps. Each set of maps specifies one table in the row.
<default-configuration> <ci:list-property name="table"> <c:map-property name="row"> <c:simple-property name="column1" type="integer"/> <c:simple-property name="column2" type="boolean"/> <c:simple-property name="column3" type="string"/> </c:map-property> <ci:values> <ci:map-value> <ci:simple-value property-name="column1" value="1"/> <ci:simple-value property-name="column2" value="true"/> <ci:simple-value property-name="column3" value="a"/> </ci:map-value> <ci:map-value> <ci:simple-value property-name="column1" value="2"/> <ci:simple-value property-name="column2" value="true"/> <ci:simple-value property-name="column3" value="b"/> </ci:map-value> <ci:map-value> <ci:simple-value property-name="column1" value="3"/> <ci:simple-value property-name="column2" value="false"/> <ci:simple-value property-name="column3" value="c"/> </ci:map-value> </ci:values> </ci:list-property> </default-configuration>
<default-configuration> <ci:simple-property value="false" type="boolean" name="updateAllSchedules" /> <ci:list-property name="metricUpdateOverrides"> <c:map-property summary="false" required="true" readOnly="false" name="metricUpdateOverride"> <c:simple-property type="string" summary="false" required="true" readOnly="false" name="metricName" /> <c:simple-property type="string" summary="false" required="true" readOnly="false" name="resourceTypeName" /> <c:simple-property type="string" summary="false" required="true" readOnly="false" name="resourceTypePlugin" /> <c:simple-property type="boolean" summary="false" required="true" readOnly="false" name="updateSchedules" /> </c:map-property> <ci:values> <ci:map-value> <ci:simple-value name="metricName" value="MCBean|ServerInfo|*|freeMemory"/> <ci:simple-value name="resourceTypeName" value="JBoss AS Server"/> <ci:simple-value name="resourceTypePlugin" value="JBossAS5"/> <ci:simple-value name="updateSchedules" value="true"/> </ci:map-value> </ci:values> </ci:list-property> </default-configuration>
<ci:simple-property>
element to name="updateAllSchedules"
.
metricUpdateOverride
and set the updateSchedules
property value to true
.
6.4.2.2.2. Changing the Synchronizer Settings Programmatically
setValue
configuration object to add or remove keys from the list. For the settings synchronizer, this lists the key name to import:
configurationObject.getSimple('propertiesToImport').setValue(defaultSettingsToImport + ', keyName')
var update = new PropertyMap('metricUpdateOverrides') update.put(new PropertySimple('propertyName', 'resourcePluginName'))
- Get the default definition.
rhqadmin@localhost:7080$ var systemSettingsImportConfigurationDefinition = SynchronizationManager.getImportConfigurationDefinition('org.rhq.enterprise.server.sync.SystemSettingsSynchronizer')
- Create a new configuration instance.
rhqadmin@localhost:7080$ var configurationObject = systemSettingsImportConfigurationDefinition.configurationDefinition.defaultTemplate.createConfiguration() rhqadmin@localhost:7080$ var systemSettingsImportConfiguration = new ImportConfiguration(systemSettingsImportConfigurationDefinition.synchronizerClassName, configurationObject)
- Change the settings in the new instance.For example, for the server settings synchronizer:
rhqadmin@localhost:7080$ var defaultSettingsToImport = configurationObject.getSimple('propertiesToImport').stringValue rhqadmin@localhost:7080$ configurationObject.getSimple('propertiesToImport').setValue(defaultSettingsToImport + ', CAM_BASE_URL')
For the metrics template synchronizer:configurationObject.getSimple('updateAllSchedules').setBooleanValue(true) var updateList = new PropertyList('metricUpdateOverrides') var update = new PropertyMap('metricUpdateOverride') update.put(new PropertySimple('metricName', 'MCBean|ServerInfo|*|freeMemory')) update.put(new PropertySimple('resourceTypeName', 'JBossAS Server')) update.put(new PropertySimple('resourceTypePlugin', 'JBossAS5')) update.put(new PropertySimple('updateSchedules', 'true')) updateList.add(update) configurationObject.put(updateList)
6.4.2.3. Importing the Configuration
- Log into the JBoss ON CLI.
[root@server bin]# installDir/bin/rhq-cli.sh -u rhqadmin -p rhqadmin
- Import the XML file containing the configuration:
rhqadmin@localhost:7080$ var data = getFileBytes('export.xml.gz'); rhqadmin@localhost:7080$ SynchronizationManager.importAllSubsystems(ex, null);
The null parameter means that the import process uses the default settings in the XML file or, if the defaults are missing from the XML, that it uses the settings defined on the target server. If alternate settings were constructed in Section 6.4.2.2, “Changing the Synchronizer Configuration”, then they can be specified programmatically instead. For example:rhqadmin@localhost:7080$ var configsToImport = new java.util.ArrayList() rhqadmin@localhost:7080$ configsToImport.add(systemSettingsImportConfiguration); rhqadmin@localhost:7080$ configsToImport.add(metricTemplatesImportConfiguration); rhqadmin@localhost:7080$ SynchronizationManager.importAllSubsystems(ex,
configToImport
);
7. Configuring Agents
rhq-agent.sh
script.
7.1. Registering and Re-registering the Agent
7.1.1. About the Security Token and Agent Registration
Figure 8. Agent Registration
Figure 9. Different Agent Connection Attempts
- An agent cannot register with an existing agent name without the corresponding security token.To register an agent with an existing agent name, you must first install the corresponding security token, as described in Section 7.1.2, “Re-installing a Lost Security Token”.
- An agent cannot register with an existing IP address/port combination without having the corresponding security token and using the original agent name.This essentially means that you cannot rename an agent. If an agent is registered with an existing IP address/port combination, then both the original security token and the original name must also be used. This re-establishes the original identity of the agent and prevents one agent from effectively stealing the identity of another agent.
- An agent can register with an existing name and a new IP address/port combination if it has the security token which corresponds to that agent name.While the agent name cannot be changed during re-registration, the agent IP address, the agent port, or both can be changed. This is a common and useful scenario in cloud, virtual, or DHCP environments where an existing agent needs to re-register with a new IP address or port.
Note
--cleanconfig
. This allows the agent to re-register easily.
7.1.2. Re-installing a Lost Security Token
- Stop the agent.
- Log into the web UI as a user with manage security permissions.
- Click the Administration tab and select the Agents link under the Topology section on the left.
- Select the agent from the list, and click its name to open its details page.
- Copy the security token.
- Restart the agent, and use the
-D
option to set therhq.agent.security-token
property to the security token.agentRoot/rhq-agent/bin/rhq-agent.sh -Drhq.agent.security-token=abcd1234
7.1.3. Reinstalling the Agent with a New Security Token
- The agent's persisted Java configuration should be purged.
- The agent's inventory should be purged, along with any resource history and configuration.
- The agent must be removed from the JBoss ON inventory. This can be done by removing the agent from the JBoss ON topology (Section 7.2, “Removing an Agent”) or by deleting the platform entry.
- The agent's original identifying information (name, IP address, and port) can be changed.
- Make sure that the original agent instance is properly removed.
- Stop the agent process.
- Remove the platform entry from the JBoss ON server inventory.
- Restart the agent with the
--fullcleanconfig
option. This registers the agent with a new security token and fresh configuration settings.agentRoot/rhq-agent/bin/rhq-agent.sh --fullcleanconfig
Note
7.1.4. Cleaning the Agent Configuration, with the Original Security Token
- The agent's persisted Java configuration is purged.
- The agent's inventory, along with any resource history and configuration, is saved.
- The agent (via the platform entry) remains in the JON inventory.
- The agent's name must remain the same (though the IP address or port number can be changed).
--cleanconfig
option. This registers the agent with fresh configuration settings (from the conf/agent-configuration.xml
file) and reuses its previous security token.
agentRoot/rhq-agent/bin/rhq-agent.sh --fullcleanconfig
Note
7.2. Removing an Agent
7.2.1. Removing an Agent from a Managed Platform
- Stop the agent service.
- Delete the agent from JBoss ON topology.
- In the JBoss ON UI, click the Administration tab in the top menu.
- In the Topology section in the left menu, select the Agents item.
- Select the row for the agent to delete, in the list of installed agents.
- Click the Delete button at the bottom of the page.
- Confirm that the agent should be deleted.
- If the agent was configured as a system service, remove the associated files. On Windows, this can be done with the
rhq-agent.bat remove
command. On Linux, this means removing theinit.d/
file and updatingchkconfig
. - Remove the agent installation directory.
7.2.2. Removing an Agent on a JBoss ON Server Machine
- Stop the agent service.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl.sh stop --agent
- Remove the agent.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl remove --agent
- Delete the agent from JBoss ON topology.
- In the JBoss ON UI, click the Administration tab in the top menu.
- In the Topology section in the left menu, select the Agents item.
- Select the row for the agent to delete, in the list of installed agents.
- Click the Delete button at the bottom of the page.
- Confirm that the agent should be deleted.
7.3. Working with the Agent Command Prompt
7.3.1. Opening the Agent Command Prompt
$ rhq-agent.sh
7.3.2. Agent Start Options
rhq-agent.sh
start script; these mainly relate to passing persistent configuration options to the server by loading external preferences through input files or passed parameters. These options are listed in Table 15, “Options for the rhq-agent.sh Script”.
Table 15. Options for the rhq-agent.sh Script
Short Argument | Long Argument | Description |
---|---|---|
-a | --advanced | Runs the agent script in setup mode, rather than basic start mode. |
-c | --config=filename | Specifies an agent configuration preferences file on filesystem or classpath. |
-d | --daemon | Runs the agent in daemon mode, which means it will not read additional commands from stdin. |
-Dname[=value] | Overrides an agent configuration preference and sets a system property. | |
-e | --console=type | Specifies the implementation to use when reading console input. The three available values are jline, sigar, and java. |
-h | --help | Opens the help message. |
-i | --input=filename | Specifies a script file to use for input. |
-l | --cleanconfig | Clears out any existing configuration and data files so the agent starts with blank configuration, with the exception of the agent security token, which is preserved. |
-L | --fullcleanconfig | Clears out any existing configuration and data files so the agent starts with a totally clean slate, including purging the security token. |
-n | --nostart | Runs the agent script without starting the agent process. |
-o | --output=filename | Specifies a file to write all output from the script, excluding log messages (which are always written to the agent logs). |
-p | --pref=preferences_name | Specifies the Java preference node to use for the agent configuration. |
-s | --setup | Forces the agent to ask setup questions. |
-t | --nonative | Forces the agent to disable the native system. |
-u | --purgedata | Purges persistent inventory and other data files. |
-- | Stops the agent from processing options. |
7.3.3. Agent Prompt Commands
Table 16. Agent Prompt Commands
Prompt Command | Description |
---|---|
avail
|
Provides availability of inventoried resources.
|
config
|
Manages the agent configuration.
|
debug
|
Provides features to help debug the agent.
|
discovery
|
Asks a plug-in to run a server scan discovery.
|
download
|
Downloads a file from the JBoss ON server.
|
dumpspool
|
Shows the entries found in the command spool file.
|
exit
|
Shuts down the agent's communications services and kills the agent.
|
failover
|
Shows or updates the high availability server failover list.
|
gc
|
Helps free up memory by invoking the garbage collector.
|
getconfig
|
Displays one, several or all agent configuration preferences.
|
help
|
Shows help for a given command.
|
identify
|
Asks to identify a remote server.
|
inventory
|
Provides information about the current inventory of resources.
|
log
|
Configures some settings for the log messages.
|
metrics
|
Shows the agent metrics.
|
native
|
Accesses native system information.
|
pc
|
Starts and stops the plug-in container and all deployed plug-ins.
|
ping
|
Pings the JBoss ON server.
|
piql
|
Executes a PIQL query to search for running processes.
|
plugins
|
Updates the agent plug-ins with the latest versions from the server.
|
quit
|
Exits the agent prompt (without stopping the agent).
|
register
|
Registers this agent with the JBoss ON server.
|
schedules
|
Retrieves measurement schedule information for the specified resource.
|
sender
|
Controls the command sender to start or stop sending commands.
|
setconfig
|
Sets an agent configuration preference.
|
setup
|
Sets up the agent configuration by asking a series of questions.
|
shutdown
|
Shuts down all communications services without killing the agent.
|
sleep
|
Puts the agent prompt to sleep for a given amount of seconds.
|
start
|
Starts the agent comm services so it can accept remote requests.
|
timer
|
Times how long it takes to execute another prompt command.
|
update
|
Provides agent update functionality.
|
version
|
Shows the agent version information.
|
7.4. Interactions with System Users for Agents and Resources
- JBoss EAP servers
- PostgreSQL databases
- Tomcat servers
- Apache servers
- Generic JVMs
Table 17. Cheat Sheet for Agent and Resource Users
Resource | User Information |
---|---|
PostgreSQL | No effect for monitoring and discovery.
The agent user must have read/write permissions to the PostgreSQL configuration file for configuration viewing and editing.
|
Apache | No effect for monitoring and discovery.
The agent user must have read/write permissions to the Apache configuration file for configuration viewing and editing.
|
Tomcat | Must use the same user or can't be discovered |
JMX server or JVM | Different users are fine when using JMX remoting; cannot be discovered with different users and the attach API |
JBoss AS/EAP | Different users are all right, but requires read permissions on run.jar and execute and search permission on all ancestor directories for run.jar |
7.4.1. The Agent User
7.4.2. Agent Users and Discovery
- For JBoss EAP resources, the agent must have read permissions to the
run.jar
file, plus execute and search permissions for every directory in the path to therun.jar
file. - Tomcat servers can only be discovered if the JBoss ON agent and the Tomcat server are running as the same user. Even if the agent is running as root, the Tomcat server cannot be discovered if it is running as a different user than the agent.
- If a JVM or JMX server is running with JMX remoting, then it can be discovered if the agent is running as a different user. However, if it is running with using the attach API, it has to be running as the same user as the agent for the resource to be discovered.
7.4.3. Users and Management Tasks
- Discovery
- Deploying applications
- Executing scripts
- Running start, stop, and restart operations
- Creating child resources through the JBoss ON UI
- Viewing and editing resource configuration
sudo
.
7.4.4. Using sudo with JBoss ON Operations
sudo
is for long-running operations, such as starting a service or a process, or for scripts which are owned by a resource user. The user which executes the script should be the same as the resource user because that user already has the proper authorization and permissions.
sudo
rights to the given command.
- There can be no required interaction from the user, including no password prompts.
- It should be possible for the agent to pass variables to the script.
sudo
for resource scripts:
- Grant the JBoss ON agent user
sudo
rights to the specific script or command. For example, to run a script as thejbossadmin
user:[root@server ~]# visudo jbosson-agent hostname=(jbossadmin) NOPASSWD: /opt/jboss-eap/jboss-as/bin/*myScript*.sh
Using theNOPASSWD
option runs the command without prompting for a password.Important
JBoss ON passes command-line arguments with the start script when it starts an EAP instance. This can be done either by including the full command-line script (including arguments) in thesudoers
entry or by using thesudo -u
user command in a wrapper script or a script prefix.The second option has a simplersudoers
entry - Create or edit a wrapper script to use. Instead of invoking the resource's script directly, invoke the wrapper script which uses
sudo
to run the script.Note
For the EAP start script, it is possible to set a script prefix in the connection settings, instead of creating a separate wrapper script:/usr/bin/sudo -u jbosson-agent
For example, for a start script wrapper,start-myScript.sh
:#!/bin/sh # start-myScript.sh # Helper script to execute start-myConfig.sh as the user jbosson-agent # sudo -u jbosson-agent /opt/jboss-eap/jboss-as/bin/start-myConfig.sh
- Create the start script, with any arguments or settings to pass with the
run.sh
script. For example, forstart-myConfig.sh
:nohup ./run.sh -c MyConfig -b jonagent-host 2>&1> jboss-MyConfig.out &
7.5. Running the Agent as a Non-Root User
Note
setfacl
command will vary depending on the resource type.
- Log into the system as root.
- Make sure that the acl package is installed on the system.
# rpm -q acl acl-2.2.39-6.el5
Theacl
option must be applied to the filesystem. This can be done by editing the/etc/fstab
file or usingtune2fs
. For example:# vim /etc/fstab LABEL=/ / ext3 defaults
,acl
1 1 ...Then re-mount the filesystem.# mount -o remount /
- Optionally, create a system user to use for the agent.
useradd jbosson-agent
- For PostgreSQL, the agent needs to be able to access the
postgresql.conf
file. Open the PostgreSQL directory:# cd /var/lib/pgsql
- Grant read and write access to the
postgresql.conf
file to the agent user. For example:# setfacl -m u:jbosson-agent:rw $PGDATA/postgresql.conf
- Then, grant access to the
data/
directory to the agent user. For example:# setfacl -m u:jbosson-agent:x $PGDATA
- Check that the new ACLs were added properly using the
getfacl
command:# getfacl . # file: . # owner: postgres # group: postgres user::rwx user:jbosson-agent:--x group::--- mask::--x other::---
7.6. Enabling Debug Mode for the Agent
log4j
for its logging. To troubleshoot agent performance or server-agent communication, enable debug logging for the agent, which enables the log4j
debug log.
agentRoot/rhq-agent/logs
directory.
7.6.1. Using an Environment Variable
RHQ_AGENT_DEBUG
environment variable to any value before starting the agent. When you start the agent, both the launcher scripts and the agent itself will output debug messages.
RHQ_AGENT_DEBUG
and then install the service:
rhq-agent-wrapper.bat install
7.6.2. Setting log4j Priorities
log4j
categories support priorities for logging levels. This means that different areas of the agent can be configured for different log levels.
Note
RHQ_AGENT_DEBUG
environment variable if you are setting priorities in the log4j.xml
file. The environment variable overrides this log4j.xml
configuration.
DEBUG
:
- Open the agent
log4j
file:# vim agentRoot/rhq-agent/conf/log4j.xml
- Reset the
priority
element for the category. By default, the agent configuration has logging for both incoming and outgoing server-agent communication and for the baseorg.rhq
class. Optionally, logging can be enabled for plug-in class loaders and JBoss remoting communication.<!-- ================ --> <!-- Limit categories --> <!-- ================ --> <!-- RHQ --> <category name="org.rhq"> <priority value="INFO"/> </category> <!-- RHQ outgoing command tracing - set to TRACE to trace commands sent by the agent --> <category name="org.rhq.enterprise.communications.command.client.OutgoingCommandTrace"> <priority value="NONE"/> <appender-ref ref="COMMANDTRACE"/> </category> ...
- Restart the agent to load the new configuration.
log4j
file format is described more in the Apache log4j documentation.
7.6.3. Using the Agent debug Prompt Command
debug
command in the agent command prompt (Section 7.3.1, “Opening the Agent Command Prompt”).
--enable
option enables the log4j
debug log.
> debug --enable log4j:WARN No appenders could be found for logger (org.rhq.core.pc.measurement.MeasurementCollectorRunner). log4j:WARN Please initialize the log4j system properly. Switched to log file [log4j-debug.xml]. Root log level is [DEBUG] started>
--comm
option to true.
> debug --comm=true Agent-server communications tracing has been enabled. You may set the following, additional configuration settings to collect more detailed trace data. You can set these using the setconfig prompt command. Please refer to the documentation for more information on these settings. The values you see here are the current settings: rhq.trace-command-config=true rhq.trace-command-response-results=256 rhq.trace-command-size-threshold=99999 rhq.trace-command-response-size-threshold=99999
debug
command can also be used to check all of the agent threads, to the server and to the system management handlers, using the --threaddump
option. This prints the information for each thread, whether the thread is running or any errors that the agent is encountering, per thread. For example:
> debug --threaddump "DestroyJavaVM" Id=47 RUNNABLE "RHQ Agent Prompt Input Thread" Id=46 RUNNABLE "EventManager.sender-2" Id=49 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17d7c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17d7c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.DelayQueue.take(DelayQueue.java:193) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:688) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:681) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) ...
7.7. Changing the Agent IP Address
rhq.communications.connector.bind-address
configuration preference. This is the IP address the agent binds to when it starts its server socket, meaning this is the site that the agent uses to listen for incoming messages from the server.
Note
agent-configuration.xml
file. The agent does not use this file once the initial setup is complete, so any changes to this file aren't loaded automatically by the agent.
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
with therhq.communications.connector.bind-address
configuration preference and new value.> setconfig rhq.communications.connector.bind-address=1.2.3.4
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
7.8. Managing the Agent as a Resource
Important
Table 18. Agent Child Resources
Child Resource | Description |
---|---|
The agent itself | Provides monitoring, configuration, and control functionality for the agent and its internal components. These configuration settings correspond to the preferences defined in the agent-configuration.xml file and are persisted on the agent machine as Java preferences.
Important
The operations for the agent resource normally do not affect the agent process directory. These do not provide control over the JVM settings or process or the JRE options. Controlling the JVM is done through the agent child resources, not the agent resource.
|
Agent measurement subsystem | Provides data on the measurement collection and reporting components in the agent. |
Agent JVM | Provides fine-grained monitoring and management of the JVM that is running the agent and all its plugins, which includes the classloader, threading and memory management subsystems, among others. This is a child server. |
Agent environment setup script | Configured environment variables that server set when the agent launcher script is started. |
Agent plug-in container | Provides a view into the embedded plug-in container and gives management data related directly to the plug-in container. The plug-in container runs within the agent and handles the deployment of all management plug-ins and infrastructure necessary to run those plug-ins. |
Java service wrapper launcher (Windows) | Controls the Java service wrapper. This is a third-party library that installs and runs the agent as a Windows service. There is one primary configuration file for the Java service wrapper, the read-only rhq-agent-wrapper.conf file. This defines the base set of configuration settings necessary for the agent to start and operate properly. Two additional groups of configuration settings can customize the agent's environment. The Environment group defines environment variables that are used by the main configuration in addition to the environment variables defined by the common Environment Setup Script. The Includes group defines any of the wrapper configuration settings. These groups should almost never be edited except to configure debugging or to pass new JVM options to the agent JVM. |
Agent launcher script (UNIX) | Controls the agent. If the agent is running as a background daemon process that was spawned by the launcher script, the launcher script stops or restarts it. There is no additional configuration. The launcher script is configured by the Environment Setup Script. |
7.9. Configuring the Agent Quiet Time (Timeout Period)
- Click the Administration tab in the top menu.
- In the Configuration menu table on the left, select the System Settings item.
- Scroll to the JON General Configuration Properties section in the main work area.
- Change the Agent Max Quiet Time Allowed to the desired interval for the server to wait for the agent heartbeat before marking the agent as down.
- Click the Save button. The changes are applied to all servers immediately.
7.10. Configuring Agent Update Settings
<entry key="rhq.agent.agent-update.enabled" value="true" />
Important
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
command with the new value forrhq.agent.agent-update.enabled
configuration preference.> setconfig rhq.agent.agent-update.enabled=false
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
- Click the Administration tab in the top menu.
- In the Configuration menu table on the left, select the System Settings item.
- Scroll to the JON General Configuration Properties section in the main work area.
- Set the Enable Agent Auto-Updates radio button to No. This prevents the server from sending new binaries to installed agents.
- Click the Save button. The changes are applied to all servers immediately.
7.11. Managing the Agent's Persisted Configuration
- On Windows, the backing store is located in the Windows registry.
- On Linux and Unix systems, the backing store is in the agent user's home directory, in
~/.java
.Important
The agent's configuration is determined by what user is running the agent. If the agent is run as one user and then later run as another user, the agent will have a different configuration that second time because it will use a different backing store for its configuration settings.For example, if the agent is configured by a system user namedjsmith
, its persisted configuration is in~jsmith/.java
. If the agent is then configured to run as a background service as the root user, the agent looks for its configuration in~root/.java
, and it finds different configuration settings.This means that if one user is used to configure the agent when it is installed, that same user must be used to run the agent subsequently, or the agent will apparently lose its configuration and need to be reconfigured under the new user.
agent-configuration.xml
file when the agent needs to initialize its backing store, either at its first configuration or if the agent was started with --cleanconfig
and fresh configuration settings should be loaded.
7.11.1. Viewing the Persisted Configuration
rhq-agent-env.sh
file. The agent's persisted configuration can be viewed in several different ways:
- If the agent is in the JBoss ON inventory, then its complete configuration settings are visible through the Configuration tab, with collapsible tables that display each configuration area.
- The configuration can also be returned through the
getconfig
orconfig
prompt commands for the agent. These commands can be run through a terminal, if the agent is running through a command prompt, or through the Execute Command Prompt operation in the JBoss ON UI for the agent resource.>
getconfig
rhq.agent.agent-update.enabled=true rhq.agent.client.command-preprocessors=org.rhq.enterprise.agent. SecurityTokenCommandPreprocessor: org.rhq.enterprise.agent. ExternalizableStrategyCommandPreprocessor rhq.agent.client.command-spool-file.compressed=true rhq.agent.client.command-spool-file.name=command-spool.dat rhq.agent.client.command-spool-file.params=10000000:75 rhq.agent.client.command-timeout-msecs=600000 rhq.agent.client.max-concurrent=5 rhq.agent.client.max-retries=10 rhq.agent.client.queue-size=50000 rhq.agent.client.queue-throttling=200:2000 rhq.agent.client.retry-interval-msecs=15000 rhq.agent.client.send-throttling=100:1000 rhq.agent.client.server-polling-interval-msecs=60000 rhq.agent.configuration-schema-version=5 rhq.agent.configuration-setup-flag=true rhq.agent.data-directory=data rhq.agent.disable-native-system=false rhq.agent.name=localhost.localdomain rhq.agent.plugins.directory=plugins ... - The agent configuration is persisted in Java preferences, so any tool which examines Java preferences can be used to view the persisted configuration.
Warning
7.11.2. Changing Preferences in the Persisted Configuration (Agent Preferences)
agent-configuration.xml
and overlaid with the values entered at the setup prompts at start up. After the agent is initially configured, the agent persists that configuration and never refers to the agent-configuration.xml
again, unless the configuration is purged and reloaded. Most configuration changes are made to the rhq-agent-env.sh
file, which is loaded every time the agent starts.
setconfig
command at the agent prompt.
- Open the agent prompt.
agentRoot/rhq-agent/bin/rhq-agent.sh
- Send the
setconfig
with the name of the preference to edit and its new value. The preference name is whatever the entry name is in theagent-configuration.xml
file. For example:> setconfig rhq.agent.client.max-concurrent=20
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
7.11.3. Overriding Persisted Configuration Settings
agent-configuration.xml
file for the agent can be overridden using the -D
option, the configuration parameter name, and the new value when the agent is started.
rhq.agent.wait-for-server-at-startup-msecs
), pass this argument with the start command:
agentRoot/rhq-agent/bin/rhq-agent.sh -Drhq.agent.wait-for-server-at-startup-msecs=90000
7.12. Managing the Agent JVM
rhq-agent-env.sh
file and passed to the JVM.
RHQ_AGENT_JAVA_OPTS
resets the any of the default JVM settings.RHQ_AGENT_ADDITIONAL_JAVA_OPTS
adds JVM settings without changing any of the default settings.
Note
7.13. Installing Multiple Agents with a Shared Directory or Account
default
, and the node location is agentUserHomeDir/.java/.userPrefs/rhq-agent/default
.
- Its name, which is defined as an agent configuration setting
- Its location, which is itself a Java option
- Editing the agent configuration files directly
- Setting an explicit Java option
7.13.1. Editing the Configuration Files
agent-configuration.xml
file and is loaded from there. The node location is derived from the node name setting.
- Edit the
agent-configuration.xml
file to use the new node name:[rhquser@server ~]$ vim agentRoot/rhq-agent/conf/agent-configuration.xml <node name="agent01-node">
- Then, start the agent with the
--config
option to load the edited configuration file and the--prefs
option to point to the specific node location:[rhquser@server ~]$ agentRoot/rhq-agent/bin/rhq-agent.sh --prefs=agent01-node --config=agent-configuration.xml
Important
agent-configuration.xml
file, then every time the agent restarts, the node location has to be passed to the agent using the --prefs
option.
7.13.2. Setting a Java Option
agent-configuration.xml
file only sets the node name; the node location still has to be passed every time the agent is started.
rhq-agent-env.sh
file, the Java preferences node information is set once and then persisted, so you can restart the agent as a service, without having to pass --prefs
options or edit and reload the configuration.
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.[rhquser@server ~]$ agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Use the
setconfig
command to set theRHQ_AGENT_ADDITIONAL_JAVA_OPTS
value with the preference node. For example:> setconfig RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Djava.util.prefs.userRoot=agentUserHomeDir/.java/.userPrefs/rhq-agent/agent01-node"
The preference node can be in the user preferences directory with a different name, such asagent01-node
, or it can be in an entirely different location, such as/etc/agent-preferences
, which is not a shared or filesystem-mounted location. - Restart the agent process to load the new configuration. For example, if the agent is running as a service:
[rhquser@server ~]$ service rhq-agent-wrapper.sh stop [rhquser@server ~]$ service rhq-agent-wrapper.sh start
Note
rhq-agent-env.sh
file directly, and then restart the agent.
Important
$USERHOME/.java/.userPrefs/rhq-agent/default
.
7.14. Setting Discovery Scan Intervals
- The scan interval for servers, set in the
rhq.agent.plugins.server-discovery.period-secs
. The default is 900 seconds (15 minutes). - The scan interval for services, set in the
rhq.agent.plugins.service-discovery.period-secs
. The default is 86400 seconds (24 hours). - The scan interval for low-level child services, set in the
rhq.agent.plugins.child-discovery.delay-secs
. The default is five (5) seconds.Immediate children are discovered as soon as the parent resource is discovered. However, lower-level children are discovered on a subsequent discovery scan. This sets a delay between the initial import and the next discovery scan.
agent-configuration.xml
file, so the configuration must be cleanly reloaded before the changes take effect.
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Use the
setconfig
command to reset the discovery scan intervals. The preference name is whatever the entry name is in theagent-configuration.xml
file. For example:> setconfig rhq.agent.plugins.server-discovery.period-secs=600 > setconfig rhq.agent.plugins.service-discovery.period-secs=1440 > setconfig rhq.agent.plugins.child-discovery.delay-secs=60
- Restart the agent process to load the new configuration. For example, if the agent is running as a service:
[root@server ~]# service rhq-agent-wrapper.sh stop [root@server ~]# service rhq-agent-wrapper.sh start
7.15. Viewing the Server Failover Lists for Agents
agent-configuration.xml
file, and that is the server that the agent sends its initial registration request. After registration, the agent joins the high availability cloud, and it sends its updates — monitoring information, resource changes — to any server in the cloud. At registration, the agent gets its first affinity group assignment. If its primary server is different than its registration server, then the agent switches communication over to the primary server.
> failover --list localhost.localdomain:7080/7443 server2.example.com:7080/7443 1.2.34.56:7080/7443
- Click the Administration tab in the top menu.
- In the High Availability menu, select the Agents item.
- The agent high availability page shows information about the agents, including three things that are relevant for high availability:
- The JBoss ON server that the agent is currently connected to (or the one it was most recently connected to).
- The time that the last agent availability report was sent to the server.
- The affinity group that the agent is assigned to.
- Click the name of the agent. This opens the agent's server failover list. The first server listed is the primary server for the agent; all other servers are available in the high availability cloud. The connected server is usually also the primary server, unless the primary is offline.
7.16. Setting the Agent to Detect or Poll the Server
7.16.1. Settings for Polling the JBoss ON Server
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
with therhq.agent.client.server-polling-interval-msecs
setting and a value (in milliseconds). Setting this value to zero (0) or a negative number disables server polling.> setconfig rhq.agent.client.server-polling-interval-msecs=500
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
7.16.2. Setting up Multicast Detection
- Setting to enable both server detection and multicast traffic (
rhq.agent.server-auto-detection
andrhq.communications.multicast-detector.enabled
, respectively). - A wait interval between server communications (
rhq.communications.multicast-detector.default-time-delay
); if the server is silent longer than that interval, then the server is considered offline. - Await, or heartbeat, interval for the agent's own messages (
rhq.communications.multicast-detector.heartbeat-time-delay
). This value must be shorter than the JBoss ON server's heartbeat interval (rhq.communications.multicast-detector.default-time-delay
), or it results in unnecessary messages and network traffic.
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
with the multicast settings. The time-delay values are in milliseconds.> setconfig rhq.agent.server-auto-detection=true > setconfig rhq.communications.multicast-detector.enabled=true > setconfig rhq.communications.multicast-detector.default-time-delay=75000 > setconfig rhq.communications.multicast-detector.heartbeat-time-delay=60000
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
7.17. Throttling the Agent
Table 19. Agent Parameters for Throttling Agent Operations
Parameter | Description |
---|---|
rhq.agent.client.queue-size | Sets the maximum number of commands the agent can queue up for sending to the JBoss ON server. The larger the number, the more memory the agent can use, and setting this to zero (0) means the queue size is unlimited. Setting this to 0 could allow the agent to queue up more commands than the machine has memory for, if the server goes offline for a long time. |
rhq.agent.client.max-concurrent | Sets the maximum number of messages the agent can send to the server at any one time. A larger number allows the agent to process its queue more quickly, but this can also set the agent to use more CPU cycles. |
rhq.agent.client.command-timeout-msecs | Sets the amount of time the agent waits for a reply from the JBoss ON server for an agent command before it aborts the command. A long interval can give the server the time it needs to complete some commands, but it also means that other messages are queued up waiting to be processed. |
rhq.agent.client.retry-interval-msecs | Sets the time that the agent waits before retrying a command. Only commands with the guaranteed delivery tag are retried. |
rhq.agent.client.send-throttling |
Sets a limit on the number of commands than an agent can send before it enters a quiet period, when the agent suspends sending commands. This setting only affects commands which can be throttled, which are commands that are sent to the server frequently and in large numbers, such as metric collection. Send-throttling prevents messages storms to the server.
This parameter sets both the number of commands and the quiet period, in the form commands:timeout_milliseconds. For example,
50:10000 sets a limit of 50 commands and a quiet period of 10000 milliseconds.
|
rhq.agent.client.queue-throttling |
Limits the amount of commands that can be dequeued in a given amount of time; this is the burst period. If more commands are attempted to be dequeued during the burst period than allowed, those dequeue requests are blocked until the next burst period begins.
As with send throttling, this parameter sets both the number of commands and the quiet period, in the form commands:timeout_milliseconds. For example,
50:10000 sets a limit of 50 commands and a quiet period of 10000 milliseconds.
Queue throttling prevents the agent from spinning the CPU by trying to process and send commands as fast as possible. Queue throttling is one way to reduce the amount of CPU required by the agent.
When setting the queue throttling value, be sure to set the queue size to a large enough value that the agent has room to queue up the additional commands.
|
7.18. Setting Guaranteed Delivery for Commands
- A time interval that sets how frequently the agent should try to resend a failed command (
rhq.agent.client.retry-interval-msecs
) - A filename for the spool file (
rhq.agent.client.command-spool-file.name
) - A setting that configures the spool file (
rhq.agent.client.command-spool-file.params
). This settings has the format max_file_size:purge_percentage. The file size is defined in bytes; once the file hits that file size, then a purge operation trims the file down to whatever the percentage is. So, if the file is set to be 100 KB (100000) and the purge percentage is 90, then the file is trimmed back to 90 KB after a purge operation. The purge operation first tries to compress unused space, and then begins purging commands, starting with the oldest. - An optional setting that allows the spool file to be compressed (
rhq.agent.client.command-spool-file.compressed
). Compressing the spool file can reduce its size 30-40%, but in some corner cases, it can adversely affect agent performance (such as when the agent shuts down before all of the guaranteed commands have been sent).
rhq.agent.client.command-spool-file.compressed=true rhq.agent.client.command-spool-file.name=command-spool.dat rhq.agent.client.command-spool-file.params=10000000:75 rhq.agent.client.retry-interval-msecs=15000
- Open the agent prompt. For example, if the agent process is already running, the prompt can be opened by re-running the
rhq-agent.sh
script with the-n
option.agentRoot/rhq-agent/bin/rhq-agent.sh -n
- Send the
setconfig
with the new guaranteed delivery settings.> setconfig rhq.agent.client.command-spool-file.compressed=true rhq.agent.client.command-spool-file.name=my-spool.dat rhq.agent.client.command-spool-file.params=25000000:67 rhq.agent.client.retry-interval-msecs=25000
- Restart the agent process to load the new configuration.
agentRoot/rhq-agent/bin/rhq-agent-wrapper.sh stop agentRoot/rhq-agent/bin/rhq-agent.sh
7.19. Configuring Agent Communication
- A parameter which defines the protocol that the agent uses to talk to the server (
rhq.agent.server.transport
) and any additional transport parameters (rhq.agent.server.transport-params
) - A parameter which defines the protocol that the agent expects for incoming communications from the server (
rhq.communications.connector.transport
) and then any optional transport parameters (rhq.communications.connector.transport-params
)
- servlet (only for agent to server communications)
- sslservlet
- socket (only for server to agent communications)
- sslsocket
Note
protocol://hostname:port/?param1=value¶m2=value
socket://server.example.com:16163/?serverBindAddress=127.0.0.1&serverBindPort=16163&numAcceptThreads=3&maxPoolSize=303&clientMaxPoolSize=304&socketTimeout=60000&enableTcpNoDelay=true&backlog=200
rhq.communications.connector.transport-params
configuration settings which allows transport parameters to be set. These parameters are appended to the end of the URL and can configure both server-side and client-side behavior. For example, the backlog
parameter is used by JBoss ON servers; with this URL, the server sets its backlog value to 200, but this setting is ignored by agents since they are clients. Likewise, the enableTcpNoDelay
parameter is used by agents when they connect to servers, but is ignored by the servers themselves.
8. Managing Databases Associated with JBoss ON
8.1. Running SQL Commands from JBoss ON
Note
Note
- Open the administrative page, with the location
coregui/#Test/ServerAccess/SQL
. For example:http://server.example.com:7080/coregui/#Test/ServerAccess/SQL
- Enter the SQL commands, as appropriate for the JBoss ON Oracle or PostgreSQL database instance. If there are multiple commands, make sure the Continue if statements fail? checkbox is selected. That way, even if one command fails, the other commands will be submitted. Otherwise, the series will be terminated at the first failure.
- Click the Execute SQL button.
8.2. Changing Database Passwords
rhq-server.properties
file. The database password is encoded automatically by the installer before it is stored in the rhq-server.properties
file, to provide some extra protection against unauthorized access to the database password.
rhq-server.properties
file has to be manually encoded and updated for JBoss ON to continue to function.
- Change the password for the JBoss ON user (
rhqadmin
by default) in the database. - Use the
rhq-encode-password.sh
script to encode the password.serverRoot/bin/rhq-encode-password.sh mypassword Encoded password:
1d31b70b3650168f79edee9e04977e34
JBoss ON stores its database password in an encoded form in therhq-server.properties
file. Therefore, the new database has to be properly encoded before it's added to therhq-server.properties
file so that the server reads it correctly. - Edit the
rhq.server.database.password
value in therhq-server.properties
file so that it has the new encoded password value.vim serverRoot/bin/rhq-server.properties rhq.server.database.password=
1d31b70b3650168f79edee9e04977e34
8.3. Editing the JBoss ON Server's Database Configuration
rhq-server.properties
.
Example 8. Default Configuration for a PostgreSQL Database
# Database rhq.server.database.connection-url=jdbc:postgresql://127.0.0.1:5432/rhq rhq.server.database.driver-class=org.postgresql.Driver rhq.server.database.xa-datasource-class=org.postgresql.xa.PGXADataSource rhq.server.database.user-name=rhqadmin rhq.server.database.password=1eeb2f255e832171df8592078de921bc rhq.server.database.type-mapping=PostgreSQL rhq.server.database.server-name=127.0.0.1 rhq.server.database.port=5432 rhq.server.database.db-name=rhq hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
Table 20. rhq-server.properties Parameters for Database Configuration
Parameter | Description |
---|---|
rhq.server.database.type-mapping | Gives the type or vendor of the database that is used by the JBoss ON server. This is either PostgreSQL or Oracle. |
rhq.server.database.connection-url | The JDBC URL that the JBoss ON server uses when connecting to the database, such as jdbc:postgresql://localhost:5432/rhq or jdbc:oracle:oci:@localhost:1521:orcl. |
rhq.server.database.driver-class | The fully qualified class name of the JDBC driver that the JBoss ON server uses to communicate with the database, such as oracle.jdbc.driver.OracleDriver. |
rhq.server.database.xa-datasource-class | The fully qualified class name of the JDBC driver that the JBoss ON server uses to communicate with the database, such as org.postgresql.xa.PGXADataSource or oracle.jdbc.xa.client.OracleXADatasource. |
rhq.server.database.user-name | The name of the user that the JBoss ON server uses when logging into the database. |
rhq.server.database.password | The password of the database user that is used by the JBoss ON server when logging into the database. This password is stored in a hash in the rhq-server.properties file. When the password is changed in the database, then the password must be manually hashed and copied into the rhq-server.properties file. This is described in Section 8.2, “Changing Database Passwords”. |
rhq.server.database.server-name | The server name where the database is found. This must match the server in the connection URL. This is currently only used when connecting to PostgreSQL. |
rhq.server.database.port | The port on which the database is listening. This must match the port in the connection URL. This is currently only used when connecting to PostgreSQL. |
rhq.server.database.db-name | The name of the database. This must match the name found in the connection URL. This is currently only used when connecting to PostgreSQL. |
rhq.server.quartz.driverDelegateClass | The Quartz driver used for connections between the server and the database. The value of this is set by the installer and depends on the type of database used to store the JBoss ON information. For PostgreSQL, this is org.quartz.impl.jdbcjobstore.PostgreSQLDelegate , and for Oracle, this is org.quartz.impl.jdbcjobstore.oracle.OracleDelegate . |
9. Deploying and Managing Storage Nodes
9.1. About High-Speed Metrics Storage
Note
- The agent sends the storage node configuration to the JBoss ON server. The JBoss ON server then sends that updated storage cluster information to every agent associated with a storage node.Each companion agent then updates its storage cluster configuration, in the
rhq-storage-auth.conf
, with the hostname or IP address of the new node. (Likewise, when a node is removed, the server sends the information to each of the companion agents, and the agent removes the hostname or IP address from the list in the local storage node'srhq-storage-auth.conf
file.) - The server receives monitoring data from all agents (not just those associated with a storage node), and sends that information to an available storage node to be stored.
- The storage nodes replicate their monitoring data among each other for high availability and integrity.
Figure 10. Server, Agent, and Metrics Storage Node Communication
- The hostname or IP address of every storage node, stored in the
rhq-storage-auth.conf
- A common port number for the JBoss ON server to use to communicate with the storage node (the client port)
- A common port number for the other storage nodes in the cluster to use to sync data between each other (the gossip port)
- Replicating data between the storage nodes (over the gossip port)
- Taking local snapshots and backing up the data locally
9.2. Deploying and Undeploying Storage Nodes
Note
- The bits are installed on a local system and the storage node is registered with the JBoss ON server.
- The new node information is deployed to the cluster.
Warning
rhq-storage-auth.conf
file so that the allowed hosts list cannot be altered to allow an attacker to gain access to the cluster and the stored data.
9.2.1. Storage Node Requirements
- The hostnames or IP addresses of all storage nodes and the hostname and bind addresses of the JBoss ON server and agents must all be fully-resolvable in DNS. If the IP addresses and hostnames of the storage nodes, servers, or agents are not properly formatted in DNS, then all communication between the different JBoss ON components will fail.
- The firewall must allow communication over the two ports used by the storage nodes. By default, the ports are 9142 and 7100 for the server/client and gossip ports, respectively.
9.2.2. Installing the Storage Node with the Server
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl install --start
9.2.3. Installing Storage Nodes Before Installing the Server
Warning
Warning
rhq-storage-auth.conf
file so that the allowed hosts list cannot be altered to allow an attacker to gain access to the cluster and the stored data.
Important
hosts
file.
- Determine the node and cluster configuration information to use.
- Identify the hostname or IP address of each system which will host a node.
- Define the two ports which the cluster uses for communication (9142 and 7100 by default).
- Before installing any storage node, edit the storage properties file with all of the node and cluster information.
[root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-storage.properties
For example, this configures three nodes, set in therhq.storage.seeds
parameter.rhq.storage.cql-port=9142 rhq.storage.gossip-port=7100 rhq.storage.seeds=192.68.0.0, 192.68.0.1, 192.68.0.2 start=false
- Install the storage node on each system, with its companion agent. This requires the IP address of the JBoss ON server, even though the server is not yet installed.Do not start the storage node or the agent at this point. Do not use the --start option with the installation script.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl install --storage --agent-preference="rhq.agent.server.bind-address=192.68.0.2"
- For each storage node, edit its local
rhq-storage-auth.conf
file. This lists the hostnames or IP addresses for all of the storage nodes in the cluster, one per line.[root@server ~]# vim serverRoot/jon-server-3.2.GA/rhq-storage/conf/rhq-storage-auth.conf 192.68.0.0 192.68.0.1 192.68.0.2
After the server is configured, the local agent will update therhq-storage-auth.conf
file with node hostnames or IP addresses as nodes are deployed and removed from the cluster. - Start each node.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --storage
- Before installing the server, edit the
rhq-server.properties
file to include the connection information for the storage nodes.Add each storage node in a comma-separated listed to therhq.storage.nodes
parameter. Then, add the client and gossip port values.[root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-server.properties rhq.storage.nodes=192.68.0.0,192.68.0.1,192.68.0.2 rhq.storage.cql-port=9142 rhq.storage.gossip-port=7100
- Install the server and an agent. Specifying the
--server
and--agent
options only installs those two components; the storage database is excluded.[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl install --server --agent --start
If you are upgrading an existing JBoss ON agent, then run the upgrade script with the--use-remote-storage-note
option, to load the storage database information from the properties file rather than installing a storage node.[root@server]# serverRoot/jon-server-3.2.GA/bin/rhqctl upgrade --use-remote-storage-node=true
9.2.4. Installing Additional Nodes
rhq-storage.properties
file with the correct ports before running the installation script.
--agent-preference
option to supply the server bind address. For example:
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl install --storage --agent-preference="rhq.agent.server.bind-address=0.0.0.0"
Note
rhqctl
command must be run as root. On Windows, the command prompt must be opened with the option Run as Administrator
.
Figure 11. Joining the Cluster
Warning
rhq-storage-auth.conf
file so that the allowed hosts list cannot be altered to allow an attacker to gain access to the cluster and the stored data.
9.2.5. Setting Automatic Deployment for Nodes
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- Open the Cluster Settings tab.
- The Automatic Deployment option sets whether to deploy a storage node to the cluster as soon as it is installed in JBoss ON or to wait to deploy it manually.The default is to deploy nodes automatically. This can be changed if the environment has other provisioning rules in place.
- Click Save at the bottom of the page.
9.2.6. Deploying Nodes Manually
Warning
rhq-storage-auth.conf
file so that the allowed hosts list cannot be altered to allow an attacker to gain access to the cluster and the stored data.
- Install a node using the
rhqctl install
command. - Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- In the Nodes tab, select the row of the node to deploy.
- Click the Deploy button.
Deploy
operation on the storage node resource.
Note
// deploy a storage node nodes = StorageNodeManager.findStorageNodesByCriteria(StorageNodeCriteria()); node = nodes.get(0); StorageNodeManager.deployStorageNode(node);
9.2.7. Removing Nodes
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- In the Nodes tab, select the row of the node to remove. To select multiple rows, hold the Ctrl key and click the desired rows.
- Click the Undeploy Selected button, and confirm the operation.
Undeploy
operation on the storage node resource.
Note
// undeploy a storage node nodes = StorageNodeManager.findStorageNodesByCriteria(StorageNodeCriteria()); node = nodes.get(0); StorageNodeManager.undeployStorageNode(node);
9.3. Viewing Storage Node Metrics and States
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- The Nodes tab shows the number of unacknowledged alerts for each node.
Figure 12. Storage Node Details
9.4. Changing Cluster Ports
- Change the port settings used by the server.
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- Open the Cluster Settings tab.
- Edit the port numbers.
- The CQL Port sets the client port that the JBoss ON server uses to communicate with all storage nodes to send metrics data.
- The Gossip Port sets the port that nodes use to contact each other to replicate metrics data.
- Click Save at the bottom of the page.
- Change the port settings for each node.
- On the Nodes tab of the storage administration area, click Link to Resource. This opens the node's resource page.
- Open the Configuration tab for the storage node.
- Edit the port numbers.
- The CQL Port sets the client port that the JBoss ON server uses to communicate with all storage nodes to send metrics data.
- The Gossip Port sets the port that nodes use to contact each other to replicate metrics data.
- Click Save at the bottom of the page.
- Restart each node.
- If the client (CQL) port was changed, then also restart each JBoss ON server.
9.5. Viewing Storage Node Alerts
- The JVM heap hits its maximum threshold and causes performance degradation.
- The storage node begins using too much disk space on its system.
- High heap usage, which can lead ot out of memory errors and performance degradation. A dampening rule is in place to prevent alerts for momentary memory spikes.
- High disk usage, which can lead to problems with compaction and other routine operations.Compaction operations are particularly important because compaction merges datafiles on the disk into a single disk file. This frees disk space and improves read performance. If this operations fails, then performance can degrade.An alert is fired for high disk usage if any one of several conditions is met:
- The size of the storage node data exceeds 50% of total disk space.
- The overall amount of disk space used exceeds 75% of the total disk space (regardless of how much disk space the storage node is using).
- The ratio of free disk space to storage node data is less than 1.5. This is calculated by taking the amount of free disk space divided by the disk space used by the storage node. If there is 50MB of free space, and the storage node is using 35MB of disk, then the ratio is 50/35 or 1.42. That is too low and would trigger an alert.
A dampening rule is in place to prevent alerts for momentary usage spikes. - Snapshot failure, meaning a local routine backup operation has failed.
- Maintenance operation failure, meaning either a deploy or undeploy operation for a node failed. Any underlying causes, like an unavailable resource, can be addressed and then the operation can be re-run.
Table 21. Storage Resources for Alerts
Alert | Parent Resource | Resource Type | Area to Address |
---|---|---|---|
High Heap Usage | Cassandra Server JVM | Memory Subsystem | Edit the heap sizes in the storge node JVM configuration |
High Disk Usage | Database Management Services | Storage Service | Increase the disk space for the system hosting the node |
Snapshot Failure | Database Management Services | Storage Service | |
Maintenance Operation Failure | Storage Node | Unavailable storage nodes in the cloud (which prevent updates) |
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- The Nodes tab shows the number of unacknowledged alerts for each node.
- To view the list of alerts, open the Cluster Alerts tab.Every alert is listed with a description of the condition which triggered it, the affected resource, and the time of the alert.
9.6. Enabling Debug Mode for the Storage Node
- On the storage node machine, stop the storage node.
[root@node ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl.sh stop --storage
- Open the storage node's
log4j-server.properties
file.[root@server ~]# vim serverRoot/jon-server-3.2.GA/rhq-storage/conf/log4j-server.properties
- Add or edit the lines for the log threshold property (
log4j.appender.R.Threshold
) and the apache property (log4j.logger.org.apache.cassandra
) to enable debug logging.log4j.appender.R.Threshold=DEBUG log4j.logger.org.apache.cassandra=DEBUG
- Restart the storage node.
[root@node ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --storage
9.7. Managing the Storage Node Heap
- Click the Administration tab in the top navigation bar.
- In the Topology area on the left, select the Storage Nodes item.
- In the Nodes tab, click the hostname or IP address of the storage node to edit.
- In the Configuration area, reset the Max Heap Size setting.
- As recommended, adjust the Heap New Size setting in proportion to the heap size.
- Click the Save button at the bottom of the page.
- Restart the storage node. For example, on the storage node machine:
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --storage
9.8. Backing Up and Restoring the Metrics Storage Database
9.8.1. About Storage Data Files
rhq-data/
directory, which is in the same root directory are the JBoss ON server (such as /opt/jon
). This directory is divided into three subdirectories:
- A commit log for write data that has yet to be written to the database
- A cache directory for archives
- A data directory which contains both all monitoring data (aggregated and raw), snapshot archives of monitoring data, and node (system) data
/opt/jon | rhq-data/ | --------------------- | | | commit_log/ data/ saved_caches/
commit_log/
directory contains binary commit logs which store the writes before they are written to the database. Each commit log is named by the timestamp of when it was created.
--- commit_log | |-- CommitLog-2-timestamp.log
saved_caches/
directory contains archives for the major tables within the storage node, both the metrics tables and the storage node configuration.
--- saved_caches/ | |-- rhq-metrics_index-KeyCache-b.db | |-- rhq-one_hour_metrics-KeyCache-b.db | |-- rhq-raw_metrics-KeyCache-b.db | |-- rhq-six_hour_metrics-KeyCache-b.db | |-- system-local-KeyCache-b.db | |-- system-schema_columnfamilies-KeyCache-b.db | |-- system-schema_columns-KeyCache-b.db | |-- system_auth-credentials-KeyCache-b.db | |-- system_auth-users-KeyCache-b.db
data/
directory contains the most critical data. This directory is organized by keyspace and then table. The keyspaces are rhq (monitoring data), system_auth, system_traces, and system.
rhq/
directory contains all of the metrics data. This is covered in the Using JBoss Operations Network for Monitoring, Deploying, and Managing Resources, but all collected monitoring data is aggregated into averages and trends at periodic intervals. The is raw monitoring data which has recently been collected, and then rolling aggregations for the past hour, past six hours, and past 24 hours. Each time period has its own metrics table. snapshot directory is a UNIX timestamp in milliseconds.
rootDirectory | rhq-data/ | data/ | ------------------------ | | system databases... rhq/ | ------------------------------------------------------------------------------------------- | | | | | | one_hour_metrics raw_metrics schema_version six_hour_metrics twenty_four_hour_metrics metrics_index | ------------------------------------------------ | | |-- rhq-metrics_index-ic-1-Data.db snapshots |-- rhq-metrics_index-ic-1-Digest.sha1 | |-- rhq-metrics_index-ic-1-Filter.db |-- timestamp |-- rhq-metrics_index-ic-1-Index.db |-- timestamp |-- rhq-metrics_index-ic-1-Statistics.db |-- rhq-metrics_index-ic-1-Summary.db |-- rhq-metrics_index-ic-1-TOC.txt |-- rhq-metrics_index-ic-2-Data.db |-- rhq-metrics_index-ic-2-Digest.sha1 |-- rhq-metrics_index-ic-2-Filter.db |-- rhq-metrics_index-ic-2-Index.db |-- rhq-metrics_index-ic-2-Statistics.db |-- rhq-metrics_index-ic-2-Summary.db |-- rhq-metrics_index-ic-2-TOC.txt |-- rhq-metrics_index-ic-3-Data.db |-- rhq-metrics_index-ic-3-Digest.sha1 |-- rhq-metrics_index-ic-3-Filter.db |-- rhq-metrics_index-ic-3-Index.db |-- rhq-metrics_index-ic-3-Statistics.db |-- rhq-metrics_index-ic-3-Summary.db |-- rhq-metrics_index-ic-3-TOC.txt
data/
directory is comprised of database files for different node areas, including users and permissions (system_auth/
), database sessions (system_traces
), and schema and configuration (system
).
rootDirectory | rhq-data/ | data/ | ------------------------------------------------------------ | | | | system_auth/ system_traces/ system/ rhq/ | | | | --------------------------- ------------ | ........ | | | | | | credentials/ permissions/ users/ events/ sessions/ |-- HintsColumnFamily |-- IndexInfo |-- LocationInfo |-- Migrations |-- NodeIdInfo |-- Schema |-- batchlog |-- hints |-- local |-- peer_events |-- peers |-- range_xfers |-- schema_columnfamilies |-- schema_columns |-- schema_keyspaces
9.8.2. Running a Maintenance Operation
- When a node is added or removed from the cluster.
- As part of weekly maintenance.The weekly maintanenace snapshot is scheduled for Sunday at 00:30. The schedule is fixed and cannot be changed.
- Take a snapshot of every node in the cluster.
- Run a repair operation on every node. The repair operation is run on each node consecutively, to update and resolve any data issues.
rhqadmin@localhost:7080$ StorageNodeManager.runClusterMaintenance()
Note
9.8.3. Restoring the Cluster
Note
Important
- Shut down every node in the storage cluster. Run the stop command on every storage machine:
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl.sh stop --storage
- Remove the
commit_log/
directory for each node.[root@server ~]# rm * /opt/jon/rhq-data/data/commit_log/*
- Delete all files in the
metrics_index
directory, except for the snapshot files.[root@server ~]# rm /opt/jon/rhq-data/data/data/rhq/metrics_index/*.*
- Copy all files from the desired snapshot directory into the
metrics_index
directory.[root@server ~]# cp /opt/jon/rhq-data/data/data/rhq/metrics_index/snapshots/timestamp/* /opt/jon/rhq-data/data/data/rhq/metrics_index
- Restart each storage node.
[root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl start --storage
10. Document Information
10.1. Giving Feedback
- Select the Other products group.
- Select RHQ Project from the list.
- Set the component to Documentation.
- Set the version number to 3.2.
- For errors, give the page number (for the PDF) or URL (for the HTML), and give a succinct description of the problem, such as incorrect procedure or typo.For enhancements, put in what information needs to be added and why.
- Give a clear title for the bug. For example,
"Incorrect command example for setup script options"
is better than"Bad example"
.
10.2. Document History
Revision History | |||
---|---|---|---|
Revision 3.2-11 | Jun 06, 2016 | Scott Mumford | |
| |||
Revision 3.2-10 | Oct 03, 2014 | Jared Morgan | |
| |||
Revision 3.2-9 | May 24, 2014 | Ella Deon Ballard | |
| |||
Revision 3.2-7 | December 11, 2013 | Ella Deon Ballard | |
|
Index
A
- affinity groups
- high availability, Creating Affinity Groups
- agent
- default ports, Default Server, Agent, and Storage Node Ports
- persisted configuration
- quiet time, Configuring the Agent Quiet Time (Timeout Period)
- running as a daemon, Running the Agent as a Daemon or init.d Service
- starting, Starting the JBoss ON Agent
- starting (basic), Starting the JBoss ON Agent on a Managed System
- starting as a Windows service, Configuring the Agent as a Windows Service
- starting command console, Starting the JBoss ON Agent on a Managed System
- starting with init.d, Running the Agent as a Daemon or init.d Service, Starting an Agent with a Custom Command
- update settings, Configuring Agent Update Settings
- agents
- load balancing
- load balancing communication with server, Using High Availability and Agent Load Balancing
- authentication
- between JBoss ON servers and JBoss ON agents, Setting up Client Authentication Between Servers and Agents
C
- communication
- settings, Configuring Communication Settings
- configuration
- JBoss ON server, Configuring Servers
- rhq.server properties, Editing JBoss ON Server Configuration in rhq-server.properties
D
- database
- changing passwords, Changing Database Passwords
- editing configuration, Editing the JBoss ON Server's Database Configuration
- management, Managing Databases Associated with JBoss ON
- running SQL commands from JBoss ON, Running SQL Commands from JBoss ON
- discovery
- scan interval, Setting Discovery Scan Intervals
E
- encryption
- configuring, Setting up Encryption
- events
- partitions, Managing Partition Events
F
- failover
- JBoss ON server and high availability, Viewing the Server Failover Lists for Agents
- files
- locations, JBoss ON File Locations
G
- groups
- high availability and affinity, Creating Affinity Groups
- guaranteed delivery
- JBoss ON agent, Setting Guaranteed Delivery for Commands
H
- high availability
- configuring, Using High Availability and Agent Load Balancing
- creating affinity groups, Creating Affinity Groups
- database impact, Server Availability: Multiple Servers in a Single Cloud
- JBoss ON agent, Viewing the Server Failover Lists for Agents
- listing affinity groups, Creating Affinity Groups
- maintenance mode, Putting Servers in Maintenance Mode
- removing JBoss ON servers from the cloud, Removing Servers from the High Availability Cloud
- removing partition events, Removing Partition Events
- viewing partition events, Managing Partition Events
J
- JBoss ON agent
- authentication with JBoss ON servers, Setting up Client Authentication Between Servers and Agents
- changing the IP address, Changing the Agent IP Address
- configuration, Configuring Agents
- directories and files, Agent File Locations
- discovery scan, Setting Discovery Scan Intervals
- failover, Viewing the Server Failover Lists for Agents
- guaranteed delivery, Setting Guaranteed Delivery for Commands
- JVM options, Managing the Agent JVM
- persistent configuration, Viewing the Persisted Configuration
- prompt commands, Agent Prompt Commands
- throttling, Throttling the Agent
- transport parameters, Configuring Agent Communication
- transports, Configuring Agent Communication
- Windows service configuration, Configuring the Agent as a Windows Service
- JBoss ON server
- authentication with JBoss ON agents, Setting up Client Authentication Between Servers and Agents
- changing the URL, Changing the JBoss ON Server URL
- concurrency limits, Setting Concurrency Limits
- configuration, Configuring Servers
- configuring communication settings, Configuring Communication Settings
- configuring rhq.server.properties, Editing JBoss ON Server Configuration in rhq-server.properties
- debug logging, Enabling Debug Logging for the JBoss ON Server
- maintenance mode, Putting Servers in Maintenance Mode
- removing JBoss ON servers from high availability, Removing Servers from the High Availability Cloud
- JVM
- options in the JBoss ON agent, Managing the Agent JVM
L
- load balancing
- agent-server communication, Using High Availability and Agent Load Balancing
- round robin, Agents and Server Partitions: Distributing Agent Load
M
- metrics
- storage database, Deploying and Managing Storage Nodes
P
- persisted configuration
- ports
- defaults for servers and agents, Default Server, Agent, and Storage Node Ports
S
- server
- configuring SMTP for email notifications, Configuring the SMTP Server for Email Notifications
- default ports, Default Server, Agent, and Storage Node Ports
- detection and polling, Setting the Agent to Detect or Poll the Server
- directories and files, Server File Locations
- high availability, Using High Availability and Agent Load Balancing
- silent install, Installing a Server Silently
- starting, Starting the JBoss ON Server, Starting the JBoss ON Server (Basic)
- starting as a Windows service, Running the JBoss ON Server as a Service
- servers
- agent load balancing
- load balancing communication with agent, Using High Availability and Agent Load Balancing
- SSL
- authentication between servers and agents, Setting up Client Authentication Between Servers and Agents
- configuring connections for server-agent communication, Configuring SSL Connections for Server-Agent Communication
- setting up encryption, Setting up Encryption
- storage node
- directories and files, Storage Node File Locations
- starting, Starting the Storage Node
- storage nodes, Deploying and Managing Storage Nodes
- security implications, Deploying and Undeploying Storage Nodes, Installing Storage Nodes Before Installing the Server
T
- throttling
- JBoss ON agent, Throttling the Agent