JBoss Operations Network 2.3

Installation Guide

Installation and Initial Setup of the JBoss Operations Network

Edition 1

Logo

JBoss Documentation

JBoss ON Team

Red Hat JBoss

Legal Notice

Copyright © 2009 Red Hat, Inc..
This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.

Abstract

A guide to download, install, and set up JBoss Operations Network 2.x servers and agents.
Preface
1. Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. We need feedback
1. Download JBoss ON
1.1. Red Hat customers
1.2. Open source community
2. JON 2.x prerequisites
2.1. Operating system
2.2. Java
2.3. Database
2.3.1. PostgreSQL preparation
2.3.2. PostgreSQL quick start installation
2.3.3. Oracle
3. JON Server Installation
3.1. Overview
3.2. Upgrading the JON Server
3.2.1. Remove obsolete alert definitions
3.2.2. Remove unwanted platforms from inventory
3.2.3. Stop the Server and all Agents
3.2.4. Unzip the new version of the JON Server
3.2.5. Install the new versions of JON Agent Plugin Packs
3.2.6. Setup the JON Server
3.2.7. Install the new version of the JON Agent to each Agent machine
3.2.8. Start the new Server and Agents
3.2.9. JON Server resource in inventory
3.3. JON High Availability
3.4. Installation preparation
3.5. Running the installer
3.5.1. Advanced settings
3.5.2. Database settings
3.5.3. Installation settings
3.5.4. Server settings
3.5.5. Complete the JON Server Installation
3.5.6. Installing as a boot-time service
3.6. Startup properties
3.6.1. Database
3.6.2. Server bindings
3.6.3. High Availability bindings
3.6.4. Cluster bindings
3.6.5. JON Console transport security
3.6.6. Incoming Agent communication bindings
3.6.7. Incoming Agent communications transport security
3.6.8. Outgoing Agent communications transport security
3.6.9. Outgoing email
3.6.10. Operation invocation default timeout
3.6.11. Concurrency limits
3.7. Adding and updating Agent plugins
4. JON Agent Installation
4.1. Overview
4.2. Obtaining the Agent
4.3. Installation preparation
4.4. Preparing the Agent for auto-update
4.4.1. Starting the Agent using the wrapper service
4.4.2. Do not alter the launcher scripts
4.4.3. Configuring the Agent using support scripts
4.4.4. Installing Keystores and Truststores
4.4.5. Installing the Agent in a writable directory
4.5. Agent auto-update
4.6. Manually Upgrading the JON Agent
5. Running the JON Agent
5.1. Running on Windows
5.1.1. Running in a Windows console
5.1.2. Installing and running as a Windows service
5.2. Running on UNIX®
5.2.1. Running in a console
5.2.2. Running as a daemon
5.2.3. Running with init.d
5.3. Command line options
5.4. Running embedded in a JON Server
5.5. Using the Agent plugin
5.5.1. Inventorying the Agent Server and its services
5.5.2. Top-level resource
5.5.3. Common services
5.5.4. Java service wrapper launcher (Windows-only)
5.5.5. Agent launcher script (UNIX-only)
5.5.6. Rebooting the Agent and its Java VM
6. Initial auto-discovery and import
6.1. Login
6.2. Installing a license
6.3. Initial auto-discovery and import
7. High Availability configurations
7.1. When should I define an HA configuration?
7.2. HA infrastructure impact
7.2.1. Database impact
7.2.2. Server and Agent endpoints
7.3. When should I use Affinity?
7.3.1. Physical efficiency
7.3.2. Logical efficiency
7.3.3. Warm backup
7.3.4. Affinity behavior
7.4. How do I move to an HA installation?
7.4.1. Moving to HA from JON 2.0.0/2.0.1
7.4.2. Server requirements
7.4.3. Agent requirements
7.5. How do I manage my HA installation?
8. Command Line Interface (CLI) installation
8.1. Remote Client and CLI installation
8.2. Download the Remote Client binary
8.3. Unzip the binary
8.4. Running the JON CLI
8.4.1. Introduction
8.4.2. Running the JON CLI
8.4.3. JON Command line options
8.4.4. Built-in commands
8.4.5. Working with the CLI
8.4.6. Proxy
8.5. References
9. JON Best Practices
9.1. Best practices for managing JBoss Application Servers
9.1.1. DynaGroups
9.1.2. Useful alerts
9.1.3. JBoss Application Server setup
9.1.4. Common issues
9.1.5. Debugging issues
9.1.6. Log tracking
9.2. Best Practices for managing JBoss Tomcat Servers
9.2.1. Enabling JBoss ON monitoring of your Tomcat Server
9.2.2. Enabling response time metrics for your Tomcat Server
9.3. Best practices for remote clients
9.3.1. Deciding on a Remote Client Approach
Index

Preface

1. Document Conventions

This manual uses several conventions to highlight certain words and phrases and draw attention to specific pieces of information.
In PDF and paper editions, this manual uses typefaces drawn from the Liberation Fonts set. The Liberation Fonts set is also used in HTML editions if the set is installed on your system. If not, alternative but equivalent typefaces are displayed. Note: Red Hat Enterprise Linux 5 and later include the Liberation Fonts set by default.

1.1. Typographic Conventions

Four typographic conventions are used to call attention to specific words and phrases. These conventions, and the circumstances they apply to, are as follows.
Mono-spaced Bold
Used to highlight system input, including shell commands, file names and paths. Also used to highlight keys and key combinations. For example:
To see the contents of the file my_next_bestselling_novel in your current working directory, enter the cat my_next_bestselling_novel command at the shell prompt and press Enter to execute the command.
The above includes a file name, a shell command and a key, all presented in mono-spaced bold and all distinguishable thanks to context.
Key combinations can be distinguished from an individual key by the plus sign that connects each part of a key combination. For example:
Press Enter to execute the command.
Press Ctrl+Alt+F2 to switch to a virtual terminal.
The first example highlights a particular key to press. The second example highlights a key combination: a set of three keys pressed simultaneously.
If source code is discussed, class names, methods, functions, variable names and returned values mentioned within a paragraph will be presented as above, in mono-spaced bold. For example:
File-related classes include filesystem for file systems, file for files, and dir for directories. Each class has its own associated set of permissions.
Proportional Bold
This denotes words or phrases encountered on a system, including application names; dialog box text; labeled buttons; check-box and radio button labels; menu titles and sub-menu titles. For example:
Choose SystemPreferencesMouse from the main menu bar to launch Mouse Preferences. In the Buttons tab, select the Left-handed mouse check box and click Close to switch the primary mouse button from the left to the right (making the mouse suitable for use in the left hand).
To insert a special character into a gedit file, choose ApplicationsAccessoriesCharacter Map from the main menu bar. Next, choose SearchFind… from the Character Map menu bar, type the name of the character in the Search field and click Next. The character you sought will be highlighted in the Character Table. Double-click this highlighted character to place it in the Text to copy field and then click the Copy button. Now switch back to your document and choose EditPaste from the gedit menu bar.
The above text includes application names; system-wide menu names and items; application-specific menu names; and buttons and text found within a GUI interface, all presented in proportional bold and all distinguishable by context.
Mono-spaced Bold Italic or Proportional Bold Italic
Whether mono-spaced bold or proportional bold, the addition of italics indicates replaceable or variable text. Italics denotes text you do not input literally or displayed text that changes depending on circumstance. For example:
To connect to a remote machine using ssh, type ssh username@domain.name at a shell prompt. If the remote machine is example.com and your username on that machine is john, type ssh john@example.com.
The mount -o remount file-system command remounts the named file system. For example, to remount the /home file system, the command is mount -o remount /home.
To see the version of a currently installed package, use the rpm -q package command. It will return a result as follows: package-version-release.
Note the words in bold italics above — username, domain.name, file-system, package, version and release. Each word is a placeholder, either for text you enter when issuing a command or for text displayed by the system.
Aside from standard usage for presenting the title of a work, italics denotes the first use of a new and important term. For example:
Publican is a DocBook publishing system.

1.2. Pull-quote Conventions

Terminal output and source code listings are set off visually from the surrounding text.
Output sent to a terminal is set in mono-spaced roman and presented thus:
books        Desktop   documentation  drafts  mss    photos   stuff  svn
books_tests  Desktop1  downloads      images  notes  scripts  svgs
Source-code listings are also set in mono-spaced roman but add syntax highlighting as follows:
package org.jboss.book.jca.ex1;

import javax.naming.InitialContext;

public class ExClient
{
   public static void main(String args[]) 
       throws Exception
   {
      InitialContext iniCtx = new InitialContext();
      Object         ref    = iniCtx.lookup("EchoBean");
      EchoHome       home   = (EchoHome) ref;
      Echo           echo   = home.create();

      System.out.println("Created Echo");

      System.out.println("Echo.echo('Hello') = " + echo.echo("Hello"));
   }
}

1.3. Notes and Warnings

Finally, we use three visual styles to draw attention to information that might otherwise be overlooked.

Note

Notes are tips, shortcuts or alternative approaches to the task at hand. Ignoring a note should have no negative consequences, but you might miss out on a trick that makes your life easier.

Important

Important boxes detail things that are easily missed: configuration changes that only apply to the current session, or services that need restarting before an update will apply. Ignoring a box labeled 'Important' will not cause data loss but may cause irritation and frustration.

Warning

Warnings should not be ignored. Ignoring warnings will most likely cause data loss.

2. We need feedback

To see all outstanding issues regarding this guide, please visit: https://jira.jboss.org/jira/browse/JOPR
If you find a typographical error in the Installation Guide, or if you have thought of a way to make this manual better, we would love to hear from you! Please submit a report in JIRA: https://jira.jboss.org/jira/browse/JOPR against the component Documentation.
If you do have a suggestion for improving the documentation, try and be as specific as possible when describing it. If you have found an error, please include the section number and some of the surrounding text so we can find it easily.

Chapter 1. Download JBoss ON

1.1. Red Hat customers

JBoss Operations Network 2.3 is available for download from the Customer Support Portal.

Procedure 1.1. Download JBoss Operations Network 2.3 and the License

  1. Log into the Customer Support Portal, click on Software and then click on the Product drop-down box arrow. Navigate to the JBoss Operations Network software download. If you do not have access to the JBoss ON downloads section please contact your sales representative.
  2. Download the following files to run JBoss ON:
    • Download the JBoss Operations Network 2.3 Base Distribution by clicking on the Download icon.
    • You will also need to download the required plugin pack for JBoss ON 2.3 by navigating to the required JBoss Operations Network software download. For example, to download the plugin pack for JBoss ON for EAP navigate to the JBoss ON for EAP software download in the Product drop-down box. Then click on the Download icon next to the EAP Plugin Pack for JBoss ON 2.3 link.
    Unlike previous releases of JBoss ON, there are no operating system specific versions of the distribution.
  3. Next you will need to download the JBoss ON license. There are three types of licenses available:
    • License With Monitoring - this license gives you access to all features of JBoss ON (Monitoring, Inventory, Configuration, Alerts, Control Operations and Software feeds)
    • License Without Monitoring - this license gives you limited access to JBoss ON (Inventory, Configuration, Control Operations and Software Feeds)
    • Evaluation License With Monitoring - this is a 30-day evaluation that gives you access to all features of JBoss ON.
    To download the license right click on the license and save the file to your desktop. After the JBoss ON installation is complete, you will be prompted to upload the license.
    If you do not have access to the correct license, please contact your sales representative.

1.2. Open source community

Jopr is the open source and unsupported release of the JBoss Operations Network. The RHQ project also provides an open source release of JBoss ON. The license file for RHQ and Jopr is either located in the SVN repository or is generated by the code.

Procedure 1.2. Download Jopr 2.3

  1. Navigate to the Jopr homepage. Click on the Downloads tab. Under Jopr - Releases > Jopr 2.3.0 click on the Download link located in the Download table under Link.
  2. Download the Jopr Server.

Chapter 2. JON 2.x prerequisites

2.1. Operating system

JBoss ON 2.x Server and Agent are supported on Linux and Windows with the amd64, i686 and ia64 architectures. Other platforms that support Java 5 are supported, but you must disable native support as this impacts upon the functionality of the JON Agent.

Note

All 64-bit platforms, with the exception of Linux on x64, are only supported by JBoss ON when the operating system is in 32-bit compatibility mode.

2.2. Java

JBoss ON 2.0 requires Java 5 in order to run both the Server and the Agent; either a JRE or a JDK can be used. Java 5 must be installed on all machines monitored with an Agent, as well as on the machine hosting the JON Server. Download Java 5 from http://java.sun.com/javase/downloads/index_jdk5.jsp.

Note

JBoss ON 2.3 requires Java 5 or Java 6

Important

GNU libgcj is not supported.
After downloading and installing the latest JRE or JDK you will need to configure your system. This consists of two tasks:
  1. Setting the JAVA_HOME environment variable to the installation directory; and
  2. Ensuring the system is using the correct JDK (or JRE) using the alternatives command.
The JAVA_HOME environment variable tells the JON Server and the JON Agent where to locate the JVM. Setting JAVA_HOME generally overrides the values for java, javac and java_sdk_1.5.0 in alternatives, however it is recommended that you specify the value for consistency.
Setting the JAVA_HOME Environment Variable on Linux
After installing the JDK, ensure the JAVA_HOME environment variable exists and points to the location of the JDK installation.
Determine whether JAVA_HOME is set by executing the following command:
~]$ echo $JAVA_HOME
If JAVA_HOME is not set, the value must be set to the location of the JDK or JRE installation on your system. This can be achieved by adding two lines to the .bashrc configuration file. Open .bashrc and add a line similar to the following one anywhere inside the file:
export JAVA_HOME="/usr/lib/jvm/jdk1.5.0_<version>"
Selecting the Correct System JVM on Linux using alternatives
Next you will need to set java, javac and java_sdk_1.5.0 using the alternatives command.
On systems with the alternatives command, including Red Hat Enterprise Linux and Fedora, it is possible to choose which JDK (or JRE) installation to use, as well as which java and javac executables should be run when called.
As the root user, call /usr/sbin/alternatives with the --config java option to select between the JDKs and JREs installed on your system. For example:
				
home]$ sudo /usr/sbin/alternatives --config java
				
There are 3 programs which provide 'java'.
				
Selection    Command
-----------------------------------------------
 1           /usr/lib/jvm/jre-1.5.0-gcj/bin/java
 2           /usr/lib/jvm/jre-1.6.0-sun/bin/java
 *+ 3        /usr/lib/jvm/jre-1.5.0-sun/bin/java
				
Enter to keep the current selection[+], or type selection number:

The Sun JDK, version 5, is required to run the java executable. In the alternatives listing above, a plus (+) symbol next to a number indicates the option currently being used. Press Enter to keep the current JVM, or enter the number corresponding to the JVM to select that option.
As the root user, repeat the procedure above for the javac command and the java_sdk_1.5.0 environment variable:
home]$ sudo /usr/sbin/alternatives --config javac
home]$ sudo /usr/sbin/alternatives --config java_sdk_1.5.0

2.3. Database

In order to run JBoss ON 2.0, an external database must be installed. The supported databases are:
  • Postgres: To download and install Postgres, navigate to the Postgres website.
  • Oracle: To download and install Oracle, navigate to the Oracle website.
After installing the database, make note of the JDBC URL, username and password for the database. This information is required during the JBoss ON Server installation.

Unsupported databases

Please note that although the installer offers a choice of other databases (for example, embedded H2 and MS SQL server), these are for demonstration purposes only and are not supported.

2.3.1. PostgreSQL preparation

PostgreSQL version

JBoss ON currently supports versions 8.2.4-8.4.x of PostgreSQL. Earlier 8.2 versions are not recommended. Postgres 8.1 is not supported.

Important information for users of PostgreSQL version 8.4

Users of PostgreSQL 8.4 should ignore the suggested fsm_max_pages setting in the database configuration section below. The FSM pages setting no longer appears in the postgresql.conf file in versions 8.4 and onwards as this is now managed by PostgreSQL itself. If you try to add this setting, PostgreSQL 8.4 will fail to start. See the PostgreSQL 8.4 release notes for more information.

Note

If you do not have PostgreSQL installed, refer to Procedure 2.1, “Download and install PostgreSQL” for steps on how to install PostgreSQL.

postgresql.conf

PostgreSQL requires minor changes to the database configuration. Make the following changes to the postgresql.conf file:
## not necessary  if the database is started with the -i flag
listen_addresses = '*' 

## performance changes for JBoss ON
shared_buffers = 80MB 		#  default is 32MB 
work_mem = 2048 		#  default is 1MB 
statement_timeout = 30s 	#  default is 0s
checkpoint_segments = 10	#  default is 3
max_fsm_pages = 100000        #  default is 204800
JBoss ON can use up to 55 database connections for the server (prior to JON 2.1, this number was 110). PostgreSQL also allows for connections reserved for administrators. These connections are counted in the pool of max_connections and therefore need to be added to the total number of max_connections. Assuming we have 5 connections reserved for the administrator, edit the postgresql.conf file as follows:
max_connections = 60	   #  default is 100
superuser_reserved_connections = 5 #  default is 3 
max_prepared_transactions = 60     #  default is 5 (in v8.3) or 0 (in v8.4)

max_prepared_transactions

Note that max_prepared_transactions is set to the same value as max_connections as explained in the PostgreSQL documentation. See specifically the max_prepared_transactions (integer) section.
If you are using the Postgres plugin to monitor this database instance, add one more connection per (logical) database you have setup in PostgreSQL. For further information about this plugin refer to the Postgres Server section of the Managed Resources Guide.

Kernel parameters

Depending on the operating system you are using, you may need to adjust some kernel parameters. Refer to the Managing Kernel Resources documentation for more information.

pg_hba.conf

Update the pg_hba.conf file to allow the newly created role to connect from the machine the JBoss ON Server is installed on, (for example localhost). For details on how to do this please see Client Authentication.
After completing the above step, restart PostgreSQL for the changes to take effect. If no errors are displayed, the database is now ready to support a JBoss ON installation.
For more information on tuning Postgres, see Tuning your PostgreSQL Server.
Fixes for "Relation RHQ_Principal does not exist" error
Sometimes the database connection is marked as valid but the install still fails with the Relation RHQ_Principal does not exist error. This occurs when a new database is created by running initdb in a non-C locale through PostgreSQL instances.
To fix this error:
  1. Using a database explorer, create an empty table called RHQ_PRINCIPAL in the database used for JBoss ON.
  2. Click on Install server
    The installer displays a warning about an existing schema. Overwrite the existing schema as it only consists of one empty table.
Another option is to specify the encoding of the created database as SQL-ASCII at creation time. For example:
initdb -D /my/test/data -E SQL_ASCII --locale en_US.UTF-8
Replace the directory after -D with your target directory for the database.

2.3.2. PostgreSQL quick start installation

Procedure 2.1. Download and install PostgreSQL

  1. Download the latest release of PostgreSQL.
    For UNIX®: Download the source from http://www.postgresql.org/ftp/source/.
    For Windows: Download the installer from http://www.postgresql.org/ftp/binary/.
  2. Install postgres.
    For UNIX: To build and install PostgreSQL please refer to the Installation Instructions
    For Windows: Follow the installations instructions described at: PostgreSQL Installer
    Postgres can also be installed via a package installer such as yum on Fedora.
  3. You must ensure that the Postgres authentication mechanism is properly configured for the following commands to work. To give all users on the computer access to the database, add the following lines to the data/pg_hba.conf configuration file.
    local all all              trust
    host  all all 127.0.0.1/32 trust
    
    To change the postgres user's password:
    sudo passwd postgres
  4. Initialize the database.
    The database must be initialized before starting the server.
    service postgresql initdb
    
  5. Start Postgres.
    For UNIX:
    service postgresql start
    
    For Windows:
    If Postgres is installed as a service then it may already be running. Otherwise:
    net start pgsql-8.3
    
  6. Create a PostgreSQL role named rhqadmin with password rhqadmin. Do this via pgAdmin or from the command line as follows:
    cd <postgres-install-dir>
    createuser -h 127.0.0.1 -p 5432 -U postgres -S -D -R rhqadmin
    
  7. Create a PostgreSQL database named rhq, specifying the rhqadmin role as the owner. Do this via pgAdmin or from the command line as follows:
    cd <postgres-install-dir>
    createdb -h 127.0.0.1 -p 5432 -U postgres -O rhqadmin rhq
    
  8. If you are using this PostgreSQL installation for more than just demonstration or testing purposes, refer to Section 2.3.1, “PostgreSQL preparation” to configure your machine for production use with JBoss ON.

2.3.3. Oracle

Oracle version

JBoss ON currently supports version 10g of Oracle. Earlier versions are not supported.
To install Oracle, please consult the Oracle documentation.
When you have an Oracle server installed, continue to Section 2.3.3.1, “Oracle preparation” to prepare your database.

2.3.3.1. Oracle preparation

To use Oracle with JBoss ON, you will need to follow the following three steps. More preparation is required if the advanced Oracle setup will be used. Please refer to Section 2.3.3.6, “Advanced Oracle configuration”.
  1. Create or determine an Oracle instance to be used for the JBoss ON database.
    It is recommended that the Oracle server to be used by JBoss ON runs on its own hardware. Install Oracle on the machine to be used, and create a database. You can choose any name for the database. The Oracle instance is now ready for the next step.
  2. Create a user that JBoss ON will use to access Oracle.
    There are several ways to create a user in Oracle. One way is with SQL*Plus. First, log into the Oracle instance via as the system user with SQL*Plus, then issue the CREATE USER command:
    SQL> CREATE USER jon IDENTIFIED BY jon;
    
    Create a user named jon with a password of jon.
  3. Grant the required permissions to the Oracle user.
    The Oracle user must possess the connect and resource roles. This can be easily done in SQL*Plus with the GRANT command:
    SQL> GRANT connect, resource TO jon;
    
  4. Make sure DB_BLOCK_SIZE is at least 8k
    SQL> show parameter db_block_size;
    NAME                                 TYPE        VALUE
    ------------------------------------ ----------- ------------------------------
    db_block_size                        integer     8192	
    
    
Oracle is now ready to accept a JBoss ON installation.

2.3.3.2. JON user requirements for XA

JBoss ON uses internally two phase commit for some of its database actions. To be able to recover from two phase commit failures, it is required to grant special privileges to the 'jon' datasource user. If you do not do this, XAException.XAER_RMERR errors will occur. The privileges you need to grant include:
GRANT SELECT ON sys.dba_pending_transactions TO jon;
GRANT SELECT ON sys.pending_trans$ TO jon;
GRANT SELECT ON sys.dba_2pc_pending TO jon;
GRANT EXECUTE ON sys.dbms_system TO jon;

GRANT login

In order to execute the above GRANT statements, you will need to be logged in as SYS. To login as SYS, use the following: CONNECT sys/your_sys_password AS sysdba;

2.3.3.3. Sessions and processes

Currently the recommended algorithm for determining the maximum number of Oracle processes (as kept in v$resource_limit) which JBoss ON should use is the following:
Take the maximum of:
  1. 1.5 * total number of JBoss ON Agents in the environment, and;
  2. 60 * total number of JBoss ON Servers in the environment
then add 40 if you are also using Oracle Enterprise Manager (EM).
For example if you had 100 JBoss ON Agents and 2 JBoss ON Servers, and you were using Oracle EM, you would have:
  1. 1.5 * 100 = 150
  2. 60 * 2 = 120
So the maximum of the two is 150. Then add 40 to support Oracle EM and that gives 190 processes.
The number of sessions (as kept in v$resource_limit) should be:
  • 1.1 * number of processes
In this example a maximum of 209 sessions should be sufficient. These settings should prevent you from seeing errors such as ORA-00018: maximum number of sessions exceeded.

2.3.3.4. SGA and PGA sizes

Oracle settings for SGA and PGA sizes are very important in the performance of JBoss ON. If they are too small, your database will perform very slow, affecting JBoss ON in a very negative way. Talk to your DBA for proper sizing of your Oracle's SGA and PGA requirements. The settings you need to tune are sga_target and pga_aggregate_target.

2.3.3.5. Tuning open cursors

If the following sql:
select max(a.value) as highest_open_cur, p.value as max_open_cur
from v$sesstat a, v$statname b, v$parameter p
where a.statistic# = b.statistic#
and b.name = 'opened cursors current'
and p.name= 'open_cursors'
group by p.value;
returns a max_open_cur value of less than 300 then execute the following:
ALTER SESSION SET OPEN_CURSORS = 300 SCOPE=BOTH;

2.3.3.6. Advanced Oracle configuration

There are optional configurations that can help Oracle perform effectively with large JBoss ON environments. This configuration is not necessary for small to medium environments. An example of an environment where this type of configuration would help performance is an environment with hundreds of JBoss ON Agents.

Procedure 2.2. Advanced Oracle configuration

  1. Create a new database using the Oracle Database Configuration Assistant. Select New Database (Includes datafiles = No). Decline to install the Example Schemas to save space.
  2. If an advanced configuration is being used, Oracle should be installed on a dedicated host. So select Typical Memory configuration. Select OLTP as the type of database sizing to use. Allocate the highest percentage of system resources that you can afford. This should be set between 70-90%, ideally in the higher range.

    Warning

    Locally manage all tablespaces.
  3. Create the JBoss ON user.
    CREATE USER jon IDENTIFIED BY jon;
    
  4. Grant permissions to the new user.
    GRANT CONNECT, RESOURCE TO jon;
    

Chapter 3. JON Server Installation

This chapter describes the JBoss Operations Network 2.3 Server installation process. We recommended that you also read the JON Server Guide for more information about the JON Server.

3.1. Overview

The JBoss ON Server can run on either a Windows or UNIX platform. The following provides a summary of the installation steps you need to perform:
  1. If you already have a JBoss ON Server installed and you need to upgrade it, go to Section 3.2, “Upgrading the JON Server”.
  2. It is recommended you read through Chapter 7, High Availability configurations. This section will help you decide if you need high availability features and if so, how many servers you should install and how they should be configured.
  3. Before you install your Server, read Section 3.4, “Installation preparation” to prepare your environment.
  4. After you are familiar with the high availability features and you have prepared your environment for the JBoss ON Server, read Section 3.5, “Running the installer” to install your server.
  5. After the server has been installed and configured, you can run it as a Windows service, from a console, or run it as a daemon or init.d script in a UNIX environment. Go to Running the JBoss ON Server for more information.

3.2. Upgrading the JON Server

Supported upgrades

Upgrade is not supported for version JON 1.x or any other version of JON prior to JON 2.x GA. You will need to perform a new JON installation if moving from a pre-JON 2.x GA version.

Loss of data

Loss of minimal monitoring data is inevitable given the down-time involved in shutting down instances of the Agents during the upgrade process. If you have a Resource in inventory corresponding to the JON Server itself, all monitoring data for that resource will be lost.
Follow the steps outlined below to upgrade your JON Server:

3.2.1. Remove obsolete alert definitions

Important

This applies to JON 2.0.0 only.
Before upgrading JON you must remove alert definitions with conditions for obsolete metrics. This is because alert definitions are not easily removed after the upgrade. Remove alert definitions, including alert templates, for the following metrics:

Table 3.1. Metrics

ResourceType
Metric
Postgres Server
User Time
Kernel Time
Physical Memory
Virtual Memory

3.2.2. Remove unwanted platforms from inventory

Remove "out of service" platform resources prior to the upgrade.

3.2.3. Stop the Server and all Agents

The JON Server must be upgraded before any Agents are upgraded. Shut down all agents and wait until all agents show red availability in the GUI before shutting down the server.
  • Stop all the JON Agents

Auto-upgrade

In JBoss ON 2.2 agents hve the ability to upgrade automatically. So once you install JBoss ON 2.2, you will find that upgrading the agents thereafter will be simpler, because the agent will be able to detect when a server has been upgraded and the agent will upgrade itself accordingly. Therefore, if you have prepared your agents for auto-update, you will not want to shut them down. Let them detect when the new JBoss ON Server comes online and upgrade themselves.
  • Stop the JON Server after all agents are down
Do not stop the JON Database.

3.2.4. Unzip the new version of the JON Server

Unzip the server distribution to the directory where it will be executed from.
cd /opt 
unzip jon-server-2.3.0.GA.zip

Important

Do not copy the new server installation on top of a previous server installation.
The directory structure within the zipfile ensures that the new server has a version-specific installation directory name. The above commands create a directory named /opt/jon-server-2.3.0.GA.
If you have made modifications to your original server's Startup Properties (e.g. enabling SSL, SMTP), either when first installing or through the ./bin/rhq-server.properties file, merge these changes into the new server's rhq-server.properties file.

Merging Startup Properties

If you prefer not to edit the new server's ./bin/rhq-server.properties file, you can merge your changed values during installation via the Advanced Settings checkbox.
If you are running the server on Windows and installed the original server as a Windows service, uninstall the Windows service by executing the following command:
cd <old-server-install-dir>/bin
./rhq-server.bat remove
Install a Windows service for the new Server by executing the following command:
cd <new-server-install-dir>/bin
./rhq-server.bat install

3.2.5. Install the new versions of JON Agent Plugin Packs

The JON server distribution includes the core agent plugins it requires for operation. Additional plugins are delivered via plugin packs specific to the needs of the installation (for example, EWS, EAP, SOA-P). Each plugin pack contains one or more plugins in a zip file. The plugin packs can be unzipped anywhere. The following example places them at the top of the server distribution. For each plugin pack execute the following command:
cd [install-dir]
unzip jon-plugin-pack-<name>-2.3.0.GA.zip
Then follow the instructions in the README.txt supplied with the plugin pack to install the agent plugins.

Note

In an HA environment it is only necessary to install the plugin packs to one HA server. Once installed, all of the HA servers will have access to the agent plugins.

3.2.6. Setup the JON Server

Backup Your Database

We recommend that you back up your database prior to proceeding. If problems arise during the database upgrade, having a backup will allow you to restore to your previous state.

Locate Your License

Register your license with the new version of JON. Your existing license file is located in old-server-install-dir/jbossas/server/default/deploy/rhq.ear/license/license.xml. Copy this to a safe location and reuse it during the upgrade process.
Follow the instructions in Chapter 3, JON Server Installation to install the Server. Once you have entered your database connection info, the installer should detect that there is an existing JON database and display the following prompt:
A database schema already exists. What do you want to do?
Choose the default, Keep (maintain existing data). Do not choose Overwrite (lose existing data), otherwise all the existing inventory, metric, and history data from your original JON installation will be lost.

Important

During the upgrade you may see error messages in the console similar to the following:
ERROR [ClientCommandSenderTask] {ClientCommandSenderTask.send-failed}Failed to send
command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.timeout=1000,
rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.enterprise.communications.Ping,
invocation=NameBasedInvocation[ping]}]]. Cause: org.jboss.remoting.CannotConnectException:[.....]
These can be safely ignored.

3.2.7. Install the new version of the JON Agent to each Agent machine

Follow the Agent upgrade instructions in Section 4.6, “Manually Upgrading the JON Agent”.

3.2.8. Start the new Server and Agents

Start the new Server and Agents

3.2.9. JON Server resource in inventory

  1. If a JBoss ON Server Resource corresponding to your old JBoss ON Server installation is in the inventory, remove it from the inventory.
  2. (optional) If desired, import the new JBoss ON Server resource into the inventory.

3.3. JON High Availability

Starting with version 2.1.0, JON supports a multi-server environment. Running multiple JON servers provides a High Availability (HA) environment. A multi-server HA environment allows JON Agent failover and distribution of load.
HA is integrated into the standard installation process and there is no separate HA installer. It is important to understand the following:
  • Multiple servers can be configured against the same database.
  • Each server requires a unique name for identification.
  • Each server requires a publicly accessible endpoint (public to the population of Agents).
Upgrading an existing server, or installing a new server, will result in a single-server environment. To add more servers to the HA "Server Cloud" run the installer on the target server machines, each time configuring for the same database, endpoint information and selecting a unique server name. Servers can be added and removed from the cloud at any point in time.
To learn more about whether an HA environment is appropriate for your needs, and how to move to HA from an existing JON installation, see Chapter 7, High Availability configurations.

3.4. Installation preparation

Java and Database prerequisites
Before running the installer, ensure you have installed a supported external database and JDK 5 or JDK 6. For instructions on how to install a JDK refer to Section 2.2, “Java”.
The JDBC URL, database username and password are required during the installation process.
Synchronized machine clocks
All JBoss ON Servers and Agents must have synchronized clocks. Clock variations will cause issues in resource synchronization and availabilities, metric measurements, graphing, and importing resources into inventory. Refer to Network Time Protocol (NTP) for information on installing and configuring NTP to ensure your clocks are synchronized.
DNS setup
DNS forward and reverse mapping entries must be present for all hosts involved in monitoring. This includes all JBoss ON Servers and all machines running Agents.

3.5. Running the installer

Unlike JBoss ON 1.x, there is no longer a standalone installer. You should have a JON Server distribution file (jon-server-2.X.zip).

Procedure 3.1. Run the installer

  1. Unzip the server distribution to the directory where will be executed from. For example:
    cd /opt
    unzip jon-server-2.3.0.GA.zip
    
    The directory structure within the zipfile ensures that the Server has a version-specific installation directory name. For example, the above commands will create a directory named /opt/jon-server-2.3.0.GA. The /opt/jon-server-2.3.0.GA directory should not exist prior to the unzip operation.

    Windows installation

    Do not install the JON Server into a directory with a path longer than 19 characters (e.g. use C:\Program Files\ rather than C:\Documents and Settings\myusername\ ). Excessively long pathnames may cause errors during execution of the server.
  2. Install Agent Plugin Packs:
    The JON server distribution includes the core agent plugins it requires for operation. Additional plugins are delivered via plugin packs specific to the needs of the installation (for example, EWS, EAP, SOA-P). Each plugin pack contains one or more plugins in a zip file. The plugin packs can be unzipped anywhere, the following example places them at the top of the server distribution. For each plugin pack:
    					
    cd <install-dir>
    unzip jon-plugin-pack-<name>-2.3.0.GA.zip
    
    Follow the instructions in the README.txt supplied with the plugin pack to install the agent plugins.

    Note

    In an HA environment it is only necessary to install the plugin packs to one HA server. Once installed all of the HA servers will have access to the agent plugins.
  3. Run the JON Server:
    For UNIX, execute the following from the command line:
    cd <install-dir>/bin
    ./rhq-server.sh start
    
    For Windows, execute the following from the command line:
    cd <install-dir>\bin
    rhq-server-console.bat start
    
  4. Point your browser to http://localhost:7080/. This will display the JON Server Installer web application.
  5. Clicking the Click here to continue the installation link brings you to the main installer page.

3.5.1. Advanced settings

By default, the installer displays only the typical settings required for installation. If you have advanced requirements, click the Show Advanced Settings check box to display the advanced settings in the next two sections.

Important

Toggling the Show Advanced Settings check box initializes the values for displayed settings. Be careful not to toggle this setting after entering your installation data.

3.5.2. Database settings

The main installer page will appear differently depending on the database settings. You must configure the database for this server installation. The database settings area of the installer will look like this:

Procedure 3.2. Configure the database

  1. Set your database connection properties by updating the default values as necessary.

    Important

    In Oracle, if you select the 'overwrite tables' option but there is nothing to overwrite, because this is a first-time installation, you may see an error message in your server log:
    ERROR [DBSetup] {DBSetup.dropped-table-error}Failed to drop table 
    [SOME_RHQ_TABLE] or one of its sequences. Cause: ErrorCode=[2289]; 
    SQLState=[42000]; Message=[ORA-02289: sequence does not exist]; 
    Type=[java.sql.SQLException]
    
    These messgaes can be safely ignored
  2. Test your database connection by clicking the Test Connection button. If this test fails, check your connection properties and ensure your database server is running.
  3. Existing Database options: If an existing JON Server schema is discovered from a previous JON installation, the installer will give you the choice of either upgrading it to the latest schema version, or overwriting it and losing all existing data. Choose the default (Keep (maintain existing data)) unless you are confident you do not need any previous JON data. If any errors occur during the database schema installation, check the logs.

Important

The JBoss ON 1.x database schema is incompatible with the JBoss ON 2.0 schema; you cannot upgrade a 1.x schema to a 2.0 schema.

3.5.3. Installation settings

The installation settings page will differ depending on whether a server has previously been installed for the configured database. If this is a new database, or an upgrade from a version prior to 2.1.0, the installer page will look like this:
If you are upgrading or reinstalling a 2.1.0 server, or installing a new server into a High Availability cloud, the installer page will look identical with the exception of an additional drop-down list of existing registered servers as follows:
Registered Servers
The Registered Servers drop-down list does not appear unless a 2.1.0 server has been previously installed to the configured database.
This drop-down displays the servers currently registered for the configured database. For single-server environments, only one entry is displayed. For High Availability environments, the list contains an entry for each server in the server cloud. When upgrading or reinstalling an existing 2.1.0 server, select that server from the list.
Select the appropriate server from the list. The installation settings configured for that server are now displayed; ensure the values are correct.
Server name
The Server Name uniquely identifies the JON Server being installed. It defaults to the resolved host name of the machine, but can be set to a different value if necessary.
Selecting a Registered Server from the drop-down and then modifying the Server Name will result in a new server registration. The installer treats this as an additional server being added to the HA Server Cloud. When adding a new server it is not necessary to select from the drop-down list.

Server Name

Avoid using generic names such as server, localhost or s1 in favor of explicit names that easily identify the JON Server.
Public server endpoint
The remaining fields define the public server endpoint information for the JON Server-JON Agent communication. After installation, you can update the Server Endpoint information via the JON GUI Administration pages.

Important

The Server Endpoint Address must be public so that all JON Agents are able to resolve the supplied value. This is particularly important in High Availability (multi-Server) environments where JON Agents are connecting to various JON Servers in the cloud.

3.5.4. Server settings

The settings displayed depend on whether the Show Advanced Settings option is enabled.
Standard settings
The standard installation process contains the typical settings required to install JON. The default values are usually appropriate for most standard and advanced settings. If necessary, the server settings can be modified at a later time by editing the jon-server-home/bin/rhq-server.properties file.
If you bind the server to an address other than 0.0.0.0 you must also edit the jon-server-home/bin/rhq-server.properties file. Enable the java.rmi.server.hostname property and set it to the same address to which the server is bound.
Ensure your SMTP settings are set correctly to enable e-mail notification of alerts. You can update these settings if your SMTP server is not on the same machine as the JON server.
Set the Embedded Agent Enabled field to No if you are not monitoring the JON Server, or Yes if you are using the standalone agent.
Advanced settings
All advanced settings are explained in detail in Section 3.6, “Startup properties”.

Important

If you change any startup properties that require a restart, restart the server immediately after the installation so the settings can take effect. The server will not correctly function until this is done.

3.5.5. Complete the JON Server Installation

Click on Install JON Server. An intermediate page is displayed during the installation. Once the installation has completed the message Starting up, please wait... is displayed. Click the Done! Click here to get started. link to begin using your JON server.

Important

Some browsers such as Safari and Opera cannot display the login page, but instead show the last page. If this happens, press the refresh button on your browser and click on the link again.

Install errors on Postgres

There have been some cases where the installation of PostgreSQL failed with an error "Relation RHQ_Principal" does not exist. Refer to Section 2.3.1, “PostgreSQL preparation” for a workaround.

3.5.6. Installing as a boot-time service

You can install the JON Server to run when your computer boots up. On Windows, this means installing the JON Server to start as a Windows Service. On UNIX, it means installing the JON Server startup script as an init.d startup script. To install as a boot-time service on either Windows or UNIX, refer to the Running the JON Server section in the JON Server Guide.

3.6. Startup properties

When the JON Server starts, it will load the properties defined in the bin/rhq-server.properties file. These startup properties provide settings that are necessary to bootstrap the JON Server (for example, what host address to bind the web application to, or what ports the server will listen to for incoming agent requests).

Note

These startup properties are different to the values stored in the database and can be modified via the JON Console's Administration page.
Startup properties are set when you first run the installer using the advanced installation checkbox. Refer to Section 3.5, “Running the installer”.
The startup parameters are documented below.

3.6.1. Database

These properties define the JON server's database and how it can connect to it.

Database type

rhq.server.database.type-mapping
Defines the vendor of your database that will be used as JON's back end persistence store. The only supported values are PostgreSQL or Oracle.

Database connection URL

rhq.server.database.connection-url
The JDBC URL that the JON Server uses when connecting to the database. An example is jdbc:postgresql://localhost:5432/rhq or jdbc:oracle:oci:@localhost:1521:orcl.

Database JDBC driver class

rhq.server.database.driver-class
The fully qualified class name of the JDBC driver that the JON Server uses to communicate with the database. An example is oracle.jdbc.driver.OracleDriver.

Database XA DataSource class

rhq.server.database.xa-datasource-class
The fully qualified class name of the JDBC driver that the JON Server uses to communicate with the database. Examples of this are org.postgresql.xa.PGXADataSource or oracle.jdbc.xa.client.OracleXADatasource.

Database user name

rhq.server.database.user-name
The name of the user that the JON Server uses when logging into the database.

Database password

rhq.server.database.password
The password of the database user that is used by the JON Server when logging into the database.

Database server name

rhq.server.database.server-name
The server name where the database is found. This must match the server in the connection URL. This is currently only used when connecting to PostgreSQL.

Database the port

rhq.server.database.port
The port on which the database is listening. This must match the port in the connection URL. This is currently only used when connecting to PostgreSQL.

Database DB name

rhq.server.database.db-name
The name of the database. This must match the name found in the connection URL. This is currently only used when connecting to PostgreSQL.

3.6.2. Server bindings

These settings bind the JON Server to specific IP addresses and ports. You may need to change these properties if you have firewall issues that require specific addresses and ports to be used.

Server bind address

jboss.bind.address
The JON GUI Console, among other services, bind to this IP address. This is the host in the JON Console URLs; e.g. the myhost in http://myhost:7080.

Prior to JON 2.1

If you change this value while the Incoming Agent Communications Transport parameter (rhq.communications.connector.transport) is set to servlet, you must also change the value of the Incoming Agent Communications Bind Address parameter (rhq.communications.connector.bind-address) to match this address.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

HTTP the port

rhq.server.startup.web.http.port
The JON GUI Console listens to this port for unsecured HTTP requests coming in. This is the port number in the JON Console URLs; e.g. the 7080 in http://localhost:7080. This is also the unsecure port used when defining the server's HA endpoint.

Prior to 2.1

If you change this value while your Incoming Agent Communications Transport (rhq.communications.connector.transport) is set to servlet, you must change the value of the Incoming Agent Communications The port (rhq.communications.connector.transport) parameter to match this port number.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Secure HTTPS the port

rhq.server.startup.web.https.port
The JON GUI Console listens to this port for secured HTTPS requests coming in. This is the port number in the JON Console URLs; e.g. the 7443 in http://localhost:7443. This is also the secure port used when defining the server's HA endpoint.

Prior to 2.1

If you change this value, and your Incoming Agent Communications Transport (rhq.communications.connector.transport) is set to sslservlet, you must change the value of the Incoming Agent Communications The port (rhq.communications.connector.transport) parameter to match this port number.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Web service the port

rhq.server.startup.webservice.port
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Naming service the port

rhq.server.startup.namingservice.port
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Naming Service RMI the port

rhq.server.startup.namingservice.rmiport
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

JRMP Invoker RMI The port

rhq.server.startup.jrmpinvoker.rmiport
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Pooled Invoker RMI The port

rhq.server.startup.pooledinvoker.rmiport
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

AJP Port

rhq.server.startup.ajp.port
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Unified invoker port

rhq.server.startup.unifiedinvoker.port
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

Aspect delpoyer port

rhq.server.startup.aspectdeployer.bind-port
The port used by an internal service.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

3.6.3. High Availability bindings

Since 2.1, there are settings that configure the JON Server to identify itself to, and communicate with, other JON Servers or server-side components. They are required for a single-server installation and also facilitate membership in a multi-server, High Availability, JON Server Cloud.

Note

These settings cannot be changed in the rhq-server.properties files but are managed via the JON GUI post-installation.

3.6.3.1. Server name

rhq.server.high-availability.name
Each JON Server is identified by a unique name. The installer will default this value to the localhost name although it does not need to be a resolvable machine name. The only restriction is that all server names be unique in the JON server cloud.
Although present in rhq-server.properties this setting is managed only via the JON HA Administration Console.

3.6.3.2. Server public address

The public server endpoint IP address. This is the address that must be recognized by all agents needing access to this server. The installer will default this value to the localhost address.
This setting is managed only via the JON HA Administration Console.
The server endpoint address is used in conjunction with the servlet and sslservlet transport over either the HTTP Port (rhq.server.startup.web.http.port) or the Secure HTTPS Port (rhq.server.startup.web.https.port) to provide High Availability server endpoints to agents.

3.6.3.3. Server Affinity Group

JON performs load balancing and agent fail-over when operating with a multi-server configuration (JON Server Cloud). In certain situations, regional datacenters for example, it may make sense for particular JON Agents to prefer, or have affinity for, certain servers. JON Agents assigned to a particular Affinity Group will prefer JON Servers assigned to the same Affinity Group. Note that the pairing is not strict and JON can service any agent with any server depending on resource load and availability. By default a server is not assigned an Affinity Group. For larger Server Clouds Affinity Group management is performed via the JON GUI.

3.6.3.4. Registered servers

When installing or upgrading the JON server against an existing database, the installer will present a list of previously installed JON Server Names. These Server Names represent the servers in the current JON Server Cloud. If you are performing an upgrade, or re-installing a JON Server, select the appropriate server name from the list. The installer will populate the other High Availability settings as previously configured for the server. The server name should not be changed otherwise the installer will treat this as a new server installation, but the other HA settings can be updated at this time.
If you manually enter a new server name, it is installed as a new server. If no other servers exist, this is called a single-server configuration. Otherwise, the new server is added to the existing High Availability Server Cloud.
This list is not presented if the database is new or there are no previously installed servers.

3.6.4. Cluster bindings

Prior to 2.1, these are settings that configure the JON Server to participate in a JON Server cluster setup. You will normally only need to change these if you have firewall issues that require specific addresses and ports to be used. For more details about JBoss cluster settings see http://www.jboss.org/community/docs/DOC-12460.
If you change any cluster binding values, you must shutdown and restart the JON Server in order for the changes to take effect.

Partition name

jboss.partition.name
The name of the JON Server cluster partition. All JON Servers participating in the cluster must use the same name.

Partition bind address

jgroups.bind_addr
The bind address used by JGroups to support clustering communications. If you have a multi-homed machine, set this to the appropriate NIC IP address. If you do not plan on running the JON Server as part of a cluster of JON Servers, you may set this to 127.0.0.1 so it binds only to the local loopback interface.

Partition UDP multicast group IP address

jgroups.udp.mcast_addr
The multicast address used by the JGroups channel to broadcast messages to members of the cluster.

Partition UDP multicast the port

jboss.hapartition.mcast_port
Used by an internal clustering service.

Partition UDP EJB3 entity cache multicast the port

jboss.ejb3entitypartition.mcast_port
Used by an internal clustering service to support EJB3 entity caching.

Partition UDP alert cache multicast the port

jboss.alertcachepartition.mcast_port
Used by an internal clustering service to support alert caching.

Partition UDP loopback

rhq.server.startup.partition.udpLoopback
On UNIX, this can typically be left as false. On Windows machines, because of the media sense feature being broken with multicast, even after disabling media sense, set this UDP loopback setting to true.

HA JNDI the port

rhq.server.startup.hajndi.port
Used by an internal service to define the port on which the HA-JNDI stub is made available.

HA JNDI RMI the port

rhq.server.startup.hajndi.rmiport
Used by an internal service. To be used by the HA-JNDI service once bound.

HA JNDI auto discovery group the port

rhq.server.startup.hajndi.autodiscoverygroupport
Multicast port used for cluster auto-discovery.

HA JRMP invoker RMI the port

rhq.server.startup.hajrmpinvoker.rmiport
Used by an internal service.

HA pooled invoker the port

rhq.server.startup.hapooledinvoker.port
Used by an internal service.

JGroups UDP IP time to live

jgroups.udp.ip_ttl
Used by JGroups as a performance tuning parameter. See the JBossAS and JGroups documentation on the UDP protocol's "ip_ttl" setting for more information.

3.6.5. JON Console transport security

JON Console Keystore

rhq.server.startup.keystore.filename
The JON Console can accept browser requests over HTTPS. If you want to authenticate your JON Console to remote browsers, you need to put an SSL certificate in a keystore file. This setting points to the location of your keystore file. Note that this file name must be a relative file path relative to the <JON Server Install Dir>/jbossas/server/default/conf directory. The JON Server ships with a self-signed certificate in a default keystore file.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

JON Console keystore password

rhq.server.startup.keystore.password
The password of the keystore file. This is so the JON Console can access the keystore file in order to be able to serve the certificate to browser clients.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

JON Console SSL protocol

rhq.server.startup.keystore.sslprotocol
The protocol that browser clients should use to communicate with the JON Console.
If you change this value, you must shutdown and restart the JON Server in order for the change to take effect.

3.6.6. Incoming Agent communication bindings

These settings define how the JON Server will accept incoming messages from the JON Agents. See Communications Configuration for more information on setting up the JON Server's agent communications subsystem.

Incoming Agent communications transport

rhq.communications.connector.transport
Defines how the JON Agents need to transport messages to the JON Server. Typical values are servlet, and sslservlet.

Important

Any other transports such as "socket" and "sslsocket" have not been tested for functionality and performance and therefore are not supported.
If you elect to use the only two currently supported transports, servlet or sslservlet, it means the agent requests will go through the JON Server web application layer (i.e. the secure Tomcat Connector).
If you elect to use sslservlet, not only will agent requests route through the web application layer, but they are also secured though the secure Tomcat Connector. What this means is that the keystore used for incoming agent message authentication will be the same as that which you set up as described in Section 3.6.5, “JON Console transport security”.
Note that this transport setting does not restrict agents from only going over that particular transport. By default, the JON Server always deploys the comm connector that allows for both servlet and sslservlet traffic. What this means is that the agent decides what transport is used when it sends messages to the server. If the server has its transport set to servlet, but the agent is configured to talk to the server via sslservlet, the messages the agent sends will be via sslservlet.

Incoming Agent communications bind address

rhq.communications.connector.bind-address
This is the address that is placed in the server's JBoss/Remoting locator URL.
Prior to 2.1: The address that the JON Server binds its connector to so it can receive JON Agent messages.
Since 2.1: This defines the endpoint that the JON Server will bind its connector to. It also represents the public endpoint address that all agents can use to connect to the server.

Incoming Agent communications the port

rhq.communications.connector.bind-port
Prior to 2.1: The port that the JON Server listens to in order to accept JON Agent messages.
Since 2.1: This defines the endpoint that the JON Server will bind its connector to. It also represents the public endpoint address that all agents can use to connect to the server.
This will be hidden from view in the installer, although it will still appear in the rhq-server.properties file. It can be left blank; the server will set this to either the HTTP port or HTTPS port depending on the transport you configured the server with (i.e., servlet or sslservlet). This port will be used in the JBoss/Remoting locator URL of the server and represents the public port that all agents can use to connect to the server. However, the agents are free to go over the HTTP or HTTPS port depending if the agent wants to use servlet or sslservlet.

Incoming Agent communications transport parameters

rhq.communications.connector.transport-params
This defines additional transport parameters the JON Server will set on its connector that will accept incoming messages from the JON Agents. See Communications Configuration - TransportParameters for further information on transport parameters.

Agent multicast detector enabled

rhq.communications.multicast-detector.enabled
If true, the JON Server will attempt to auto-detect JON Agents coming online and going offline using multicast detection. Your network must support multicast traffic for this to work.

Agent multicast detector bind address

rhq.communications.multicast-detector.bind-address
The address that the multicast detector directly binds to. This is not used, or needed, if you have not enabled multicast detection.

Agent multicast detector multicast address

rhq.communications.multicast-detector.multicast-address
The address that the multicast detector will broadcast messages to. This is not used, or needed, if you have not enabled multicast detection.

Agent multicast detector the port

rhq.communications.multicast-detector.port
The port that the multicast detector will broadcast messages to. This is not used, or needed, if you have not enabled multicast detection.

3.6.7. Incoming Agent communications transport security

These settings define how the server secures its communication channel when accepting incoming messages from the JON Agents. See Securing Communications for more detailed information on setting up server-agent communication security.

Note

If you are not using an Incoming Agent Communications Transport (rhq.communications.connector.transport), these settings are not used.
The other half of the communications channel, the outgoing side, has analogous settings. You can share keystores, truststores and other settings for both sides, or you can customize your communications for either side independently.

Incoming secure socket protocol

rhq.communications.connector.security.secure-socket-protocol
The secure protocol that agents must use when communicating with this JON Server.

Incoming keystore file

rhq.communications.connector.security.keystore.file
The keystore file that contains a certificate that authenticates the JON Server to the agents.

Incoming keystore algorithm

rhq.communications.connector.security.keystore.algorithm

Incoming keystore type

rhq.communications.connector.security.keystore.type
The file format of the keystore.

Incoming keystore password

rhq.communications.connector.security.keystore.password
The password that is used to gain access to the keystore file.

Incoming keystore key password

rhq.communications.connector.security.keystore.key-password
The password that is used to gain access to the key inside the keystore.

Incoming keystore alias

rhq.communications.connector.security.keystore.alias
The alias that identifies the JON Server's key within its keystore.

Incoming truststore file

rhq.communications.connector.security.truststore.file
The truststore file that contains certificates that this JON Server trusts. If you need the JON Server to authenticate JON Agents, you must set this; otherwise it is not needed. This truststore contains certificates for all JON Agents that need to communicate with this JON Server. Refer to the Incoming Client Authentication Mode.

Incoming truststore algorithm

rhq.communications.connector.security.truststore.file

Incoming truststore type

rhq.communications.connector.security.truststore.type
The file format of the truststore file.

Incoming truststore password

rhq.communications.connector.security.truststore.type
The password that is used to gain access to the truststore file.

Incoming client authentication mode

rhq.communications.connector.security.client-auth-mode
Indicates if you need the JON Server to authenticate the JON Agents that are sending it messages. If you are using a secured transport but do not have trusted certificates for all of your JON Agents in a truststore, you must set this to none.
Valid values are: none, want or need

3.6.8. Outgoing Agent communications transport security

These settings define how the server will secure its communication channel when sending outgoing messages to the JON Agents. Refer to Securing Communications in the JON Agent Guide for more information on setting up the Server-Agent communications security.
The other half of the communications channel, the incoming side, has analogous settings. You can share keystores, truststores and other settings for both sides, or you can customize your communications for either side independently.

Note

Note that if your JON Agents are not using a secure transport, these settings are not used.

Outgoing secure socket protocol

rhq.server.client.security.secure-socket-protocol
The secure protocol that the server must use when sending outgoing messages to agents. The agents must be expecting to use this protocol for messages that it receives.

Outgoing keystore file

rhq.server.client.security.keystore.file

Outgoing keystore algorithm

rhq.server.client.security.keystore.algorithm

Outgoing keystore type

rhq.server.client.security.keystore.type

Outgoing keystore password

rhq.server.client.security.keystore.password

Outgoing keystore key password

rhq.server.client.security.keystore.key-password

Outgoing keystore alias

rhq.server.client.security.keystore.alias

Outgoing truststore file

rhq.server.client.security.truststore.file

Outgoing truststore algorithm

rhq.server.client.security.truststore.algorithm

Outgoing truststore type

rhq.server.client.security.truststore.type

Outgoing truststore password

rhq.server.client.security.truststore.password

Outgoing Server authentication mode enabled

rhq.server.client.security.server-auth-mode-enabled
Indicates if you want the JON Server to authenticate the JON Agents that are sending it messages. If you are using a secured transport but do not have trusted certificates for all of your JON Agents in a truststore, you must set this to none.

3.6.9. Outgoing email

Email SMTP hostname

rhq.server.email.smtp-host
The hostname of the SMTP server that is used when the JON Server sends emails.

Email SMTP the port

rhq.server.email.smtp-port
The port that the JON Server sends emails over when communicating with the SMTP server.

Email from address

rhq.server.email.from-address
The "From:" header of all emails sent by the JON Server.

3.6.10. Operation invocation default timeout

rhq.server.operation-timeout
An important feature of JON is the ability to invoke operations, or control actions, on a resource. The JON Server can ask an agent to invoke a particular operation on a particular resource managed by that agent. The remote, asynchronous nature of operation invocations, however, can be problematic. For example, the network could go down, the resource could crash, etc. Operation timeout is the default number of seconds that the JON Server will wait for an operation to complete and an agent to provide the results. If this timeout expires, the operation is considered to have failed.

Note

Note that this is only a fallback default value - individual operations can define their own timeouts (in the plugin descriptor) or individual operation invocations can themselves specify a timeout. Those override this default rhq.operation-timeout setting.

3.6.11. Concurrency limits

JON is designed to scale to large numbers of agents. The JON Server must, therefore, take into account the possibility that it could get flooded with messages if many or all agents attempt to communicate with the server simultaneously. For example, this could happen if the JON Server is restarted after being down for a period of time. JON Agents will then detect that the JON Server has come back up and attempt to immediately send it a backlog of messages.
To help alleviate problems that could occur during high load, the JON Server is configured to limit the number of concurrent messages allowed to be processed by different subsystems within the JON Server. If more messages arrive concurrently, and the limit is exceeded, those additional messages will be dropped and the agent will be asked to send them later. The following configuration settings define types of messages that have concurrency limits associated with them.

Web connections

rhq.server.startup.web.max-connections
JON limits the number of web connections that can be concurrently created. This includes GUI connections made by browsers. It can also include connections made by agents, if agent connections are made via the servlet transport (Refer to Incoming Agent Communications Transport). Note that if agent requests are routed over web connections, you need to ensure that the Global Concurrency limit (rhq.communications.global-concurrency-limit) is slightly lower than this Web Connections limit so as not to lock out GUI users from accessing the user interface during times of high agent load.
The limit on web connections is the same for both non-secured http requests and secured https requests, so the total max connections allowed is actually twice what this setting is. For example, if the max web connections is set to 300, then 300 http requests will be allowed and 300 https requests will be allowed, for a possible total of 600 concurrent web connections.

Global concurrency limit

rhq.communications.global-concurrency-limit
Aside from the Web Connections (rhq.server.startup.web.max-connections) limit, all other concurrency limits will only affect incoming agent messages and not GUI/browser requests. The global concurrency limit will limit the total number of agent messages coming into the server, regardless of the type of message. In other words, if this global concurrency limit is set to 300, no more than 300 total agent messages can be processed at any one time, regardless of what kinds of messages are coming in. The rest of the concurrency limit settings will put a limit on the number of messages of particular message types. Keep in mind that if other concurrency limits are set higher than this global limit, they are effectively capped at this global limit since you can never have more than this global limit number of messages being processed concurrently.

Inventory report

rhq.server.concurrency-limit.inventory-report
Inventory reports are sent from the agent when the agent starts up, and periodically thereafter. Inventory reports can be large, depending on the number of resources on the agent machine.

Availability report

rhq.server.concurrency-limit.availability-report
Availability reports are regularly sent from the agent, typically every 60 seconds. Availability reports are usually very small, but occur in large numbers due to the high frequency of their transmission.

Inventory synchronization

rhq.server.concurrency-limit.inventory-sync
Inventory synchronizations occur when the agent needs to synchronize its inventory with that of the server. Agents typically synchronize at startup. Traffic that flows as part of inventory synchronizations is usually large, depending upon the number of resources managed by the agent.

Content report

rhq.server.concurrency-limit.content-report
Content reports are similar to inventory reports except they contain information about discovered content (i.e., installed packages of software). These reports can be large depending on the number of installed software the agent has discovered and is managing.

Content download

rhq.server.concurrency-limit.content-download
Content downloads occur when a resource on an agent needs to ask for the content of a package version, usually for the purpose of installing the package.

Measurement report

rhq.server.concurrency-limit.measurement-report
Measurement reports are periodically sent to the server whenever the agent completes measurement collections. The number and size of measurement reports can vary, depending on the number and frequency of measurements scheduled to be collected. The greater the number of schedule measurements the agent needs to collect, the more frequently measurement reports are sent, and the larger the reports will be.

Measurement schedule request

rhq.server.concurrency-limit.measurement-schedule-request
Similar to inventory synchronization, measurement schedule requests are sent to the agent asking the server for an up-to-date set of measurement schedules that have to be collected.

3.7. Adding and updating Agent plugins

This section explains the steps needed to install a new agent plugin or update an existing agent plugin to your JBoss ON environment.
  1. Log in to the JBoss ON GUI as a user with permissions to edit the server settings.
  2. Go to the Administration > System Configuration > Plugins page
  3. Upload the new or updated plugin jar file:
    • If you do not want to use the GUI to upload the jar file, you can explicitly copy your plugin .jar file to <server-install-dir>/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins.
    • If you have to copy more than one plugin, and those plugins have dependencies on each other, you should either; upload the plugins one at a time, with the first ones being the ones that the other plugins depend on, or; you should shut down the server and copy the plugin jars to the rhq-plugins directory and then restart the server.
    • If you have more than one JBoss ON Server in your JBoss ON Server Cloud, you only need to deploy the plugins to one of the servers; the other servers will automatically detect and copy the plugins.
  4. At this point, you can wait for the server to auto-detect your new plugin jar file, or you can force it to detect it immediately by pressing the SCAN button. The server will log messages in its log file when the plugin has been detected and deployed properly.
  5. Once the JBoss ON Server has detected and deployed the plugin jars, you can then push the plugins out to the agents. You can do this in one of several ways:
    1. If you have the agent prompt available you can execute at the agent prompt the command:
      							
      plugins update
    2. If you have the agent imported into your JBoss ON inventory, you can execute the operation "Update All Plugins" which tells the agent to pull down any new or updated plugins that the agent does not yet have.
    3. If you have multiple agents imported into inventory, you can execute a group operation of "Update All Plugins" so all of your agents pull down the plugins.

Chapter 4. JON Agent Installation

4.1. Overview

The JON Agent can run on either a Windows or UNIX platform and does not require the execution of a special installer program. By default, the agent is not fully configured out-of-the-box. The first time you start the agent, or anytime that the agent's configuration has been cleaned, you must answer a series of setup questions. This can be done via a console window or by specifying a valid configuration file. After the agent has been configured you can run it as a Windows service, from a console, or run it as a daemon or init.d script in a UNIX environment.
  1. First install one or more JBoss ON Servers.
  2. If you already have agents installed and you need to upgrade them, see Section 4.6, “Manually Upgrading the JON Agent”
  3. To install a new JBoss ON Agent, follow the steps in the next sections.
  4. After the agent has been installed and configured, you can run it as a Windows service, from a console, or run it as a daemon or init.d script in a UNIX environment. Go to Chapter 5, Running the JON Agent
  5. If you would like to be able to have your agent auto-updated in the future please see Section 4.4, “Preparing the Agent for auto-update”

4.2. Obtaining the Agent

In JON 2.3, the JON Agent is bundled with the JON Server and is not available as a separate download. This agent bundle is called the agent update binary. It is called this because it is used to not only install a new agent, but will also be used to upgrade the agent in the future. If there is a JON Server currently running in your environment, you can pull down an agent update binary .jar directly from the server using the following instructions.
  1. Point your browser to http://<your-server-hostname>:7080/agentupdate/download and save the agent binary update .jar in a directory where you want to install the agent. The file you save should have a .jar extension. <your-server-hostname> should be the hostname or IP address of the server that is running and 7080 is the port on which that the server is accepting HTTP requests.
  2. Copy the agent update binary .jar you downloaded from the JON Server to your chosen directory and run java -jar <agent-update-binary.jar> --install where <agent-update-binary.jar> is the name of the file you downloaded from the server.
    This will tell the agent update binary to extract the JON Agent distribution and install a fresh copy of it in the rhq-agent subdirectory. At this point, you will have a fully installed JON Agent located in a rhq-agent directory where your agent update binary is located.
    From here on, the instructions will refer to this "rhq-agent" directory as <agent-install-dir>.

4.3. Installation preparation

When the JON Agent is first executed, it enters into setup mode. In order for the JON Agent to contact the JON Server, you must enter values for the following parameters.

Table 4.1. The JON Agent setup mode parameters

Parameter Description
Agent Name
This is a unique name for the agent. The default is the fully qualified domain name (FQDN) of the host system. However, you can specify any name you want, as long as it is unique among all other agents in the system.
Agent Hostname or IP Address
The IP address the agent will bind to in order to listen for incoming messages. Refer to the JON Agent Communcation Services section in the JON Agent Guide for more information.
Agent Port
The port that the agent listens to for incoming messages.
RHQ Server Hostname or IP Address
The hostname or IP address the Server is listening to for incoming messages from agents. This is the address that the agent connects to when sending its initial registration request. If you are running in a RHQ High Availability (HA) environment (that is, you have multiple RHQ Servers running), this will not necessarily be this agent's primary Server. If you only have a single RHQ Server running in your environment, this address is the one the agent will always use when it needs to send messages to the Server.
RHQ Server Port
The port that the Server listens to for the incoming message registration message from the agent. Again, if you are running in RHQ HA environment, this isn't necessarily the port that all of your RHQ Servers are listening to. However, it should be set to the port that your "registration" Server is listening to.

Important

If the agent fails to register with the server and seems to hang after outputting the message The agent does not have plugins - it will now wait for them to be downloaded..., or otherwise does not work property after configuring it, please check the agent log file for error messages (<agent-install-dir>/logs/agent.log). The agent log will normally tell you what the problem is. Typically, problems occur when you bind your agent to an IP address or hostname that is not resolvable by the JBoss ON Servers or is otherwise not reachable by the JBoss ON Servers. Make sure your agents are bound to IP addresses or hostnames that are recognizable and reachable by all your JBoss ON Servers. Similarly, make sure all of your JBoss ON Servers' public endpoint addresses are resolvable by your JBoss ON Agent. Even if you entered a valid server endpoint in the agent setup question, the agent may attempt to switch to another JBoss ON Server immediately after startup registration, and if one or more JBoss ON Servers are installed with unresolvable hostnames or IPs (as seen by this agent), the agent will fail to start properly

Procedure 4.1. JON Agent Windows configuration

  1. Configure the agent by following one of the options below; option 1 is the default method. The rhq-agent.bat script is located in the <agent-install-dir>\bin\ directory.
    Option 1
    Run the rhq-agent.bat script in a console and follow the prompts, entering the appropriate values. If you need help on a particular preference setting, enter !? at the setup prompt for a description of that preference.
    C:\rhq-agent-2.3.0.GA\bin> rhq-agent.bat
    
    Option 2
    Edit the agent-configuration.xml and ensure to enter the correct settings. Run the rhq-agent.bat script and specifiy the agent-configuration.xml. For example:
    C:\rhq-agent-2.3.0.GA\bin> rhq-agent.bat --config ..\conf\agent-configuration.xml
    
    Option 3
    Run the rhq-agent.bat script and specify a valid file containing the values of all the setup parameters. For example, the command below uses a file called myAnswers.txt.
    C:\rhq-agent-2.3.0.GA\bin> rhq-agent.bat --input myAnswers.txt --nonative
    
    The --nonative temporarily disables the native system to allow reading input from a file. The contents of the myAnswers.txt file used in the example above is shown below. Each answer must be on its own line. To enable reading from standard input again, the last line must read native enable.
    [Agent Name] agentdomain.example.com
    [Agent Hostname or IP Address] agentdomain.example.com
    [Agent Port] 16163
    [RHQ Server Hostname or IP Address] serverdomain.example.com
    [RHQ Server Port] 7080
    native enable
    
  2. After the agent has connected to the server, the agent command prompt appears. Enter the command identify to identify the RHQ Server.

Important

Once the agent is configured, it persists its configuration in the Java Preferences backing store. Once this happens, agent-configuration.xml is no longer needed or used. Editing agent-configuration.xml will no longer have any effect on the agent, even if you restart the agent. If you want the agent to pick up changes you make to agent-configuration.xml, you must either restart the agent with the --cleanconfig command line option or use the config --import agent prompt command.
Reconfiguring the JON Agent
If you have previously installed a JON Agent on a machine and need to wipe all of its old persistent configuration data, start with a clean JON Agent by passing in the command line option -l, or --cleanconfig. This forces the agent to enter setup mode. If at any time, you need to start the JON Agent in setup mode, specify the -s, or --setup, command line option.
If you only want to reconfigure one or a few agent settings, you have a couple of options:
  1. Assuming the agent is imported in inventory, you can reconfigure the agent using JBoss ON itself. Go to the agent's Configuration tab in the UI and you can change the agent's configuration there.
  2. Use the setconfig agent prompt command. This will set the new configuration setting and persisted it so it survives agent restarts.

Important

Just changing agent-configuration.xml will not affect the agent. A fully configured agent will ignore agent-configuration.xml; only if you restart an agent with --cleanconfig will the agent reload agent-configuration.xml.
Configuring the JBoss ON Agents Runtime Environment
The agent has an environment script that is read when the agent is started up. This file is called rhq-agent-env.bat on Windows and rhq-agent-env.sh on UNIX. Edit this file to change the environment of the agent. The comments in that file document the different settings. Some of the settings are also explained in the Running the JBoss ON Agent pages.
Debug mode
If you need to run the agent and its launcher scripts in debug mode so that the scripts and the agent itself log debug messages, define the environment variable RHQ_AGENT_DEBUG to "true". To disable debug mode, unset the RHQ_AGENT_DEBUG environment variable or set it to "false". You can alternatively use the agent's debug prompt command, in order to toggle the debug mode while the agent is running (specifically debug --file - execute help debug from the agent prompt for more information on this command).

4.4. Preparing the Agent for auto-update

With the introduction of High Availability (HA), your environment can now support multiple agents. How you manage those agents now becomes very important. The agent auto-update feature introduced in JBoss ON 2.2 allows your agents to automatically update themselves, without the need for an administrator to manually log onto every agent box and do it manually.
In order to auto-update your agent, you must keep a few rules in mind. If you follow these recommendations, you should be able to update your agents to new versions easily, without any additional manual configuration necessary. If you do not follow these recommendations, you will still be able to have your agent update itself, but you will most likely need to perform some additional manual administration tasks in order for your agent to start back up in the same configuration it was in before the update.

Important

Agent auto-updates are only available when going from JBoss ON version 2.2 to a later version. You cannot auto-update agents from any previous JBoss ON version.

4.4.1. Starting the Agent using the wrapper service

On any platform (Windows or UNIX), the agent can be started in two basic ways:
  1. In a console; or
  2. As a background daemon.
To run the agent in a console, use rhq-agent.sh or the rhq-agent.bat directly - this provides you with a command prompt allowing you to type commands to the agent from your keyboard. To run the agent as a background daemon process, you use rhq-agent-wrapper.sh or rhq-agent-wrapper.bat. The rhq-agent-wrapper.sh and rhq-agent-wrapper.bat is called the wrapper script and it runs the agent as a service with no console input (just like any other service such as "sshd"). On Windows platforms, this wrapper script installs and runs the agent as an actual Windows Service.
In almost all production environments, you will want to start the agent as a background daemon process. On UNIX, this means you usually run the agent at boot time from init.d. On Windows, this means you install the agent as a Windows Service that automatically starts at boot time. The agent auto-update feature assumes you start your agent in this way. If you do not, and the agent auto-updates itself, the old agent running in the console will be shutdown and the new agent will be restarted as a background service if at all possible (caveat: on Windows, if the agent is not installed as a Windows Service, the agent will attempt to be restarted in a console window).

4.4.2. Do not alter the launcher scripts

The agent ships with several launcher scripts along with several support scripts in the /bin directory. You must not change the launcher scripts. If you need to customize the agent's configuration, you can do so by editing the support scripts that are used by the launcher scripts for their configuration. Therefore, you must not edit rhq-agent.sh, rhq-agent.bat, rhq-agent-wrapper.sh or rhq-agent-wrapper.bat. You can edit the others. Below are the script files that are used by the agent; those listed in bold italic are the launcher scripts that must not be changed, the others are the support scripts that can be modified:

Table 4.2. Script Files

Script File Description
rhq-agent.sh
Launcher script to start the agent in a console window on UNIX
rhq-agent.bat
Launcher script to start the agent in a console window on Windows
rhq-agent-wrapper.sh
Launcher script to start the agent as a background daemon on UNIX
rhq-agent-wrapper.bat
Launcher script to start the agent as a Windows Service on Windows
rhq-agent-env.sh
Script to setup the agent environment on UNIX
rhq-agent-env.bat
Script to setup the agent environment on Windows
wrapper/rhq-agent-wrapper.conf
Configuration for the agent's Windows Service on Windows
wrapper/rhq-agent-wrapper.env *
Sets up additional environment variables for the Windows Service on Windows
*This do not exist out of the box when you install an agent but you can create and configure it when appropriate. Most people, however, will not need this.
wrapper/rhq-agent-wrapper.inc *
Overrides or adds Windows Service configuration settings on Windows
*This do not exist out of the box when you install an agent but you can create and configure it when appropriate. Most people, however, will not need this.

If you change the support scripts, the agent-auto-update will keep your files and reuse them in your new agent. If you change the launcher scripts, the agent-auto-update will first back them up but then overwrite them with the new scripts found in the update.

4.4.3. Configuring the Agent using support scripts

Do not rely on any shell environment variables that existed at the time you started the agent. If, for example, you started the agent from within a command line shell and prior to starting the agent you had exported the RHQ_AGENT_HOME environment variable in that shell, the agent will use that value. However, once the agent is shutdown and restarted (as what happens after an auto-update completes) that setting will be lost. If you need to customize the settings that your agent uses at startup (e.g. RHQ_AGENT_HOME or RHQ_AGENT_ADDITIONAL_JAVA_OPTS, to name a few), you should do so by editing the support scripts and setting the values in there. This persists the settings and can therefore be picked up the next time the agent restarts. This provides the added benefit of being able to configure, reconfigure and track/rollback configurations using the agent resource's Configuration tab in the GUI.
You must also ensure that the agent has been fully configured prior to running it the very first time. This means you must either start the agent in a console to answer the setup questions or you must fully configure the agent's configuration file so it can start up without needing additional input from an administrator (i.e. so it does not have to ask the initial setup questions). Once fully configured, you can start the agent as a background daemon thereafter.
You can fully configure the agent by changing the support scripts/configuration files directly. But if you have the agent imported into inventory, you can optionally use the GUI to configure the agent (in effect, we are using JBoss ON to manage JBoss ON).

4.4.4. Installing Keystores and Truststores

If you configured your old agent with SSL such that it uses your own custom keystores and truststores, you must ensure that you install those stores in the following manner:
  1. The keystore files must have the word "keystore" in their filenames. For example, "my-agent-keystore.dat".
  2. The truststore files must have the word "truststore" in their filenames. For example, "my-agent-truststore.dat"
  3. You must have your keystore and truststore files in the agent's <agent-install-dir>/data directory.
If you follow those three simple rules, then your new, updated agent will remain fully secured just as your old agent was. This is because your keystore and truststore files will be copied directly from your old agent to your new agent.

4.4.5. Installing the Agent in a writable directory

During the update process, files will need to be written in the directory where the agent is currently installed. This means that the parent directory of the agent's install directory must be writable by the user that is running the agent. As an example, if the agent's $RHQ_AGENT_HOME (i.e. where the agent is installed) is the directory called /opt/rhq-agent-parent/rhq-agent, the agent will need to write files to the /opt/rhq-agent-parent directory and therefore must have write permissions there.

4.5. Agent auto-update

There is really nothing you have to do when an agent needs to be updated and it is able to auto-update itself. As long as you have configured your server and agents to allow for auto-agent updates (they are configured this way by default), it all happens automatically at the appropriate time.
When a new agent update has been released, it is placed in the server making it available for download by the old, existing agents running in the network. The server will notify the agent that it needs to update as soon as the server detects that the agent is of an older version. Once the agent knows it needs to update itself, it will stop everything it is doing, download the agent update binary, spawn a new Java process that is responsible for applying the update and then kill itself. That new Java process will update the existing agent and then restart the now updated agent. The old, existing agent will be backed up in case it needs to be rolled back (in the event the update fails for some reason).
If you wish to immediately tell the agent to update itself (or just to see if the agent needs to update or is even allowed/enabled to update), use the prompt command "update". Execute the agent prompt command "help update" for more information on that command.

Important

You may notice that it takes several minutes for an agent to fully complete its update. The agent has many threads running in it that need to be gracefully shutdown, it then has to download and verify the integrity of the new agent update binary and then it must actually apply the update before being able to restart. Depending on the situation, it's possible several minutes could elapse before the new agent is fully functional again.

4.6. Manually Upgrading the JON Agent

Important

Upgrading from JON 1.x or the JON 2.0 Beta or CR releases is not supported.

Important

To ensure compatibility with the JON Server, each Agent must be upgraded to the same version of JON as the Server.

Agent Auto-Updating

Since JBoss ON 2.2, agents have the ability to auto-update themselves. So, under most conditions, it isn't necessary to manually upgrade agents. However, if the auto-update fails for some reason or you disabled agent auto-update and you want to manually upgrade yourself, you need to follow the steps below.
If you need to manually upgrade a JBoss ON Agent (i.e. you do not want or cannot utilize the agent auto-ugprade capabilities), then follow these steps below. You will need to follow these steps if you are upgrading the JBoss ON Agent to version 2.2.

Note

All agents must be upgraded at the same time, having agents of different versions in your JBoss ON environment is not supported.
  1. Shutdown your JBoss ON Agent.
  2. If you are running the Agent on Windows and installed the original Agent as a Windows service, uninstall the Windows service:
    cd <old-agent-install-dir>/bin
    ./rhq-agent-wrapper.bat remove
    
  3. Upgrade the JON Server. The JON Server must be upgraded before any Agents are upgraded. Please shut down all agents and wait until all agents show red availability in the GUI before shutting down the server. On completing this procedure, please follow the Upgrading the JON Server instructions to upgrade the JON server.
  4. Restart your upgraded JBoss ON Servers if they are not yet started.
  5. If the current JBoss ON Agent that is to be upgraded is at version 2.2 or newer:
    • Download the agent update binary from the server
    • Copy the agent update binary .jar file into the parent directory where your agent is installed. For example, if your agent is installed in /opt/rhq-agent-parent/rhq-agent, copy the agent update binary .jar file to the /opt/rhq-agent-parent directory.
    • Extract the new JBoss ON Agent from the agent update binary by running the following command:
      java -jar <agent-update-binary.jar> --upgrade
      
      This will tell the agent update binary to extract the JBoss ON Agent distribution and update your currently existing agent that is found in "rhq-agent" subdirectory. At this point, you will have a fully upgraded JBoss ON Agent located in the original "rhq-agent" directory. The old agent has been backed up to the "rhq-agent-old" directory. If the upgrade encounters errors, you'll find log files that contain log messages that can help to diagnose the problem.
  6. If the current JBoss ON Agent that is to be upgraded is older than version 2.2:
    • Follow the instructions to install a JBoss ON Agent here: JBoss ON Agent Installation
    • If you had your own keystore and truststore files, make sure you copy them over from the old agent installation to the new agent installation
    • If you had your own customizations to the agent script files, you will need to merge them into the new agent scripts. Note that you should no longer modify rhq-agent.bat,sh or rhq-agent-wrapper.bat,sh - instead, any environment modifications should be made to rhq-agent-env.bat,sh. This is discussed further here and here, but in short, any customizations you need to make to the agent launcher scripts should be done in rhq-agent-env.bat,sh.
  7. (optional) If you need to reconfigure the JBoss ON Agent before you restart it, read the documentation at JBoss ON Agent Installation that talks about configuring and reconfiguring the agent. Typically, you won't have to reconfigure the agent - the agent will pick up its previous configuration and update it as necessary.
  8. Finally, start the JBoss ON Agent.

Chapter 5. Running the JON Agent

5.1. Running on Windows

5.1.1. Running in a Windows console

To run the JON Agent in a console, execute the rhq-agent.bat script found in the <agent-install-dir>\bin directory of the distribution. You can pass in any of the command line options that are described in Section 5.3, “Command line options”.
The rhq-server.bat script looks for specific environment variables during its execution. These variables can be modified to suit your system requirements. For example, you can point the JON Server at a new JVM or you can define VM options. The comments at the top of the rhq-server.bat file contain a detailed list of these environment variables. You do not have to set any specific variables to get the JON Server to run; sensible defaults are used.

Important

Do not edit the rhq-agent.bat file. If you need to customize the launch parameters of the Agent, either set the environment variables at the command prompt, or edit the values in the rhq-agent-env.bat.

5.1.2. Installing and running as a Windows service

Important

The agent will not ask for the configuration when installing as a Windows service. This means that you need to either pre-configure the agent and pass all required information to the agent via the agent's configuration file, or; run the agent in standard (non-service) mode once as the user the service will run as in order to answer all the setup questions prior to installing it as a service.
If you want to run the JON Agent at boot time, you can install the JON Agent as a Windows Service using the rhq-agent-wrapper.bat file. This script file accepts the following command line options:
  • install: This will install the JON Agent as a Windows Service. When you execute this command you will be prompted for the password of the user that the service will run as. Information on how to define what user the service will run as is outlined below. The Windows Service will be installed to automatically start at boot time. You can change this behavior by modifying the wrapper configuration file as described below.
  • start: This will start the Windows Service and, in turn, start the JON Agent. You must install the Windows Service first in order to be able to start it. Note that you can also start the JON Agent by using the Windows Services Administrative Tool instead.
  • stop: This will stop the Windows Service and, in turn, stop the JON Agent. You need to have the Windows Service installed and started to stop it. Note that you can also stop the JON Agent by using the Windows Services Administrative Tool.
  • remove: This will remove the Windows Service from your Windows operating system. If the service was running, it will be stopped. Once the service is removed, it will no longer be started at boot time and you can no longer start it with the wrapper script.
  • status: This tells you if the service is installed or not; if it is installed, it will tell you if it is currently running.
Two environment variables need to be mentioned here since they are important and have security implications.
  • RHQ_AGENT_RUN_AS: If this variable is set, its value will be the domain/username of the user account that the JON Agent Windows Service will be run as. The format of this variable's value must match that which Windows expects for a user account, specifically the Windows domain name followed by a backslash followed by the username. An example would be MYDOMAIN\john. This variable is used when you install the service via rhq-agent-wrapper.bat install.
  • RHQ_AGENT_RUN_AS_ME: If this variable is defined, it forces the user account to be that of the current user, (specifically ".\ %USERNAME%)". The variable's value does not have to be anything in particular, as long as it is defined to something it will take effect. If this variable is defined, it will override the RHQ_AGENT_RUN_AS environment variable. This variable is used when you install the service via rhq-agent-wrapper.bat install.
Refer to the comments at the top of the rhq-agent-wrapper.bat script file for the full list of variables you can set and the documentation on what they control.

Important

If you do not pass the agent a file containing its required configuration then you must specify one of the two environment variables mentioned above in order for the agent to properly initialize itself. If you specify RHQ_AGENT_RUN_AS_ME then you must have previously run the agent at least once in standard (interactive) mode as the current user before installing it as a service.
If you specify RHQ_AGENT_RUN_AS then you must have previously run the agent at least once in standard (interactive) mode as the user specified by RHQ_AGENT_RUN_AS before installing it as a service.
If neither of the above two environment variables are set, the JON Agent Windows Service will run as the System account. In this scenario the information the agent needs must be specified in its configuration file.

Important

If you specify either RHQ_AGENT_RUN_AS_ME or RHQ_AGENT_RUN_AS then you need to make sure that the selected user can actually start services in Windows. Do this by setting the "Logon as Service" right. Go to the Administrative Tools folder in your control panel. Open the Local Security Policy applet. Expand Local Policy and then click on User Rights Assignment. On the right side of your page, there is a logon as service policy where you can add the selected user.
In addition to setting the wrapper script file's environment variables, you can further configure the JON Agent Windows Service by modifying the service wrapper configuration file. This file is located in the agent's distribution, specifically at bin\wrapper\rhq-agent-wrapper.conf. This configuration file sets some Java Service Wrapper configuration items (Java Service Wrapper is the utility that rhq-agent-wrapper.bat script uses to install and control the Windows Service). If you wish to add, remove or modify some wrapper.* settings, it is recommended that you read the Java Service Wrapper's configuration properties documentation.

Warning

In JON 2.0.1 it is necessary to add the following to the '# Additional JVM parameters' section of bin\wrapper\rhq-agent-wrapper.conf in order for the agent to run correctly as a Windows service. This will be fixed in the JON 2.1 release.
wrapper.java.additional.6=-Djava.endorsed.dirs=%RHQ_AGENT_HOME%/lib/endorsed
A few common settings you may wish to modify are as follows:
  • wrapper.app.parameter.#: These are command line options you can pass to the JON Agent itself. Refer to the JON Agent Guide for information on agent command line options. Note that each individual option and their values must be given their own wrapper configuration property and each property must be placed in numerical order. You cannot change the wrapper.app.parameter.1 or wrapper.app.parameter.2 properties. Start with wrapper.app.parameter.3 and increment up from there. Starting from the last unused number in the sequence, option numbers should increase sequentially. For example:
    wrapper.app.parameter.3=--config
    wrapper.app.parameter.4=C:\my-configs\my-agent-config.xml
    
  • wrapper.java.additional.#: These are additional VM options you can pass directly to the VM (such as -Xmx or -D). As with the wrapper.app.parameter.# properties, you must increment each option in numerical order. You should leave wrapper.java.additional.1 alone unless you want to point to your own log configuration file. You can add, remove or modify others, but make sure you know what you are doing. For example:
    wrapper.java.additional.2=-Xms256m
    wrapper.java.additional.3=-Xmx800m
    wrapper.java.additional.4=-XX:+DisableExplicitGC
    
  • wrapper.ntservice.starttype: By default, this is set to AUTO_START, meaning the JON Agent's service will be started automatically at boot time. If you prefer to force a manual start of the service, you can set this to DEMAND_START.
There are many other Java Service Wrapper configuration properties you can set. For more information, refer to the Java Service Wrapper documentation.

5.2. Running on UNIX®

You can run the JON Agent directly from a console window, or it can be installed as an init.d process so that it starts when the computer boots.

5.2.1. Running in a console

To run the JON Agent in a console, execute the rhq-agent.sh script found in the /bin directory of the distribution. You can pass in any of the command line options that are described in the JON Agent Guide. There are a few environment variables that this rhq-agent.sh script looks for to allow you to run custom operations, such as explicitly pointing to a specific JVM that you want the agent to run with, or defining what VM options you would like to pass in. Please refer to the comments in rhq-agent-env.sh for a list of these environment variables. You do not have to set any specific environment variables to get the JON Agent to run under most circumstances; sensible defaults will be used.

Important

Do not edit rhq-agent.sh. If you need to customize the launch parameters of the Agent, either set the environment variables in your shell that you use to run rhq-agent.sh or edit the values in rhq-agent-env.sh.

5.2.2. Running as a daemon

The JON Agent script rhq-agent-wrapper.sh located in the /bin directory can be used to run the Agent as a daemon. There are two distinct scenarios in which you can run the agent as a daemon:
  1. If you have previously run the agent in a console: Assuming you run rhq-agent-wrapper.sh as the same user you initially ran the agent as, then all you should have to do is to execute the command rhq-agent-wrapper.sh start and the agent will start up in daemon mode.
  2. If you have never previously run the agent: In this case you need the modify rhq-agent-wrapper.sh file, and also ensure that you use a fully configured agent configuration file. This is necessary because you will not be able to answer the questions the agent asks when it first starts up, so these settings need to be specified in the configuration file
    • In the agent-configuration.xml make sure the agent has all the correct settings it needs to connect to the agent. For example:
      <entry key="rhq.agent.name" value="my-agent-name"/>
      <entry key="rhq.agent.server.bind-address" value="jon-server.mycompany.com" />
      
      With these files set up correctly you should be able to start the agent in daemon mode with the command rhq-agent-wrapper.sh start.

Note

You can set environment variables to control the behavior of the agent. See the rhq-agent-env.sh script for the different variables you can set.

Important

Unlike the wrapper script on Windows, the rhq-agent-wrapper.sh script does not utilize the Java Service Wrapper utility.

Important

Solaris Admins: Symbolically linking the rhq agent binaries requires invocation of readlink in rhq-agent-wrapper.sh. However, readlink is not supplied by default in some Solaris installations. Solaris users must either download readlink from a third party provider, such as Sunfreeware, or refrain from symbolically linking the agent binaries when using the wrapper script.

5.2.3. Running with init.d

To run the JON Agent at boot time, you will need to complete some additional steps. You will need either root or sudo root access to perform parts of this installation.

Important

Solaris Admins: Symbolically linking the rhq agent binaries requires invocation of readlink in rhq-agent-wrapper.sh. However, readlink is not supplied by default in some Solaris installations. Solaris users must either download readlink from a third party provider, such as Sunfreeware, or refrain from symbolically linking the agent binaries when using the wrapper script.

5.2.3.1. Configuration

Configuration Parameters
Below are the configuration parameters that appear at the top of the JON Agent script rhq-agent-env.sh located in the /bin directory:
# RHQ_AGENT_HOME=/path/to/agent/installation
# export RHQ_AGENT_DEBUG=true
# export RHQ_AGENT_JAVA_HOME=/path/to/JDK/
# export RHQ_AGENT_JAVA_EXE_FILE_PATH=/path/directly/to/java/executable 
# export RHQ_AGENT_JAVA_OPTS=VM options
# export RHQ_AGENT_ADDITIONAL_JAVA_OPTS=additional VM options 
# PIDFILEDIR=/path/to/writable/directory
Information preceded by the # symbol is commented out and not enabled. To run the JON Agent at boot time you will need to enable some of these parameters by removing the # symbol that appears at the start of each line. The following three parameters are mandatory and must be uncommented:
  1. RHQ_AGENT_HOME=/path/to/agent/installation: This is the directory above your Agent installation's bin directory.
  2. export RHQ_AGENT_JAVA_HOME=/path/to/JDK/: This is the directory above the JDK's bin folder.
  3. PIDFILEDIR=/path/to/writable/directory: A directory writable by the user you run the Agent as. It defaults to /var/run. If /var/run is not writable, use $RHQ_AGENT_HOME/bin. Please note that this is only applicable for versions 2.1.2SP1. and earlier of the JON Agent. Changes have been made in subsequent versions of the agent that will fall back to a writable directory.

Important

If you change # PIDFILEDIR=/path/to/writable/directory, and you are using RHEL, you will need to make a parallel change to the chkconfig "pidfile" location at the very top of the Agent wrapper script.
The following are optional parameters:
  1. export RHQ_AGENT_DEBUG=true: This will turn Agent debug on. Turn this off when finished debugging.
  2. export RHQ_AGENT_JAVA_EXE_FILE_PATH=/path/directly/to/java/executable: You can enable this if you need to specify a java executable. This is an optional configuration item.
  3. export RHQ_AGENT_JAVA_OPTS=VM options: Pass additional VM options. This is an optional configuration item.
  4. export RHQ_AGENT_ADDITIONAL_JAVA_OPTS=additional VM options: Pass additional java options. This is an optional configuration item.

5.2.3.2. Installation

Red Hat Enterprise Linux (RHEL) using chkconfig
The rhq-agent-wrapper script contains standard chkconfig options that can be used by the chkconfig init script management system. You will need to be root or have sudo root access to perform these steps.
  1. Symlink your edited rhq-agent-wrapper.sh file to /etc/init.d/. Example command:
     ln -s /path/to/agent/installation/bin/rhq-agent-wrapper.sh /etc/init.d/rhq-agent-wrapper.sh 
  2. Register rhq-agent-wrapper.sh with the chkconfig system by executing the following command:
     /sbin/chkconfig --add rhq-agent-wrapper.sh 
  3. Enable the Agent service at boot time and have it stop gracefully at shutdown/reboot by executing the following command:
     /sbin/chkconfig rhq-agent-wrapper.sh on 
  4. If you need to disable the Agent boot-time service execute the following command:
     /sbin/chkconfig rhq-agent-wrapper.sh off 
Other UNIX-like operating systems
Please consult your system administrator for instructions on how to install and configure the Agent init script with your init system.

5.3. Command line options

The rhq-agent.bat and rhq-agent.sh scripts can both take the following command line options.
Option Description
-a, --advanced
If setup is needed at startup, the advanced setup is run, rather than the basic
-c, --config=<filename>
explicitly specifies the configuration file the agent is to use for its configuration; if the preferences node in the file is different than "default", you must specify that preference node name via --pref
-d, --daemon
if specified, keyboard input will not be read by the agent. If the --input switch is used to specify a file, it will be processed
-D<name>[=<value>]
overrides a configuration preference with the given name with the given value. This also sets the JVM's system property with the same name/value pair (which does not necessarily have to map to a valid agent configuration preference)
-h, --help
displays the help text for the command line arguments of the agent
-i, --input=<filename>
specifies an input script containing agent prompt commands that the agent should execute upon startup; if not specified, keyboard input will be read
-l, --cleanconfig
Clears out any existing configuration and purges the agent's persisted data so the agent starts with a clean slate. The default configuration file will be loaded if --config is not specified. If you only wish to purge the agent's persisted data, without cleaning its configuration settings, use --purgedata.
-n, --nostart
If specified, the agent will not be automatically started
-o, --output=<filename>
specifies an output file that all non-log output will be written to. This does not affect the log messages - those go to the file as specified in the log4j.xml configuration
-p, --pref=<preferences name>
defines the preferences node name that the agent's configuration is known as. This allows the agent to reuse persisted preferences under this preferences name (and is how you can have multiple configuration sets defined and start the agent with one of them)
-s, --setup
Forces the agent to ask basic setup questions, even if the agent has already been fully configured. If --advanced was also specified, the advanced setup questions will be asked.
-t, --nonative
Forces the agent to disable the native system, even if the agent was configured for it.
-u, --purgedata
Purges the agent's persistent inventory and other data files. This does not erase the agent's configuration settings, use --cleanconfig if you wish to clean the agent's configuration along with purging its persisted data.

5.4. Running embedded in a JON Server

Warning

Running an embedded Agent in production is not recommended and, therefore, it is not currently supported. There are some known issues with managing certain resources with the embedded agent.
There is actually a third way you can run the JON Agent, and that is embedded in a JON Server. For those platforms that are running your JON Server, you have the option to either run a standalone JON Agent normally (as described in the sections above) or you can run the JON Agent embedded in the JON Server itself. This allows you to manage the JON Server platform and all the products running on that platform (including the JON Server itself) by simply starting the JON Server and having it start a JON Agent for you. This feature is to enable testing and demo'ing JON without needing to install and run a separate, standalone agent. It is not recommended to deploy in this way in a production JON environment.
The JON Server disables the Embedded JON Agent by default - you have to explicitly enable the Embedded JON Agent in order to run it. Refer to the section on Running the Embedded JON Agent in the JON Server Guide for information on how you can enable and configure the Embedded JON Agent.
The Embedded JON Agent implicitly runs in daemon mode, which means it has no command line interface. Its log file is located in $RHQ_SERVER_HOME/logs/embedded-agent.log.

5.5. Using the Agent plugin

The agent plugin is used to manage the agent itself; you can monitor, configure and control the agent from the JBoss ON Server GUI.
The agent can monitor its internal metrics, allowing you to track its behavior and ensure that it is operating properly. Like any other resource managed by JBoss ON, you can define alerts on this metric data which can be triggered when an agent starts behaving in an abnormal way. In addition, you can control the lifecycle of the agent by restarting it or shutting it down.

Important

If you shutdown the agent, you will not be able to use the GUI to start it back up again because there will be no agent to receive the start command. Only shutdown the agent when you have no intention of starting it up again or if you are willing to manually log onto the agent's machine and start it from a shell console. If you need to reboot the agent, use the restart operation provided by the agent's launcher child service.

5.5.1. Inventorying the Agent Server and its services

When you import the agent, one main top-level resource and several child services will be added to your inventory. Some of these services are platform-specific; that is, some services are only deployed in the agent when running on Windows or when running on UNIX. All of the resources provide specialized monitoring, configuration and control of the agent. These resources are described in more detail below.
Here is the typical resource hierarchy for the agent running on Linux and Windows:

5.5.2. Top-level resource

The agent's main top-level resource provides monitoring, configuration and control functionality for your agent. You can configure the agent's internal components. These configuration settings correspond to the preferences defined in the agent-configuration.xml file and are persisted on the agent machine as Java Preferences. You can control the agent by executing the several available operations on the agent server resource.

Important

The operations provided by the agent server resource normally do not affect the agent process in a direct way. This means you cannot kill the agent VM process, configure the agent VM's environment variables or configure the agent VM's JRE options. You can perform these tasks by working with the agent's child services, as defined below. The only exception is the shutdown operation, which will actually kill the agent process if the agent is running as a background daemon service. If it is running in a console window, the agent VM will not completely exit, allowing the user to continue to execute prompt commands from the console prompt.

5.5.3. Common services

These are the services that are common to all agents. Whether running on Windows or UNIX, all agents have these services as children to the main agent server resource.

5.5.3.1. Agent measurement subsystem

This service provides data on the measurement collection and reporting components in the agent. It provides information about the measurements being collected and sent to the server.

5.5.3.2. Agent JVM

The Agent JVM service is actually considered a child server that provides additional child services; this gives you more fine-grained monitoring and management of the VM that is running the agent and all its plugins, which includes the classloader, threading and memory management subsystems, among others.

5.5.3.3. Agent environment setup script

Whether on Windows or UNIX, all agents have an environment setup script. However, they are named differently based on the platform: rhq-agent-env.bat for Windows, rhq-agent-env.sh for UNIX. This child service allows you to configure the environment variables that are set when you start the agent launcher script.

5.5.3.4. Agent plugin container

This provides a view into the embedded plugin container, also known as the "PC". The PC runs within the agent and handles the deployment of all management plugins and infrastructure necessary to run those plugins. This resource provides management data related directly to the PC.

5.5.4. Java service wrapper launcher (Windows-only)

If the agent is running on Windows, the agent will have a child service that controls the Java Service Wrapper. This is a third party library used to install and run the agent as a Windows Service. There is one main configuration for the Java Service Wrapper, and it is read-only (rhq-agent-wrapper.conf). These define the base set of configuration settings necessary for the agent to start and operate properly. However, there are two additional groups of configuration settings that you can use to customize your agent's environment. The Environment group defines/overrides environment variables that are used by the main configuration. These are in addition to the environment variables defined by the common Environment Setup Script. The Includes group defines overrides for any of the wrapper configuration settings defined in the main configuration. Typically, you never have to touch anything in any of these Java Service Wrapper configuration groups since the defaults should work. However, there are times when you might have to turn on some additional debug configuration settings or pass in new/modified VM options to the agent VM. In those cases, this configuration is where you would put those things for those agents running on Windows as a Windows Service.

5.5.5. Agent launcher script (UNIX-only)

If the agent is running on UNIX, the agent will have a child service that controls the agent. If the agent is running as a background daemon process that was spawned by the launcher script, you can use this service to stop it or restart it. There is no additional configuration for this service; this launcher script is configured by the Environment Setup Script.

5.5.6. Rebooting the Agent and its Java VM

There may be times when you would like to recycle the internal components of the agent, but not take down the entire agent VM process. However, there may be other times when you want to reboot the entire agent, including the Java virtual machine. This section describes some of the siutations in which you would want to perform either of these options and how to go about performing these tasks.

5.5.6.1. Restart internal Agent components

The main internal agent component is called the Plugin Container. It loads all agent plugins and manages the lifecycle of all the plugins it loads. If you want to restart the internal plugin container (for debugging purposes, or to re-initialize all plugins to get them to start over, as if just booting up, for example), you can execute the operation called Restart Plugin Container located in the OPERATIONS tab of the main agent server resource. If you want to restart all internal components of the agent, but not completely reboot the Java virtual machine, you can execute the operation called Restart. This is useful, for example, if you want to reboot the internal communications subsystem. Restarting the agent in this manner will, however, also restart the plugin container.
In either case described above, the original agent Java virtual machine remains intact. That is to say, the Java virtual machine process never exited as it remains running. The operating system will still see the agent as the same process as it did before, with the same process ID. The agent's VM never changed its heap or non-heap allocations and the original VM options that were set when the agent was first started remain in effect.

5.5.6.2. Restart Agent Java VM

What if you want to completely kill the agent JVM and restart it? You normally want to do this if you changed the launcher environment configuration in such a way that would affect the agent VM. You would then need the VM to restart to pick up those changes (if you want to add a new or different -Xmx heap memory setting for the agent, for example).
You can do this by going to the OPERATIONS tab of the agent server's child service that corresponds to the agent launcher. On Windows, this is the Agent Java Service Wrapper Launcher service; on UNIX, this is the Agent Launcher Script service. Both of those resources provide an operation called Restart that shuts down the agent process and restarts it. This will effectively kill the original agent VM process and then ask the operating system to restart another one.

Note

To configure the agent VM's settings, you can edit the configuration found in the Configuration tab of the "Environment Setup Script" child service.

Chapter 6. Initial auto-discovery and import

Important

Please see Chapter 4, JON Agent Installation for instructions on how to install the JON Agent.
After installing the JON Server, you will be directed to the JON GUI Console after you click the Done! Click here to get started! message on the installer page. Alternatively, you can point your browser to http://localhost:7080.

6.1. Login

The default username/password is rhqadmin/rhqadmin. Once you login and install the license you can change the default password in the Administration section of the application.

6.2. Installing a license

Before you can access the JON Dashboard, you must first install the JON license you downloaded earlier.
Browse to the location of the saved license file and click OK to install the license. If you do not have a JON license, see Chapter 1, Download JBoss ON. If you need to update the license at a later time, you can do this in the Administration section of the application.

6.3. Initial auto-discovery and import

Once the agent has been started it will run discovery scans for resources on the machine and report them back to the JON Server. Before resources can be managed from the JON Server, they must be imported into inventory. Resources are imported through the Auto-Discovery portlet on the dashboard page. Select the newly discovered platform and its first level child servers (they will be automatically selected by default) and click the Import button.
Once the platform has been imported, the JON Agent will run a more detailed discovery scan. This discovery will check for child servers and services of the imported resources. Once complete, the Browse Resources link at the top of the page will display a summary of the JON Server inventory.
The JON Agent will continue to report back to the JON Server with data about the resources in inventory, including availability and metric data. At this point, the installation and initial setup are complete. You may begin by either exploring the JON Server user interface (UI) or continue to the Features section for more information on what JBoss ON has to offer.

Chapter 7. High Availability configurations

IN JON 2.1.0 support for JON High Availability (HA) was introduced. The goal of HA is to support multiple JON servers configured against a single database repository. JON Agent load can then be partitioned amongst the available Servers. Failover will occur for Agents whose Server becomes unreachable. A multi-server HA configuration provides fault tolerance and improved scalability.

Note

A demo that illustrates how to install a new Server into an HA Server cloud can be found in the What is JBoss ON Guide. This demo also shows you agent failover in action. The Affinity Groups demo illustrates how Affinity Groups is used to group Servers and Agents together.

7.1. When should I define an HA configuration?

In many circumstances it may be satisfactory to run a single-server configuration. However, if your environment satisfies one or more of the following criteria you may want to consider a multi-server approach:
  • Your Agent report processing is not meeting requirements, including the following:
    • Metrics
    • Alert generation
    • Event generation
    • Resource availability
  • You have a geographically distributed environment characterised by the following:
    • Multiple data centers
    • Logical grouping of agents to servers
  • There is a high agent load:
    • 100+ Agents (this number can vary widely depending on Server strength)
    • The JON server is having trouble processing the agent load (this is a better indicator)

7.2. HA infrastructure impact

Every JON 2.x configuration is an HA configuration. Even a single-server configuration can be considered a 1-Server HA Cloud, in that it can be managed via the HA Administration Console. For example, List Servers, List Agents, Setting Maintenance Mode, etc. are all applicable to a single-server and accessible from the ADMINISTRATION page of the JON GUI. Since a single-server is a 1-Server cloud, it easily adapts to an increase in cloud size.
In general, JON Servers can be added or removed from the HA Cloud at any time. A single-server environment can be turned into a multi-server environment by running the JON Server installer on a second machine, configured for the same database, and defining a new Server (unique Server Name and public Endpoint).

7.2.1. Database impact

Although JON Servers can be added to the HA Server Cloud with relative ease, it should be done cautiously due to the potential impact on the back-end database. Each JON Server limits its concurrent database connections but there is no restriction on the Cloud itself. This means that adding a second server can double the potential database connections, even if the number of Agents remains the same. The increase is linear as servers are added.
Each JON server instance has built-in mechanisms for limiting the load it will put on the database. In JON 2.2.0 that number out-of-the-box is 55 simultaneous connections. Each JON server can use less connections (that number is largely based on how many agents are connected to it and how much data it needs to process concurrently), but the limits guarantee that they will each never use more than 55 connections to the database. For example, a 2-Server configuration would require that the database be able to handle 110 connections.
The JON Administrator should work closely with the Database Administrator (DBA) to ensure an adequate configuration. In general, a large scale JON configuration requires DBA planning to handle not only connections, but to provide a database with reasonable data distribution and space allocation. HA impact is just another aspect to take into consideration.

Note

An HA configuration does not necessarily imply a large number of JON Agents. It may be the case that a relatively small JON implementation may be in place, with only a few JON Agents. Those Agents, however, may need high availability and therefore failover servers are required. In this case, the backing database will still have a high number of potential connections, but in reality will not reach that limit.

7.2.2. Server and Agent endpoints

In a multi-server HA configuration it is possible for any agent to try to connect to any server. It is critical that every JON Agent be able to resolve the Endpoint Address set for every Server in the HA Server cloud. When defining the server in the installer, it is important that the Endpoint Address be public to the degree that the Agent population can reach the server via the defined address and port, or secure port if configured in that fashion.

Note

The Server endpoint information can be updated via the HA Administration pages in the JON GUI.
Conversely, an agent connecting to a server must provide an endpoint reachable by the Server in order to allow for the necessary two-way communication.

7.3. When should I use Affinity?

By default, agent load is distributed evenly amongst the servers in the cloud. Balance can change in failover situations, but in general, by default, agent load will be evenly distributed when all agents and all servers are running.
This is fine when it is unimportant which Agents connect to which Servers. But there are use cases where it may be desirable to create stronger bonds between specific Agents and Servers. This is accomplished by defining JON HA Affinity Groups. Agents prefer connecting to Servers in the same Affinity Group. Affinity group assignment is optional and any given Agent or Server can participate in at most one Affinity Group.
The following sections outline some situations that may benefit from Affinity Group assignment.

7.3.1. Physical efficiency

In general, if it is clear that certain agent-server connections will run more efficiently than others, then defining affinity to prefer those connections makes sense. This could include Servers and Agents co-located in the same data center, or other geographic grouping, or various network topology scenarios.

7.3.2. Logical efficiency

It may not be the case that certain agents and servers will run more efficiently by talking to one another, but that there are other, organizational reasons to group agents and servers together. For example, Administration responsibilities, business units, etc.

7.3.3. Warm backup

It may be the case that certain machines should not be assigned agent load unless specifically needed for fail over purposes. In this case you would have all agents assigned affinity to a subset of the available servers, leaving some servers without any associated agents in normal operation.

7.3.4. Affinity behavior

The following points should be understood about Affinity Group behavior:
  1. Affinity is strong:
    • JON will satisfy Affinity Group preference and then apply load balancing.
  2. An Agent will fail over to an available Server in its affinity group before it fails over to a non-affinity Server.
  3. Affinity is not guaranteed:
    • Although strong, affinity can be broken if no affinity server is available.
    • All Servers will be present in an Agent's failover list.
  4. JON attempts to distribute load evenly within the Affinity Group.

7.4. How do I move to an HA installation?

After deciding that a JON High Availability strategy is appropriate for your needs, you should do two things to prepare for your installation or upgrade:
  1. Ensure your database is ready to support the load and connections. See Section 7.2.1, “Database impact” more information.
  2. Determine your affinity strategy. See Section 7.3.4, “Affinity behavior” for more information.

    Note

    Affinity assignments can be added or removed at any time, but it is useful to consider your initial approach, even if it confirms that affinity assignments are unnecessary.
From an installation and upgrade perspective an HA environment does not require different steps, and actually can be moved to incrementally.

7.4.1. Moving to HA from JON 2.0.0/2.0.1

The recommended approach to move from a JON 2.0.0 or 2.0.1 environment to a JON HA environment is as follows:

7.4.2. Server requirements

  1. All HA Servers must be running compatible versions of JON.
    • Unless specifically noted, this means the exact same version of JON.
  2. All HA Servers must be uniquely named.
    • This string is defined during Server installation.
  3. Each HA Server must define a unique endpoint that is resolvable by all JON Agents running against the HA Server Cloud.
    • This address/port or address/secure port combination is defined during installation. It can be an alias as long as that alias is resolvable by all JON Agents. This requirement exists because any given JON Agent may be running against any JON Server at a given time. It is not the case that the JON Server defined in the JON Agent's initial configuration (from the Setup Questions, for example) will be the JON Server that the Agent is told to connect to. It is the JON Server that will initially be contacted by the JON Agent for registration purposes.

Note

Note that the first JON Server installed will become the initial member in the HA Server Cloud. This means that a single-server installation can also be thought of as a 1-Server HA Cloud and therefore has the same Server requirements. The JON GUI HA Administration Console pages are still usable to inspect or manage your environment.
Perhaps more importantly this allows other servers to be added at any time, even while other Servers are running. Conversely, HA Servers can be removed from the Cloud at any time.
HA Servers communicate solely via the database and therefore it is not required that their endpoints be visible to each other. No direct server to server communication is ever made.
To install or upgrade a JON Server, please follow the instructions in Section 7.4.2, “Server requirements”.

7.4.3. Agent requirements

  1. All Agents must be running versions compatible with the JON Servers.
    • Unless specifically noted, this means the exact same version of JON.
  2. All Agents must be configured initially to contact a JON Server.
    • This can be any JON Server in the JON HA Server Cloud.
  3. All Agents must define an endpoint that is resolvable by any JON Server in the JON HA Server Cloud.
    • This is because any given JON Agent may be running against any JON Server at a given time. It can be an alias as long as that alias is resolvable by all JON Servers. This requirement exists because any given JON Agent may be running against any JON Server at a given time. It is not the case that the JON Server defined in the JON Agent's initial configuration (from the Setup Questions, for example) will be the JON Server that the Agent will be served by. It is only the JON Server that will initially be contacted by the JON Agent for registration purposes.
To install or upgrade a JON Agent, please follow the instructions in Chapter 4, JON Agent Installation.
Since HA environments typically involve many agents, it may be useful to perform pre-configured Agent installations to avoid having to answer initial Setup Questions interactively.

7.5. How do I manage my HA installation?

Even a 1-Server installation can take advantage of certain HA management capabilities. After adding a second server, or more, it will be useful to become comfortable with the HA management features available in JON. You should perform the following steps when building up an HA Server Cloud:
  1. Familiarize yourself with the HA Administration Console (HAAC)
    • You will use these administration pages in the JON GUI to manage and monitor your HA environment. These pages are available even when running a single server so it should be easy to get comfortable with their use.
  2. Add your desired JON Server(s)
    • Increasing the size of of your HA Server Cloud is done by running the installer on the desired Server node(s). To do this follow the JON Server Installation Guide. Use the List Servers page to examine your installed Servers.
  3. Add your desired JON Agents
  4. Update your Server Affinity Group membership
    • If you have decided to use Affinity Groups to enable Agent-Server connection preferences then you can use the List Affinity Groups page to create your desired Affinity Groups and associate your Servers as desired. Note, you can also associate a Server with an Affinity Group via the Installer, using the Advanced Settings option.
  5. Update your Agent Affinity Group membership
    • If you have decided to use Affinity Groups to enable Agent-Server connection preferences then you can use the List Affinity Groups page to associate your Agents as desired.
  6. Inspect your topology
    • After making these changes there can be a delay, typically less than one hour, before the Agent population establishes a steady state that is connected to the expected Primary servers. You can inspect the topology and view the Agent's migration using the HAAC pages.
  7. Optionally, test failover
    • When you are satisfied that your HA Server Cloud is servicing Agents as expected you may want to test Agent Failover. To do this you can set Servers to Maintenance Mode via the List Servers page.

Chapter 8. Command Line Interface (CLI) installation

8.1. Remote Client and CLI installation

The JBoss ON Command Line Interface (CLI) can run on either a Windows or UNIX platform and does not require the execution of a special installer program. The CLI requires a Java6 runtime environment (JRE) and, by default, should run out-of-box. You will need to ensure the following:
After the installation you can run the CLI from the console. Go to Section 8.4, “Running the JON CLI” for more information.

8.2. Download the Remote Client binary

The JBoss ON Server is bundled with a Remote Client distribution called rhq-client-<version>.zip. The bundled version is the version that should be used with that server. Earlier or later Remote Clients will not be compatible.
The Remote Client distribution can be downloaded directly from any running JON Server. You can do this by navigating to Administration > Downloads and then go to the Command Line Client Download subsection.

8.3. Unzip the binary

Copy the binary .zip file into the directory where you want the agent to be installed and then execute the following command:
unzip rhq-client-<version>.zip
You are now ready to run the CLI.

8.4. Running the JON CLI

8.4.1. Introduction

The JBoss ON CLI is a standalone Java application that uses the Java Scripting API that requires Java 6 or later. The CLI enables you to further integrate JBoss ON into your environment by allowing you to interact with JBoss ON programmatically. Java 6 ships with the Rhino JavaScript engine, and as such, JavaScript is the supported scripting language in the CLI. A large subset of JBoss ON functionality is exposed in the CLI.

8.4.2. Running the JON CLI

The CLI is a shell and interpreter that allows you to interactively execute statements. This can be useful for prototyping. Scripts stored in files can also be executed, providing you with the capability to develop more fully automated solutions.

8.4.2.1. Running on Windows

The CLI can be run within a console window.
Setting Environment Variables
The rhq-cli-env.bat file, located in the <cli-install-dir>\bin directory, contains a detailed list of the environment variables that the CLI requires to run. For most variables, sensible defaults are used and therefore do not need to be edited. It is important, however, to set the correct path to your Java installation.

Important

Do not edit rhq-cli.bat. If you need to customize the launch parameters of the JBoss ON CLI, edit the environment variable values in rhq-cli-env.bat.
Running in a Windows Console
To run the JBoss ON CLI, execute the rhq-cli.bat located in <cli-install-dir>\bin directory of the installation. You can pass in any of the Section 8.4.3, “JON Command line options”.
The rhq-cli.bat script looks for specific environment variables during its execution. These variables can be modified to suit your system requirements. For example, you can point the JBoss ON CLI at a new JVM or you can define VM options.

Note

Do not edit the rhq-cli.bat file. If you need to customize the launch parameters of the JBoss ON CLI, either set the environment variables at the command prompt, or edit the values in rhq-cli-env.bat. The comments at the top of the rhq-cli-env.bat file contain a detailed list of these environment variables. You do not have to set any specific variables to get the JBoss ON CLI to run; sensible defaults are used.

8.4.2.2. Running on Linux/UNIX

The JBoss ON CLI can be run from within a shell of a terminal window.
Setting Environment Variables
The rhq-cli-env.sh file, located in the <cli-install-dir>/bin/ directory, contains a detailed list of the environment variables that the JBoss ON CLI requires to run. For most variables, sensible defaults are used and therefore do not need to be edited. It is important, however, to set the correct path to your Java installation. Before starting the JBoss ON CLI, ensure that either the RHQ_CLI_JAVA_HOME or the RHQ_CLI_JAVA_EXE_FILE_PATH variable is set appropriately.

Important

Do not edit rhq-cli.sh. If you need to customize the launch parameters of the JBoss ON CLI, edit the environment variable values in rhq-cli-env.sh.
Running in a Shell
To run the JBoss ON CLI, run the rhq-cli.sh script located in the <cli-install-dir>/bin directory of the installation. You can pass in any of the JBoss ON CLI Command Line Options.

8.4.3. JON Command line options

Both rhq-cli.bat and rhq-cli.sh accept the following command line options:
Option Description
-h, --help
Displays the help text of the command line options of the CLI.
-u, --user
The username used to log into the JBoss ON server.
-p, --password
The password used to log into the JBoss ON server.
-P
Displays a password prompt where input is not echoed backed to the screen.
-s, --host
The JBoss ON server against which the CLI executes commands. Defaults to localhost.
-t, --port
The port on which the JBoss ON server is accepting HTTP requests. Defaults to 7080.
-c, --command
A command to be executed. The command must be encased in double quotes. The CLI will exit after the command has finished executing.
-f, --file
The pathname of a script to execute. See Executing Scripts below for additional information on running scripts.
-v, --version
Displays CLI and JON server version information once connected to the the CLI.
--transport
Determines whether or not SSL will be used for the communication layer protocol between the CLI and the JBoss ON server. If not specified the value is determined from the {port} option. If you use a port that ends in 443, SSL will be used. You only need to explicitly specify the transport when your JBoss ON server is listening over SSL on a port that does not end with 443.
--args-style
Indicates the style or format of arguments passed to the script. See the Executing Scripts below for additional information.

8.4.3.1. Examples

The following examples illustrate the various ways in which the CLI can be invoked.
$ rhq-cli
This is the simplest way to start the CLI without any options. You will be connected to the CLI but not logged into the JON server. Although most of the JON functionality is not available until you are logged in, you are still able to execute scripts and script commands.
$ rhq-cli -u rhqadmin -p rhqadmin
You will be connected to the CLI and logged in with the specified credentials on the JON server running on localhost.
$ rhq-cli -u rhqadmin -p rhqadmin -s 192.168.1.100 -t 70443
You will be connected to the CLI and logged into the JON server on 192.168.1.100 that is listening on port 70443. Because the port number ends with 443, the CLI will attempt to communicate with the JON server over SSL using the sslservlet transport strategy.
$ rhq-cli -u rhqadmin -P
You will be connected to the CLI and prompted for a password.
$ rhq-cli -u rhqadmin -u rhqadmin -c "pretty.print(ResourceTypeManager.findResourceTypesByCriteria(new ResourceTypeCriteria()))" > resource_types.txt
This connects you to the CLI, logs you into the JBoss ON server running on localhost, executes the command in quotes, and redirects the output to the file resource_types.txt. This is a detailed command so we will explain each section. We will start with ResourceTypeManager.findResourceTypesByCriteria(new ResourceTypeCriteria()). This invokes the findResourceTypesByCriteria operation on ResourceTypeManager. A new ResourceTypeCriteria object is passed as the argument. Nothing has been specified on the criteria object so all resource types will be returned. We will now look at the pretty.print(...) portion. An implicit object made available to commands and scripts by the CLI, pretty is useful for outputting objects in a readable, tabular format, designed with enhanced capabilities for domain objects. This single command provides a nicely formatted, text-based report of the resource types in your inventory.
$ rhq-cli -f my_script.js
This connects you to the CLI and executes the script file, my_script.js. The CLI will terminate immediately after the script has finished executing.

8.4.4. Built-in commands

As previously stated, the CLI is built using the Java Scripting API that was introduced in Java 6. The majority of commands and scripts that you write will be executed by the underlying script engine. Built-in commands, however, are native Java code and are not executed by the underlying script engine. This is similar to other interpreters like Python where some modules are implemented in C and built into the interpreter. This distinction is important because built-in commands cannot be processed by the script engine. Objects, however, that provide hooks into the built-in commands, are exposed to the scripting environment.
The following section provides a list of built-in commands and the corresponding options and arguments they accept. Each of the built-in commands includes a help option. Using the short form, <command> -h, will display the command syntax and a short description. Using its long form, <command> --help, will display the command syntax and a more detailed description.

8.4.4.1. login

Syntax
login <username> <password> [host] [port]
Description
This will log into a JON server with the specified username and password. The host name (or IP address) and port can be specified. The host name defaults to localhost and the port defaults to 7080. The transport argument is optional. It determines whether or not SSL will be used for the communication layer protocol between the CLI and the JBoss ON server. If not specified, the value is determined from the port argument. If you use a port that ends in 443, SSL will be used. You only need to explicitly specify the transport when your JBoss ON server is listening over SSL on a port that does not end in 443.

8.4.4.2. logout

Syntax
logout
Description
This will logout from the JBoss ON server, but does not exit from the CLI interpreter.

8.4.4.3. quit

Syntax
quit
Description
This will exit the CLI interpreter.

8.4.4.4. exec

Syntax
exec <statement> | [-s<indexed|named>] -f <file> [args]
Description
This executes a statement or a script with the specified file name. A statement wraps onto multiple lines using backslashes. See below for an example.
Option Description
-f, --file
The file name of the script to execute. A valid file name must be specified.
-s, --style
Indicates the style or format of arguments passed to the script. It must have a value of either indexed or named. See the section, Executing scripts for additional details.
Examples
Please see the following examples below.

Example 8.1. executing a single statement

localhost:7080> exec var x = 1

Example 8.2. executing a multi-line statement

localhost:7080(rhqadmin)> for (i = 1; i < 3; ++i) { \
localhost:7080(rhqadmin)>    println(i); \
localhost:7080(rhqadmin)> }
1
2
		
localhost:7080(rhqadmin)>

Example 8.3. executing a named script without arguments

localhost:7080(rhqadmin)> exec -f myscript.js

Example 8.4. executing a named script with arguments

localhost:7080(rhqadmin)> exec -f myscript.js 1 2 3


Example 8.5. executing a named script with named arguments

localhost:7080(rhqadmin)> exec --style=named -f myscript.js x=1 y=2 y=3

8.4.4.5. record

Syntax
record [-b | -e] [-a] -f
Description
Records user input commands to a file.
Option Description
-b, --start
Specify this option to start recording.
-e, --end
Specify this option to stop recording.
-a, --append
Appends output to the end of a file. If not specified, output will be written starting at the beginning of the file.
-f, --file
The file where output will be written.

8.4.5. Working with the CLI

The CLI provides two modes of operation: interactive and non-interactive. In interactive mode, you can execute an individual statement. In non-interactive mode, you can execute an entire batch of commands in the form of a script. Non-interactive mode provides the capability to automate tasks such as collecting metrics on managed resources or executing a scheduled operation. Interactive mode provides a rich environment for prototyping, testing, learning and discovering features of the CLI, and more. In the following sections, we will take a detailed look at some of the features and functionality of the CLI. Unless stated otherwise, the following features and functionality discussed are available in both modes of operation.

8.4.5.1. The API

This is a first pass at the services to be exposed.

8.4.5.2. Implicit variables

There are a number of variables that are bound to the script context. Some of these variables, however, are available only after you have logged in. Implicit variables are listed in the table below. The last column in the table indicates whether login is required for accessing the variable.
Variable Type Description Access requires login
subject
Represents the current, logged in user. For security purposes, all remote service invocations require the subject to be passed; however, the CLI will implicitly pass the subject for you.
YES
pretty
org.rhq.enterprise.client.TabularWriter
Provides for tabular-formatted printed and handles converting objects, particularly domain objects in the packages under org.rhq.core.domain, into a format suitable for display in the console.
NO
unlimitedPC
NO
pageControl
Used to specify paging and sorting on data retrieval operations
NO
exporter
org.rhq.enterprise.client.Exporter
Used to export output to a file. Supported formats are plain text in tabular format and CSV.
NO
ProxyFactory
org.rhq.enterprise.client.utility.ResourceClientProxy.Factory
NO
rhq
org.rhq.enterprise.client.Controller
Exposes methods of the same name for the built-in commands - login, logout, quit.
NO
scriptUtil
org.rhq.enterprise.client.utility.ScriptUtil
Provides methods that can be useful when writing scripts.
NO
AlertManager
Provides an interface into the alerts subsystem.
YES
AlertDefinitionManager
Provides an interface into the alerts definition subsystem.
YES
AvailabilityManager
Provides an interface into the measurement subsystem that can be used to determine resources' availability.
YES
CallTimeDataManager
Provides an interface into the measurement subsystem for retrieving call-time metric data.
YES
ChannelManager
Provides an interface into the content subsystem for working with channels.
YES
ConfigurationManager
Provides an interface into the configuration subsystem.
YES
DataAccess
Provides an interface for executing user-defined queries.
YES
EventManager
Provides an interface into the events subsystem.
YES
MeasurementBaselineManager
Provides an interface into the measurement subsystem for working with measurement baselines.
YES
MeasurementDataManager
Provides an interface into the measurement subsystem for working with measurement data.
YES
MeasurementDefinitionManager
Provides an interface into the measurement subsystem for working with measurement definitions.
YES
MeasurementScheduleManager
Provides an interface into the measurement subsystem for working with measurement schedules.
YES
OperationManager
Provides an interface into the operation subsystem.
YES
ResourceManager
Provides an interface into the resource subsystem.
YES
ResourceGroupManager
Provides an interface into the resource group subsystem.
YES
ResourceTypeManager
Provides an interface into the resource subsystem for working with resource types.
YES
RoleManager
Provides an interface into the security subsystem for working with security rules and roles.
YES
SubjectManager
Provides an interface into the security subsystem for working with users.
YES
SupportManager
Provides an interface into the reporting subsystem for getting reports of managed resources.
YES

8.4.5.3. Auto-imported packages

In the Java programming language, you do not have to import classes in the java.lang package. They are automatically made available. Classes in other packages, however, have to be explicitly imported. In the CLI, there are a number of classes from various packages that you are likely to use on a routine basis. For example, if you want to query resources, then you are likely to use the class org.rhq.core.domain.critiera.ResourceCriteria. The CLI automatically imports under and including org.rhq.core.domain. This means that while we can write:
	
var criteria = new org.rhq.core.domain.criteria.ResourceCriteria();
var resource = new org.rhq.core.domain.resource.Resource();
This can be more succinctly written as:
var criteria = new ResourceCritiera();
var resource = new Resource();

8.4.5.4. Auto completor

A feature common to Unix and Linux shells like Bash, is the auto-completion of commands. If you type the first few letters of a command and press the tab key, the shell will complete the command for you. The CLI provides an Auto Completor which is similar to that provided in Unix and Linux shells.

Important

The Auto Complete is only available in interactive mode.
The Auto Completor provides tab-completion for any object bound in the script context. If you want to view a listing of objects bound in the script context, press the tab key. You can also use the Auto Completor to view a list of available methods for a particular object. At the CLI prompt, type ResourceManager. followed by the tab key and you should see something like the following:
rhqadmin@localhost:7080$ ResourceManager.
	
findResourceComposites    findResourceLineage       findResourcesByCriteria   getResource
resource                  toString                  uninventoryResources
Then enter ResourceManager.f followed by the tab key. You should see something like the following:
rhqadmin@localhost:7080$ ResourceManager.findResource
	
findResourceComposites    findResourceLineage       findResourcesByCriteria
rhqadmin@localhost:7080$ ResourceManager.findResource
In the example above, the command prompt displays findResource because the Auto Complete was able to complete the line to that point. It then displays a list of available methods. The Auto Completor can be extremely useful for discovering objects and their methods.

8.4.5.5. Script arguments

A feature common to most programming languages is the ability to pass arguments to the program to be executed. In Java, the entry point into a program is a class's main method, and it takes a String array as an argument. That array holds any arguments passed to the program. Similarly, arguments can be passed to CLI scripts. Arguments passed to a script can be accessed in the implicit args array:

Example 8.6. Handling Script Arguments

if (args.length > 2) {
    throw "Not enough arguments!";
}
		
for (i in args) {
     println('args[' + i '] = ' + args[i]);
}

Important

The args variable is only available when executing a script in non-interactive mode or with exec -f.
In addition to the traditional style of indexed-based arguments, you can also pass named arguments to a script, as demonstrated in the following example:
rhqadmin@localhost:7080$ exec -f echo_args.js --args-style=named x=1, y=2

Example 8.7. echo_args.js

for (i in args) {
     println('args[' + i '] = ' + args[i]);
}
println('named args...');
println('x = ' + x);
println('y = ' + y);

This produces the following output:

Example 8.8. echo_args.js

args[0] = 1
args[1] = 2
named args...
		
x = 1
y = 2

Be aware of the following:
  • You have to explicitly specify that you are using named arguments via the --args-style option
  • The values of the named arguments are still accessible via the implicit args array
  • The named arguments, x and y, are bound into the script context as variables

8.4.5.6. Criteria searching

All of the managers define operations for retrieving data. Most of the managers define criteria-based operations for data retrieval. The Criteria API provides a flexible framework for fine-tuned query operations. The criteria classes reside in the org.rhq.core.domain.criteria package. The following examples illustrate how to work with the criteria APIs.
Basic Criteria
rhqadmin@localhost:7080$ var criteria = new ResourceCriteria()
rhqadmin@localhost:7080$ var resources = ResourceManager.findResourcesByCriteria(criteria)
In this example, the first thing to note is that you do not have to import the ResourceCriteria class. In the Auto-Imported Packages section we discussed how the org.rhq.core.domain.criteria package is automatically imported.
Next, take note of the method, findResourcesByCriteria. All of the criteria-based query operations are of the form findXXXByCriteria. This basic criteria search is translated into the following JPA-QL query:
SELECT r
FROM Resource r
WHERE ( r.inventoryStatus = InvetoryStatus.COMMITTED )
As we can see from the JPA-QL query, this will fetch all committed resources in our inventory.
Sorting
Next we will modify our criteria so that the resources are sorted by plugin.
rhqadmin@localhost:7080$ criteria.addSortPluginName(PageOrdering.ASC)
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
This criteria is translated into the following JPA-QL query:
SELECT r
FROM Resource r
WHERE ( r.inventoryStatus = InventoryStatus.COMMITTED )
ORDER BY r.resourceType.plugin ASC
To add sorting, you call criteria.addSortPluginName(). To add sorting to any criteria, you can use methods of the form addSortXXX(PageOrdering order).
Filtering
Now you can add some filtering to refine your search results. You can filter on the resource type name:
rhqadmin@localhost:7080$ criteria.addFilterResourceTypeName('JBossAS Server')
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
To add filtering to any criteria, you will use methods of the form addFilterXXX(). The resulting JPA-QL query will appear as follows:
SELECT r
FROM Resource r
WHERE ( r.inventoryStatus = InventoryStatus.COMMITTED
AND LOWER( r.resourceType.name )  like 'JBossAS Server' ESCAPE '\\' )
This code is all that is required to retrieve all JBoss servers in your inventory. You can further refine your criteria to find JBoss servers that have been registered by a particular agent:
rhqadmin@localhost:7080$ criteria.addFilterResourceTypeName('JBossAS Server')
rhqadmin@localhost:7080$ criteria.addFilterAgentName('localhost.localdomain')
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
This generates the following JPA-QL query:
SELECT r
FROM Resource r
WHERE ( r.inventoryStatus = InventoryStatus.COMMITTED
AND LOWER( r.agent.name )  like 'localhost.localdomain' ESCAPE '\\' )
Fetching Associations
After you have retrieved resources, you can now view child resources. Consider the following example code:
rhqadmin@localhost:7080$ criteria.addFilterResourceTypeName('JBossAS Server')
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
rhqadmin@localhost:7080$ resource = resources.get(0)
rhqadmin@localhost:7080$ if (resource.childResources == null) print('no child resources')
This code will print the string no child resources, even if the JBoss server has child resources. The reason for this is that lazy loading is used throughout the domain layer for one-to-many and many-to-many associations. If you are not familiar with lazy loading, please see Some explanations on lazy loading. Since child resources are lazily loaded, you need to specify in the criteria that they should be fetched.
rhqadmin@localhost:7080$ criteria.addFilterResourceTypeName('JBossAS Server')
rhqadmin@localhost:7080$ criteria.fetchChildResources(true)
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
rhqadmin@localhost:7080$ resource = resources.get(0)
rhqadmin@localhost:7080$ if (resource.childResources == null) print('no child resources'); else pretty.print(resource.childResources)
As you see with the call to criteria.fetchChildResources(true), all criteria methods that specify that a particular lazy association should be fetched are of the form, fetchXXX(). The output from the above is as follows:
rhqadmin@localhost:7080$ if (resource.childResources == null) print('no child resources'); else pretty.print(resource.childResources)
id  name                                                 versio resourceType
--------------------------------------------------------------------------------------------------------
222 AlertManagerBean                                            EJB3 Session Bean
222 SchedulerBean                                               EJB3 Session Bean
222 AlertDefinitionManagerBean                                  EJB3 Session Bean
222 AlertConditionConsumerBean                                  EJB3 Session Bean
222 PartitionEventManagerBean                                   EJB3 Session Bean
222 AlertTemplateManagerBean                                    EJB3 Session Bean
223 RHQ Server Group Definition / DynaGroups Subsystem          RHQ Server Group Definition / DynaGrou
222 DiscoveryTestBean                                           EJB3 Session Bean
222 PerspectiveManagerBean                                      EJB3 Session Bean
222 ResourceAvailabilityManagerBean                             EJB3 Session Bean
222 AlertDampeningManagerBean                                   EJB3 Session Bean
218 localhost.localdomain  Embedded JBossWeb Server 2.0. 2.0.1. Embedded Tomcat Server
222 ResourceGroupManagerBean                                    EJB3 Session Bean
222 FailoverListManagerBean                                     EJB3 Session Bean
222 ResourceFactoryManagerBean                                  EJB3 Session Bean
222 AccessBean                                                  EJB3 Session Bean
222 MeasurementTestBean                                         EJB3 Session Bean
223 wstools.sh                                                  Script
222 EventManagerBean                                            EJB3 Session Bean
222 ContentSourceManagerBean                                    EJB3 Session Bean
223 RHQ Server Alerts Engine Subsystem                          RHQ Server Alerts Engine Subsystem
222 AlertConditionManagerBean                                   EJB3 Session Bean
222 ResourceMetadataManagerBean                                 EJB3 Session Bean
222 ResourceManagerBean                                         EJB3 Session Bean
222 GroupDefinitionExpressionBuilderManagerBean                 EJB3 Session Bean
222 MeasurementViewManagerBean                                  EJB3 Session Bean
218 JmsXA Connection Factory                                    ConnectionFactory
222 ResourceTypeManagerBean                                     EJB3 Session Bean
223 JBoss Cache subsystem                                1.0    JBossCacheSubsystem
218 NoTxRHQDS Datasource                                        Datasource
222 DataAccessBean                                              EJB3 Session Bean
222 AlertConditionCacheManagerBean                              EJB3 Session Bean
222 MeasurementProblemManagerBean                               EJB3 Session Bean
222 ServerManagerBean                                           EJB3 Session Bean
222 OperationHistorySubsystemManagerBean                        EJB3 Session Bean
222 ClusterManagerBean                                          EJB3 Session Bean
222 run.sh                                                      Script
...
The output will vary depending on what you have inventoried. These are the child resources of the JON server we have used in these examples. The JPA-QL query that is generated appears as follows:
SELECT r
FROM Resource r
LEFT JOIN FETCH r.childResources
WHERE ( r.inventoryStatus = InventoryStatus.COMMITTED
AND LOWER( r.resourceType.name )  like 'JBossAS Server' ESCAPE '\\' )

8.4.5.7. Displaying output

8.4.5.7.1. TabularWriter
TabularWriter provides for tabular-formatted output. In addition to formatting, it handles converting objects, particularly domain objects in the packages under org.rhq.core.domain, into a format suitable for display in the interactive console. Let's look at an example to illustrate the utility of TabularWriter:
	
rhqadmin@localhost:7080$ criteria = ResourceCriteria()
rhqadmin@localhost:7080$ criteria.addFilterResourceTypeName('service-alpha')
rhqadmin@localhost:7080$ criteria.addFilterParentResourceName('server-omega-0')
rhqadmin@localhost:7080$ resources = ResourceManager.findResourcesByCriteria(criteria)
id    name            version resourceType
----------------------------------------------------------------------------------------------------------------------
11373 service-alpha-8 1.0     service-alpha
11374 service-alpha-1 1.0     service-alpha
11375 service-alpha-0 1.0     service-alpha
11376 service-alpha-4 1.0     service-alpha
11377 service-alpha-2 1.0     service-alpha
11378 service-alpha-3 1.0     service-alpha
11379 service-alpha-5 1.0     service-alpha
11380 service-alpha-9 1.0     service-alpha
11381 service-alpha-6 1.0     service-alpha
11382 service-alpha-7 1.0     service-alpha
10 rows
The TabularWriter instance that is bound in the script context under the name pretty is implicitly invoked to display the results of ResourceManager.findResourcesByCriteria. The returned resources are displayed in very readable, tabular format. Now let's look at the output if we do not use TabularWriter.
rhqadmin@localhost:7080$ println(resources)
PageList[Resource[id=11373, type=service-alpha, key=service-alpha-8, name=service-alpha-8, version=1.0],
Resource[id=11374, type=service-alpha, key=service-alpha-1, name=service-alpha-1, version=1.0], 
Resource[id=11375, type=service-alpha, key=service-alpha-0, name=service-alpha-0, version=1.0], 
Resource[id=11376, type=service-alpha, key=service-alpha-4, name=service-alpha-4, version=1.0], 
Resource[id=11377, type=service-alpha, key=service-alpha-2, name=service-alpha-2, version=1.0], 
Resource[id=11378, type=service-alpha, key=service-alpha-3, name=service-alpha-3, version=1.0], 
Resource[id=11379, type=service-alpha, key=service-alpha-5, name=service-alpha-5, version=1.0], 
Resource[id=11380, type=service-alpha, key=service-alpha-9, name=service-alpha-9, version=1.0], 
Resource[id=11381, type=service-alpha, key=service-alpha-6, name=service-alpha-6, version=1.0], 
Resource[id=11382, type=service-alpha, key=service-alpha-7, name=service-alpha-7, version=1.0]]
For display purposes, this output is not very usable. Let's look at one more example in which we display a single entity.
rhqadmin@localhost:7080$ pretty.print(resources.get(0))
Resource:
	          id: 11373
	        name: service-alpha-8
	     version: 1.0
	resourceType: service-alpha
Notice that the formatting is different when displaying a single object. Lastly, it should be pointed out that only a subset of the properties in the Resource class are displayed. TabularWriter determines the properties to display via the Summary annotation. If a field of an entity has the @Summary annotation, TabularWriter will include it in the output.
8.4.5.7.2. Exporter
An implicit script variable that can assist with writing output to a file is exporter. It uses TabularWriter; however, in addition to writing plain text in a tabular format, it also supports CSV-formatting. First, we will look at an example of exporting to a file as plain text:
rhqadmin@localhost:7080$ exporter.setTarget('raw', 'output.txt')
rhqadmin@localhost:7080$ exporter.write(resources)
Note that you do not have to worry about file IO operations like opening or closing the file. Exporter handles the IO operations for you. Now we will export the results to a CSV file:
rhqadmin@localhost:7080$ exporter.setTarget('csv', 'output.csv')
rhqadmin@localhost:7080$ exporter.write(resources)

8.4.6. Proxy

Resource Proxies are custom objects generated in the CLI that simplify interacting with the remote and domain APIs. Proxies are obtained through the ProxyFactory by calling getResource() and providing a resource identifier:
var rhelServerOne = ProxyFactory.getResource(10001);
From there, the proxy exhibits the behavior of a custom local interface for that resource. It exposes methods to invoke operations. There will be properties for viewing metrics that connect to the live metric system. There is also support for viewing and editing current configurations and updating and deploying content.
rhqadmin@localhost:7080$ ProxyFactory.getResource(10001)
ResourceClientProxy_$$_javassist_2:
		OSName: MacOSX
           OSVersion: 10.5.7
	  architecture: i386
	  contentTypes: {}
	   createdDate: Thu Aug 06 16:59:32 EDT 2009
	   description: Mac OS X Operating System
	    freeMemory: 120.2MB
	 freeSwapSpace: 984.7MB
	      hostname: ghinklembp.local
	            id: 10001
	          idle: 40%
	  measurements: [Used Memory, Wait Load, Total Memory, System Load, 
			OS Name, Free Memory, Hostname, Architecture, Idle, 
			Total Swap Space, Used Swap Space, User Load, OS Version, 
			Free Swap Space]
	  modifiedDate: Thu Aug 06 16:59:32 EDT 2009
		  name: ghinklembp
	    operations: [org.rhq.enterprise.client.proxy.ResourceClientProxy$Operation@234a98fa,
	                org.rhq.enterprise.client.proxy.ResourceClientProxy$Operation@7681572f]
	  resourceType: Mac OS X
	    systemLoad: 0%
	   totalMemory: 4GB
	totalSwapSpace: 4GB
	    usedMemory: 3.9GB
	 usedSwapSpace: 3GB
	      userLoad: 50%
	       version: MacOSX 10.5.7
	      waitLoad: 0%

8.4.6.1. Running operations

To run an operation on the proxy, call the bound method representing that operation. For example:
	
platform.viewProcessList();
	
agent.updateAllPlugins();
	
jbossas.restart();

8.4.6.2. Configurations

The resource proxy offers access both to the connection properties and the configuration of the resource itself. Configuration data for a resource such as a datasource can be viewed and edited live.
datasource.getResourceConfiguration()
The proxy also integrates with the interactive resource configuration editing wizard to update configurations via a question and answer session. This wizard asks for the values of each configuration property and has some special keystrokes to simplify editing:
  • Pressing return will select the default or existing value for a property.
  • Pressing ctrl-d is equivalent to setting the unset checkbox in the configuration UI.
  • Pressing ctrl-k will exit out of the wizard at any point.
  • Pressing ctrl-e will display the help description for the current property.
At the end of the wizard you can choose to review, save, cancel or edit the configuration again. Once you select save the configuration will be updated live on the remote resource.
datasource.editResourceConfiguration()

8.4.6.3. Content

The content features for resources that support backing content, such as JBoss Application Server EAR and WAR resources, will be exposed to the proxy allowing you to retrieve or update the content behind that deployed resource.
The following will retrieve the EAR file from the agent and download it to the CLI, saving it as a file with the same name as the deployed file. You can also pass in a relative or fully qualified path to save the content to:
myCustomEAR.retrieveBackingContent(null);
You can send a new version of the content via the following command by passing in the name of the file to send. This will deploy a new version of your custom application content and automatically increment the internal version tracking:
myCustomEAR.updateBackingContent('CustomEAR_v2.ear');

Chapter 9. JON Best Practices

The following sections provide some best practices when setting up and running your JBoss ON environment.

9.1. Best practices for managing JBoss Application Servers

9.1.1. DynaGroups

You can use the DynaGroups feature by setting up GroupDefinitions. This will automatically group resources with certain attributes. You can set an update period to automatically recalculate the groups.
JBoss AS Clusters
You can use the template definition for JBoss AS Clusters in the system to automatically create a separate group for each set of JBoss nodes in a cluster.
Down Resources
This creates a group of down resources using the All Down Resources template.

9.1.2. Useful alerts

JBossAS Server: JVM Free Memory
Alert on this getting < 10MB to indicate the application server may be about to run out of memory.
Data Source: Available Connections
Alert on this getting too low, to know when your applicaton is about to run out of free database connections
Tomcat Connector: Error Count
Alert on this being > 0 to find out when your web applications are responding with errors to requests. See also 'Web Application (WAR): Errors while processing' for information on individual web applications.
Tomcat Connector: Maximum Request Time
Alert on this going above 15 seconds, to indicate your web applications are running slow. See also 'Web Application (WAR): Max. Servlet Resp. Time' for information on individual web applications.
Tomcat Connector: Request count per Minute
Monitor this to see how load on your application changes during the day/week. See also 'Web Application (WAR) : Requests served per Minute' for information on individual web apps.
EJB3 Session Bean : Method Invocation Time
Collect the minimum, maximum, and average invocation times for each of the methods exposed by an EJB. Useful to monitor responsiveness of critical backend systems, e.g. EJBs calling into DBs or EISs.
JMS Queue : Messages in Queue
Create an alert with a dampening rule on this metric to see if messages are backing up in the queu. Look at JMS Topic : All Message Counter' for the equivalent for topics.
Hibernate : viewQueries operation
Schedule this operation and review its output regularly to determine the most used/expensive SQL in your hibernate app.

9.1.3. JBoss Application Server setup

Management invoker security
  • Always secure invoker endpoints.
Configuration
  • Symlinking is common for externalizing configs (mostly xml). It makes upgrading easier.
  • Parameterized configurations by further externalizing environment variables how to leverage check config against init files.
Application management
  • Use deploy dir or farm, sometimes symlinks to common deploy dir company wide.
  • Turn autodeploy off so that accidental touches to files do not cause redeploys.
  • Application upgrade - side-by-side or rolling restart, scripted solution and coordinate with load balancer.

9.1.4. Common issues

  • running out of file handles
  • http threads, work thread, conn pools
  • incomplete deployments / undeployments
  • running out of sockets
  • lock contention / jvm deadlock detection
  • run out of jdbc connections
  • slow memory leaks
  • load (for physical machine) and latency
  • lockup detection

9.1.5. Debugging issues

  • determine final/last state just before failure: heap, seg fault, remote restart, system was rebooted, etc
  • have "test" application that you can hit in an isolated format to verify top-to-bottom health of connection from http entry to db and back
  • delta analysis - highlight metric spikes to signify unusual or unanticipated things
  • rate of change analysis - acceleration of metric value deltas over time load simulation against some app

9.1.6. Log tracking

  • look for error counts, number of times so-and-so occurred
  • transaction rollback messages
  • any network timeout issues, any lower-level issues such as syslog messages, kernel panic, daemons dying log / list of remote JMX commands sent to app server

9.2. Best Practices for managing JBoss Tomcat Servers

The Tomcat plugin manages Tomcat servers provided by Apache or JBoss EWS. The earliest Tomcat5 version tested with this plugin is 5.5.23. The earliest Tomcat6 version tested with this plugin is 6.0.18.

9.2.1. Enabling JBoss ON monitoring of your Tomcat Server

Important

Java5 or higher is required when running Tomcat.
The JBoss ON Agent should automatically discover Tomcat Server processes and they can be imported into inventory. But the resources will have DOWN availability until they are properly JMX enabled.
Users of JBoss EWS have hooks already provided and should start by reading the RUNNING.txt in their Tomcat root directory. That includes instructions similar to those below but where possible the scripts and configuration files have already been seeded with comments and commented out settings and/or declarations.
Tomcat management is performed by utilizing the JMX interface provided by Tomcat. As such, remote JMX management needs to be enabled. To do this add the following section near the top of your bin/startup.sh (adapt for windows as necessary):
# JON/Jopr Management
#
# To enable Tomcat for JON/Jopr management it is necessary to enable remote JMX access to the Tomcat server:
# 1) Choose an unused and accessible portNum
# 2) Update as necessary and uncomment the following lines:
#
# JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.port=<port>"
# JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"
#
# 3) Follow the steps for secure access as described in the following url. Update as necessary
#    and uncomment the following lines based on authenticated (recommended) or unauthenticated access.
#      http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html.
#
# JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.access.file=<access-file-path>"
# JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.password.file=<password-file-path>"
# JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=false"
# export JAVA_OPTS
Make the appropriate edits and (re)start the Tomcat server.
If you have configured JMX with authentication you will need to edit the Tomcat Server resource and add valid credentials. To do this navigate to the already imported Tomcat Server Resource. Continue to the Inventory->Connection subtab. Edit the definition, setting the user/password to valid settings. On successful update the Server availability should go green within a short period of time.

9.2.2. Enabling response time metrics for your Tomcat Server

Before enabling Response Time metric collection in your Tomcat Server Monitoring Schedules you need to add a servlet filter to your application's web.xml and add the filter jar, provided by JBoss ON.
It is typically case that the filter is deployed across all applications, in which case you can add the following to the global web.xml in the <tomcat>/conf directory.
Add the following filter and filter mappings. Uncomment when you want to activate the filter.

9.2.2.1. Filter

		

<!-- The RHQ Response Time Metric Filter. -->
<!-- This filter gathers response time information for processed requests -->
<!-- for all hosted applications. This requires that the                  -->
<!-- rhq-rtfilter-X.X.X.jar also be placed properly in the Tomcat         -->
<!-- deployment. The jar file can be downloaded from the RHQ Server.      -->
<!-- Using the GUI, navigate to the Administration page. Then click       -->
<!-- 'Download Agent Connectors'.                                         -->
<!-- Download connector-rtfilter.zip. Unzip to extract the jar.           -->
<!--                                                                      -->
<!-- Optional parameters                                                  -->
<!--                                                                      -->
<!--   chopQueryString  Filter directly chops off the query parameters    -->
<!--                    from the URL? Default is true.                    -->
<!--                                                                      -->
<!--   logDirectory     Directory to which the logs are written.          -->
<!--                                                                      -->
<!--   logFilePrefix    Prefix to written logfile names.                  -->
<!--                                                                      -->
<!--   dontLogRegEx     Patterns that should not be logged.               -->
<!--                                                                      -->
<!--   matchOnUriOnly   Apply the dontLog pattern to the URI only?        -->
<!--                                                                      -->
<!--   timeBetweenFlushesInSec                                            -->
<!--                    Seconds between auto flushes of the logfile.      -->
<!--   maxLogFileSize   The maximum allowed size, in bytes, of the        -->
<!--                    logfiles if a logfile exceeds this limit, the     -->
<!--                    filter will truncate it; the default value is     -->
<!--                    5242880 (5 MB)                                    -->
<!--   vHostMappingFile Name of vhost mapping file. This properties file  -->
<!--                    must be in the Tomcat process classpath.          -->
<!--                                                                      -->
<!-- NOTE: When enabling, also uncomment the RhqRtFilter filter below.    -->
<!-- NOTE: When enabling, uncomment the start and end tags.               -->
	
<!-- start tag (also uncomment end tag)
    <filter>
	<filter-name>RhqRtFilter</filter-name>
	<filter-class>org.rhq.helpers.rtfilter.filter.RtFilter</filter-class>
-->
		     
<!-- Optional parameters. Note these typically remain commented. Also,
     some of these values may be configurable on the relevant            
     RHQ Server resource, via the RHQ GUI.
	
     	<init-param>
	    <param-name>chopQueryString</param-name>
	    <param-value>true</param-value>
	</init-param>
	<init-param>
	    <param-name>logDirectory</param-name>
	    <param-value>/tmp</param-value>
	</init-param>
	<init-param>
	    <param-name>logFilePrefix</param-name>
	    <param-value>localhost_7080_</param-value>
	</init-param>
	<init-param>
	    <param-name>dontLogRegEx</param-name>
	    <param-value></param-value>
	</init-param>
	<init-param>
	    <param-name>matchOnUriOnly</param-name>
	    <param-value>true</param-value>
	</init-param>
	<init-param>
	    <param-name>timeBetweenFlushesInSec</param-name>
	    <param-value>73</param-value>
	</init-param>
	<init-param>
	    <param-name>flushAfterLines</param-name>
	    <param-value>13</param-value>
	</init-param>
	<init-param>
	    <param-name>maxLogFileSize</param-name>
	    <param-value>5242880</param-value>
	</init-param>       
-->
			  
<!-- end tag (also uncomment start tag)
   </filter>
-->

9.2.2.2. Filter mapping


	
<!-- The mapping for the RHQ Filter                                       -->
<!-- To limit to only certain applications various filters can be         -->
<!-- defined. This will perform RT metric collection for all              -->
<!-- applications.                                                        -->
<!--                                                                      -->
<!-- Note: Uncomment only when also enabling the RhqRtFilter above.       -->
<!--
    <filter-mapping>
	<filter-name>RhqRtFilter</filter-name>
		    <url-pattern>/*</url-pattern>
	    </filter-mapping>
	    -->

9.2.2.3. Jar file placement

The connector-rtfilter.zip file contains two jar files:
  1. The rtfilter JAR
  2. The commons logging JAR
Placement of the jar files depend on the version of Tomcat.

Table 9.1. Jar file placement

Tomcat Version Instructions
5.5
Only one jar file is required:
Copy the rtfilter JAR file to <tomcat5>/common/lib>
6
1. Copy the rtfilter JAR file to <tomcat6>/lib>
2. Copy the commons logging JAR to <tomcat6>/lib>

9.3. Best practices for remote clients

9.3.1. Deciding on a Remote Client Approach

JBoss ON provides several options for remote clients:
  • CLI (command line interface)
  • Java Client
  • WebService (WS) Client

9.3.1.1. Command line interface (CLI)

The CLI is a powerful scripting tool for interacting with the JBoss ON Server. It offers Rhino Javascript as the scripting environment. Rhino is a Java-backed implementation of Javascript. Communication with the JBoss ON Server is through JBoss Remoting.
This is the recommended approach if you are looking for a remote script-based client.

9.3.1.2. Java client

The power of the CLI is available in a programmatic way for remote clients written in Java. From the Java program one creates a Client object and works with it as if in the CLI:
{
	Client joprClient = new Client(...)
	jopClient.getResourceManager().findResources(...)
	...
}
This is the recommended approach for a programmatic remote client, if Java is an option. Communication with the JBoss ON Server is through JBoss Remoting.

9.3.1.3. Web services

For programmatic remote clients written in languages other than Java the JBoss ON Remote API methods are available as Web Services. Or, if for some reason a Java client does not want to utilize the Client object as mentioned above, a Web Service Java client is a possibility.

Index

F

feedback
contact information for this manual, We need feedback