Red Hat Training
A Red Hat training course is available for Red Hat JBoss Operations Network
Release Notes
JBoss Operations Network 3.1
for release information
Copyright © 2012 Red Hat, Inc.
June 12, 2012
Abstract
These release notes contain important information about new features, known issues, and other technical notes available at the time that JBoss Operations Network 3.1 was released.
These release notes contain feature information, changed support, and structural changes in JBoss Operations Network 3.1.
JBoss Operations Network provides centralized control, configuration, and visibility to JBoss applications and related infrastructure. Broad and detailed monitoring, versioned application deployment, auditable configuration management, and controlled software patching through JBoss ON reduce the complexity, improve the reliability, and decrease the cost of large scale JBoss deployments.
This version of JBoss Operations Network contains new features, enhancements, and bug fixes. It is recommended that all JBoss Operations Network users upgrade to JBoss Operations Network 3.1.
1. New Features in JBoss Operations Network 3.1
This version of JBoss ON introduces both new features and feature enhancements that improve JBoss ON's performance for managing resources.
1.1. New: JBoss Resource Plug-ins for JBoss Enterprise Application Platform 6
JBoss Enterprise Application Platform (EAP) 6 emphasizes performance, speed, and deployment flexibility. In particular, there are two key features in EAP 6:
- A new architecture for managed domains, which introduce the idea of centralized domain controllers, local system host controllers, and low-level managed servers, with configuration defined in concentric levels from the domain down to local applications.A classic standalone server still exists that parallels the JBoss EAP 4 and 5 servers, which can be clustered or configured for high availability.
- Classes are modular, so classes can be loaded and unloaded as needed, lowering the EAP footprint and improving system performance.
JBoss ON manages and monitors all JBoss EAP 6-related resources: domain and host controllers, standalone servers, mod_cluster services, Infinispan services, HornetQ servers, and web contexts and applications.
The new architectures possible with JBoss EAP 6 can make simple deployment and management decisions very complex — and that's where a centralized, consistent management framework is invaluable. JBoss ON helps bring order to a diverse and intimidatingly complex EAP 6 deployment.
Working in tandem with EAP 6's native management console and CLI tools, JBoss ON offers a platform-centric perspective on the hierarchy and relationship between elements within a JBoss EAP 6 domain or a cluster. JBoss ON provides a sense of structure on otherwise unrelated EAP 6 resources and makes it easier to manage the entire deployment.
The benefits of monitoring JBoss EAP 6 through JBoss ON include:
- Define and manage application workflows. Test an application on a single server and then promote the application to a server group or deploy it on a new domain, a standalone server, or a cluster through a single provisioning path.
- Manage multiple standalone instances and EAP 6 domains from a single management console.
- Gain complete, chronological visibility into availability, performance metrics, auditable operations, events, and configuration changes.
- Organize EAP 6 resources in a platform-oriented hierarchy that offers a deployment-focused perspective on servers, services, and child applications.For example, a datasource in the EAP 6 console has three separate resources — the datasource definitions and then two separate views for metrics and configuration. In JBoss ON, this is consolidated into a single view for the datasource.
- Automatically detect runtime environment variables and launch script arguments. These start options can be viewed and edited through JBoss ON, which makes them easy to apply to other managed resources.
Note
If you installed the tech preview plug-in for JBoss AS 7.1, that resource must be uninventoried and re-added using the new EAP 6 plug-in. It cannot be migrated.
A full list of changes, current work items, and issues for the JBoss ON EAP 6 plug-in is available in Bugzilla #707223.
Documentation for managing EAP 6 standalone servers, domain controllers, manager servers, and related services is covered in the "How to Manage JBoss Servers." As always, complete reference material for the EAP 6 plug-ins is available in the Complete Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.2. Enahanced: More Control over the Start Script Environment
Part of the JBoss EAP resource connection settings define start script values that the JBoss ON agent uses when running a start operation on the EAP resource. In previous versions of the JBoss EAP plug-in, the agent used a bind address and the Java home directory to define the script environment.
In JBoss ON 3.1, both the JBoss EAP/AS 5 plug-in and the new JBoss EAP 6 plug-in perform more detailed discoveries of the start script at the time they discover the EAP resources. This improved discovery scan attempts to identify more information about the start script environment, and then recreate those settings in the settings it uses for the start operation. The discovery scan detects:
- The start script used, including custom start scripts.
- Required environment variables set in the
run.conffile or the parent process (a subset of possible or all environment variables). - Any arguments passed with the start script itself.
- EAP 6 only. What user the script is running as. If the EAP server and JBoss ON agent users are different, JBoss ON assigns a
sudocommand as a prefix command to use with the start script, such assudo -u jboss -g jboss.
The discovery process handles missing information gracefully. If it cannot detect some information or if it is not set, the agent simply ignores that parameter.
Note
On EAP 5, if the Script Arguments field is empty, then the bind address and Java home directory are still used for invoking the start script.
If any script arguments are set, then the bind address are Java home values are ignored.
1.3. Enhanced: More Responsive, Accurate, and Granular Availability Monitoring
Availability — monitoring whether a resource is running (up) or not — is one of the most basic and most critical aspects of infrastructure management.
In previous versions of JBoss ON, availability was polled for every resource, by the agent, on a set interval defined in the agent configuration (5 minutes by default). This rigid schedule led to some inefficiencies: periodic CPU spikes, perceived collection lags from slow reporting times, and gaps of several minutes between availability checks for a resource which could miss resource cycling. Additionally, the agent's own availability was tied to its sending the availability report to the server, so every agent "heartbeat" actually contained a heavyweight report for every resource in the agent's inventory.
JBoss ON 3.1 introduces a new approach to availability scheduling and monitoring that both improves overall JBoss ON performance and provides better accuracy and detail in availability reports.
Full documentation on availability monitoring is located in the availability chapter of "Setting up Monitoring, Alerts, and Operations."
Prioritized Availability Collection and Availability Schedules
One of the biggest changes to availability is allowing availability to be scheduled at different intervals for different resources and resource types, much like a metric schedule. This allows important resources to be prioritized higher than others.
Default availability schedules have been reset in three ways:
- Server availability is collected every minute.
- Service availability is collected every 10 minutes.
- Some dependent child resources have availability checking disabled entirely, and they default to the parent availability state. These resource can have availability checking enabled if desired.
Along with checking availability on schedules for individual resources and types, the availability scan itself has been changed. Previously, scans occurred every five minutes, and the agent checked every resource at the same time. Now, the agent runs a scan every 30 seconds and checks only a subset of resources, whichever ones (according to their collection schedule) are ready to be checked. On the initial scan, the resources are spaced out more or less evenly; after that, the resource is checked at the scan interval (30 seconds) plus its own configured interval. For example, a serve has an interval of one minute, so its availability is checked every 90 seconds — the time to run the agent scan plus the availability schedule.
Having a more frequent availability scan which checks only a subset of resources prevents CPU spikes and improves agent performance, while actually increasing how frequently the availability of critical resources is checked.
More Availability States
A resource can be in several different states: UP, DOWN, UNKNOWN, and DISABLED.
UP, DOWN, and UNKNOWN states have always existed, but now there are slight difference in how UNKNOWN and DOWN are used. UNKNOWN is now used for child resource when the agent goes down — because the children could be running fine. Their status is, legitimately, unknown rather than down.
Disabled is a new, administrative state. This is a signal to the JBoss ON server to ignore whatever availability changes come in. The agent still detects and sends reports, but the server ignores them. This allows resource to be cycled for maintenance or upgrades without sending spurious alert notifications or having to manually disable alert definitions.
The additional availability states for a resource have an impact on how availability is displayed for compatible groups.
Note
Availability states are evaluated "top down." If a resource is down, disabled, or unknown, then all of its children are immediately assumed to be in that state, as well.
Table 1. Group Availability States
| If the Resource States Are .... | ... the Group State Is ... |
|---|---|
| Empty Group (Unknown) | Empty |
| All Red (Down) | Red (Down) |
| Some Down or Unknown | Yellow (Mixed) |
| Some Orange (Disabled) | Orange (Disabled) |
| All Green (Up) | Green (Up) |
New Availability Alert Conditions
Previously, JBoss ON could send an alert on an availability state change, whether a resource went up or down. Alerting based on availability has been enhanced in two ways:
- A new not up option has been added for availability state changes. An alert can now be fired if a resource comes up, goes down, or goes into another (not up) state.
- A new alert condition has been added for availability duration. If a resource changes state and then stays in that state for a given amount of time, an alert is fired.
Changes to the avail Prompt Command
The avail command for the agent command prompt can send an availability report that contains only changed availability states or all current states, regardless of changes. This has always been true.
However, in previous version, running the
avail command checked every single resource in the agent's inventory. Now, it only checks the next scheduled resource by default. Checking the complete inventory requires the --force option.
New Agent Heartbeat
The agent heartbeat (the signal to the server that the agent is running) has been removed from general availability reporting. Instead, the agent sends a lightweight ping to the server.
Additionally, the agent quiet time — the amount of time the server waits between pings before assuming the agent is offline — has been lowered from 15 minutes to 5 minutes. This helps the JBoss ON server recognize more quickly when the agent goes offline and to backfill resource states.
The agent availability used to be tied only to its availability report. Even if the agent shut down, the server still waited the entire quiet time interval before marking the agent as offline. The agent now informs the server when it shuts down gracefully and all the platform child resource are immediately backfilled to UNKNOWN states, rather than waiting the full quiet time.
1.4. New: Export Reports to CSV
The JBoss ON web UI has a main tab, Reports, which defines several queries for different elements of the JBoss ON inventory and resource management, including an inventory breakdown by resource type, platform performance, and lists of configured definitions for alerts and drift.
For additional analysis and manipulation, these reports can now be exported to a comma-separated values (CSV) spreadsheet. Some reports define filters, such as filtering fired alerts by priority or by date. When a filtered report is exported, all of the filters are preserved, and only the displayed information is included in the CSV file.
For information on JBoss ON reporting, see the reporting section of "Initial Setup for the Resource Inventory, Groups, and Users."
1.5. Tech Preview: REST Interface
A new REST interface is being designed for JBoss ON. The REST interface is accessible for the JBoss ON server at:
http://server.example.com:7080/rest/
This REST API is under active development. While some interfaces are probably stable (such as resources and metrics methods), some interfaces are still being designed and could change significantly as development progresses (such as the alerts methods). As with all preview technologies, expect significant changes in subsequent releases of the REST API. Compatibility is not guaranteed, so use caution when developing clients using the tech preview version of the JBoss ON REST API.
The REST API is available for developers at http://docs.redhat.com/docs/en-US/JBoss_Operations_Network/3.1/html/Rest_API/index.html.
1.6. Enhanced: Additional Monitoring Metrics
New monitoring metrics have been added:
- A new trait has been added to datasource and connection factory monitoring metrics for JBoss AS 5 servers which determines whether the connection is available. This tests the connection settings every 15 minutes, by default.
- A new count for the number of open files for a JVM process has been added. Operating systems commonly put limits on how many open files a single process can have, which can cause otherwise performant JVM processes to fail when they hit that limit. Tracking the open files metric can help signal administrators when the JVM is approaching that limit.This metric is only relevant on Linux and Unix systems; it is not available on Windows systems.
- New actual free memory and actual total memory metrics have been added for platforms. These memory counts include cache and buffer settings in the available memory calculation.
- A new maximum thread count metric has been added to the embedded JBossWeb Connector resource for JBoss AS 5 servers.
Metrics for all support resource plug-ins are available in the Complete Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.7. New: Access Control Permission to View User Details
In previous versions of JBoss ON, any user with global permissions could view other user accounts (even though creating or editing user accounts was limited to users with Security permissions).
A new global permission has been added that explicitly grants the right to view user details such as the user name, full name, and contact information for a JBoss ON user (while excluding any view of the user's role assignments). Roles without the View User permission are implicitly denied the ability to view any user details.
Only roles with the Security permission, which is roughly equivalent to a superuser, have the ability view role assignments for users.
1.8. Enhanced: Better Discovery and Management for Custom JVM Resources
In previous versions of JBoss ON, the Generic JMX agent plug-in only automatically discovered JMX resources if they were using JMX remoting services and the agent could connect to the agent over the JMX remoting ports. In large deployments, it is not realistic to manage many open JMX remoting ports.
The Generic JMX agent plug-in in JBoss ON 3.1 has been enhanced to discover any JMX server running on the platform, as long as it is configured in one of two ways that the agent detects:
- JMX remoting is enabled, and the remoting port is set through the command line, so that the agent can read the options. This is the method that has always been used to discover JMX servers.
- The JVM process is accessible through the Java attach API and has a resource key system property set in its command line. The key allows the agent to identify the appropriate Java process and provide a unique resource identity.In this case, the agent and the JVM must be running as the same system user, or the agent cannot attach to the process.
Any JVM process explicitly discovered by another resource plug-in, such as a Tomcat server, is still preferentially discovered by the specific resource plug-in. Customer or generic JVM processes are discovered by the Generic JMX plug-in as a top-level[1] JVM server.
If a JVM server should be ignored by the Generic JMX plug-in, then an excludes rule can be defined in the JBoss ON agent configuration that instructs the agent to ignore any matching JVM processes.
Managing JMX servers and generic JVMs is covered in "How to Manage JBoss Servers."
1.9. New: Resource Plug-in to Monitor and Manage JBoss Data Grid (Infinispan)
JBoss ON 3.1 introduces a new resource plug-in to manage JBoss Data Grid 6.0. This plug-in pack is available with the JBoss Data Grid packages in the Customer Portal.
Complete reference material for the JBoss Data Grid plug-in is available in the Resource Reference, which lists all supported configuration properties, connection properties, metrics, and operations.
1.10. Enhanced: Additional Information in SNMP Notifications
The resource ancestry, including the parent and grandparent resource names, are now included in SNMP alert notifications.
1.11. New: Send the Current JBoss ON Server State to the Logs
A new button was added to the System Settings area of the Administration tab. The button exports the current server configuration to the server logs.
2012-05-14 19:44:28,587 INFO [SystemInfoManager] SystemInformation: ******** CAM_LDAP_BIND_PW: [- non null -] AlertDefinitionCount: [2] CAM_LDAP_BASE_DN: [o=JBoss,c=US] AVAILABILITY_PURGE: [31536000000] CAM_JAAS_PROVIDER: [false] BuildNumber: [ca099bc:3a46aff] ServerCount: [27] DATABASE_DRIVER_NAME: [PostgreSQL Native Driver] RESOURCE_GENERIC_PROPERTIES_UPGRADE: [false] ... 8< ...
This is useful for debugging and for improving the server performance, as well as providing a way to verify that the server configuration was updated after synchronizing configuration between JBoss ON servers.
2. Structural Changes in JBoss ON 3.1
There are some structural changes between JBoss Operations Network 3.1 and 3.0 which are called out because they affect the usage and experience of JBoss ON.
2.1. General Changes to the Remote API
All changes to the remote API, used by alert scripts and CLI scripts, are detailed in the change list for the 3.1 API.
2.2. Changes in the Resource Live Availability Method
As part of the availability monitoring enhancements, the resource manager API has been improved to change how it handles different resource states. Instead of the returning a null value for the UNKNOWN state, the
ResourceManagerRemote.getLiveResourceAvailability() method now properly returns AvailabilityType.UNKNOWN.
This may affect existing remote clients or CLI scripts.
2.3. Obfuscated Passwords for Content Providers
Passwords used to access content providers such as the JBoss Customer Portal are now stored in the JBoss ON database in an obfuscated form.
During server upgrade, existing content provider configuration is upgraded as well so that the saved passwords will be obfuscated.
2.4. Deprecated Plug-ins
- The JBoss AS/EAP 4 resource plug-in is deprecated in JBoss ON 3.1. Support will be removed in a later release, coincidental with the sunset of JBoss AS 4.
- The PostgreSQL database resource plug-in is deprecated in JBoss ON 3.1. Support will be removed in a later release.
- The Generic Database resource plug-in is deprecated in JBoss ON 3.1. Support will be removed in a later release.
- The
defaultattribute for thec:optionelement in agent plugin-descriptors is deprecated and will be removed in a future JBoss ON version.
2.5. Delayed Plug-In Releases for JBoss Enterprise Web Server and Tomcat
The latest versions of some JBoss products are scheduled after the general availability of JBoss Operations Network 3.1. The initial release of JBoss ON 3.1, then, will not include updates for these resources:
- JBoss Enterprise Web Server
- Tomcat
The EWS plug-in pack contains the resource plug-in to monitor both EWS instances and generic Tomcat servers. This plug-in will be released at some point after the 3.1 GA release.
Important
Because previous versions of JBoss ON had EWS/Tomcat plug-in packs available, existing customers who are using that plug-in to manage EWS or Tomcat instances should not upgrade to JBoss ON 3.1. Upgrading without an available EWS/Tomcat plug-in would mean that any EWS or Tomcat resource will no longer be managed by JBoss ON.
Existing JBoss ON users who are managing EWS/Tomcat resources are encouraged to stay on JBoss ON 3.0.1 until the EWS/Tomcat plug-in is available for JBoss ON 3.1.
For information on the projected release schedule for the EWS/Tomcat plug-in or to discuss issues or workarounds for EWS/Tomcat management, please contact Red Hat Global Support Service.
2.6. Deprecated Webservices Class in the Remote API
The Webservices class in the remote API is deprecated in JBoss Operations Network 3.1. Its use is not recommended.
The Webservices API will be removed in the next release of JBoss ON.
3. JBoss Operations Network Prerequisites and Basic Installation
The list of supported platforms for JBoss Operations Network are listed at http://www.redhat.com/resourcelibrary/articles/jboss-operations-network-supported-configurations.
3.1. Supported Platforms
The JBoss Operations Network server and agent require Java 6. Any operating system which supports the appropriate JVM could be run a JBoss ON server or agent. Check with Red Hat support for specific platform and environment information.
These platforms are fully supported with native JVM support:
- Red Hat Enterprise Linux 6 (32-bit and 64-bit)
- Red Hat Enterprise Linux 5 (32-bit and 64-bit)
- Red Hat Enterprise Linux 4 (32-bit and 64-bit)
- Microsoft Windows 2008 R2 (32-bit and 64-bit)
- Solaris 11 (64-bit)
- Solaris 10 (64-bit) (deprecated)
3.2. Supported Databases
JBoss Operations Network 3.1 supports these databases for the JBoss ON server:
- Oracle 11g
- PostgreSQL 9.0
- PostgreSQL 8.4.x
- PostgreSQL 8.3.x
- PostregSQL 8.2.4 and later releases (deprecated)
3.3. Hardware Requirements
Regardless of the server or database platform, there are certain minimum requirements that must be met to install the JBoss ON server and its associated database.
3.4. Required Java Versions
Java 6 is required. Both Sun and OpenJDK Java are supported.
The server requires the JDK, while the agent only requires the JRE.
3.5. Supported Web Browsers
JBoss ON supports these browser releases for running the installer and accessing the server GUI:
- Firefox 10
- Internet Explorer 8
3.6. Quick Setup for JBoss Operations Network 3.1
Important
When installing JBoss plug-in packs, only the latest versions of EAP 6 are supported. Do not use JBoss ON 3.1 Beta with an older version of EAP 6 or JBoss AS 7.1.
The Installation Guide has more information on configuring databases for the JBoss ON server, upgrading servers, and other installation information. This is a quick summary of performing an initial installation.
- Set up a database named
rhqadminand a database user namedrhqadmin. Make sure the database user has adequate permissions to perform operations on the JBoss ON database.For more detailed database setup information: - Stop any currently running JBoss ON instances.
serverRoot/jon-server-3.1.0.GA/bin/rhq-server.sh stop
If the new JBoss ON server will use a database that existing JBoss ON instances are also using, then all of the existing JBoss ON instances have to be stopped. - Download the server packages from the Customer Support Portal.
- Click Software in the top menu.
- In the Product field, select the product.
- Unzip the server distribution to the directory where it will be executed from. For example:
cd /opt unzip jon-server-3.1.0.GA.zip
This creates ajon-server-3.1.0.GA/directory in your chosen installation root. - Run the JBoss ON server:
serverRoot/jon-server-3.1.0.GA/bin/rhq-server.sh start
- Open the server UI at
http://localhost:7080/. This will display the JBoss ON server installer.Clicking the Click here to continue the installation link brings you to the main installer page. - Go through the installer and fill in the information. Only the database connection information is required; most of the default can be accepted without changes.
The main installer page appears different depending on the database settings. - When the configuration is complete, click the link to log into the JBoss ON UI.
4. Fixed Bugs in JBoss Operations Network 3.1
These bugs have been fixed in JBoss Operations Network 3.1. For a complete list of all issues addressed in JBoss Operations Network 3.1, query Bugzilla for all bugs which target the JBoss ON 3.1 release.
Table 3. JBoss ON 3.1 Fixed Bugs
| Bug Number | Component | Description |
|---|---|---|
| Bugzilla 768707 | Agent | If the RHQ_AGENT_CMDLINE_OPTS variable was set in the rhq-agent-env.sh file, only the options specified in the variable were used by the agent, and none of the default (required) values were supplied. This could cause the agent to fail to start or to have odd behavior. |
| Bugzilla 790753 | Agent |
The assumption is that the JBoss ON agent and resources like a JBoss server or Tomcat server run as the same system user. If the agent user is different than the resource user, then there can be problems when one entity makes a configuration change and the other attempts a change later.
For example, when deploying an application, the deployment operation is initiated by the agent and the content is supplied through the agent, and then the application server completes the actual deployment. When deleting an application, the application server handles the undeployment by itself.
If a WAR file is deployed exploded without a
MANIFEST.MF file, the agent creates one when it writes the SHA-256 value for the package. When the JBoss AS/EAP server tries to remove the WAR application later (and the JBoss AS/EAP user is different than the agent user), then the removal fails. The JBoss AS/EAP server cannot delete the MANIFEST.MF file. The agent then rediscovers the application directory and re-initiates the deployment operation for the removed WAR.
|
| Bugzilla 787322 | CLI | If the CLI were run as a service or in the background, even with a proper script file or commands passed with the CLI, the Java process would hang and the command execution failed. The CLI required that a terminal be attached to the Java process, even when it was not necessary. |
| Bugzilla 815869 | Groups | In some rare cases, an autocluster (compatible) group definition could be switched to a mixed group, but the autogroup definition stayed associated with the group. Attempting to delete the dynamic mixed group (with a wrongly associated cluster group) would fail with database errors. |
| Bugzilla 819897 | Groups | The number of members listed in recursive groups could be wrong because uncommitted resources listed in the discovery queue are included as implicit members. |
| Bugzilla 755703 | Monitoring | The button for a per-minute metric for a resource returned the raw value for the metric instead of the averaged per-minute value. |
| Bugzilla 786889 | Monitoring | In some rare circumstances, the agent would allow a metric schedule to be set with an invalid collection schedule, such as 0 seconds. The agent now rejects invalid metric collection schedules, automatically sets the schedule to 20 minutes, and logs an error in the agent log. |
| Bugzilla 815979 | Monitoring | The memory metrics reported by JBoss ON differed from the metrics reported by native system tools because the JBoss ON calculations did not include space from caches or buffers. The memory calculation has been changed to include buffer and cache metrics, as new actual free and actual used memory, and platform utilization metrics have been updated. |
| Bugzilla 767981 | Plug-in (JBoss EAP 4 and 5) | When deploying a new ESB through the GUI, JBoss ON always installs it in the JBoss server's deploy/ directory, regardless of whatever target directory was entered in the deployment wizard. |
| Bugzilla 646631 | Plug-in (JBoss EAP 4 and 5) | If an application is deployed to a JBoss server's farm directory and an update content operation is run, the application is no longer deployed to the farm. |
| Bugzilla 789967 | Plug-in (HornetQ) | A problem in the HornetQ plug-in meant that the agent attempted to discover the parent of a HornetQ resource as a JBoss AS 5 server, even if the actual parent was a JBoss AS 4 server. This has been fixed in HornetQ 2.0.1.Final. |
| Bugzilla 817690 | Plug-ins (HornetQ) | The HornetQ plug-in with JBoss ON 3.0.1 did not work with the version of HornetQ included with JBoss EAP 5. This caused Java exceptions when attempting to add HornetQ resources. |
| Bugzilla 799532 | Plug-ins (JMX) | The Java attach API attempts to discover Java processes which are running as the same user as the agent. When the agent is running as root, it would sent a SIGQUIT command to attempt to connect to a Java process running as another user, but the connection attempt failed, so all the agent did was kill the other Java process. The JMX plug-in has been changed so that the agent does not attempt to connect to any Java process running as a different user. |
| Bugzilla 751873 | Provisioning | If a WAR file was deployed through JBoss ON and then deleted on the local system, the JBoss server still ran the application from memory, with no attempt to undeploy the application. |
| Bugzilla 790224 | Provisioning | Attempting to revert a bundle past a certain number (about 10 times) would cause the revert process to fail with a database error that the concatenated string exceeded the column width. |
| Bugzilla 804139 | Provisioning | If the file content in a bundle was too small, the content manager did not properly initialize the content, and then deploying the bundle failed because the SHA256 of the initialized content did not match the stored SHA256. |
| Bugzilla 806214 | Provisioning | Uploading deploy.xml through the provisioning wizard actually corrupted the XML when it was rendered in the browser. This created an invalid recipe file, which caused the bundle deployment to fail. |
| Bugzilla 738209 | Security | If an LDAP group was a member of a role, that role could not be deleted. |
| Bugzilla 747925 | Server | If an agent plug-in definition was updated, the existing resources of that resource type were not updated, so new operations or metrics or other changes were not displayed for the resources. |
| Bugzilla 813083 | Server | A deleted resource (such as an EAR or WAR application) is not deleted from the server inventory; it is essentially stashed. If the resource is ever re-added, then its previous audit trails and metrics histories are also preserved. However, the server improperly attempted to sync deleted resource over to the agent. When the agent then sent subsequent inventory reports to the server, it included the deleted resources — which caused the server to reject the inventory report. Now, the server inventory sync filters out the (preserved) deleted resources before sending updates to the agent. |
| Bugzilla 789529 | UI | When checking the Inventory > Connection Settings tab for an autogroup, the server could throw a ClassCastException. It was checking the resource manager to see if updates were currently in progress, when it should have been checking the plug-in configuration manager. |
| Bugzilla 796096 | UI | All of the bottom buttons (like ) were inactivated after creating a child resource through a resource's Inventory tab. |
| Bugzilla 804303 | UI | The UI did not preserve changes to the default view for a resource's Monitoring > Graphs tab. This error also prevented users from creating a custom view for a resource's monitoring graphs. |
| Bugzilla 805210 | UI | When using an Oracle database as the backend, some portlets for the dashboard could fail to load because a property value could have an empty value, which Oracle interpreted as NULL. |
| Bugzilla 817165 | UI | Some autogroups are automatically named after their child resource. For example, embedded WAR applications belong to an autogroup that is named for the WAR file. If the child resource changes its name, then the autogroup was renamed in the UI as "...", not the new name of the resource. |
5. Known Issues in JBoss Operations Network
This section contains a subset of the known issues at the time that JBoss Operations Network 3.1 was released. When possible, workarounds are provided.
5.1. Upgrade and Installation
Warning
Upgrade from JBoss Operations Network 3.0 to JBoss Operations Network 3.1 is fully supported.
However, upgrade from JBoss Operations Network 3.1 Beta to JBoss Operations Network 3.1 is not supported. It is not possible to upgrade a beta version of JBoss Operations Network to the release version. If you install a beta server, then you must install the server, and all its resource configuration, fresh for JBoss ON 3.1 GA.
Bugzilla 821093: "HTTP Status 404 - /coregui/" error after upgrade
When the JBoss ON server is upgraded from 3.0 to 3.1, then the UI can return a 404 error when you try to open the installer page to re-install the server.
Even when opening the proper URL to access the installer, such as
server.example.com:7080, the browser has a cached redirect to the old UI, server.example.com:7080/coregui. Because the coregui/ path does not exist yet, because the new server has not yet been configured, this redirect returns a 404 error.
There are a few ways to work around this:
- Enter the full URL to the installer page:
http://server.example.com:7080/installer/welcome.jsf
- Delete the cache and reload the page.
- Use Ctrl+F to force the page to reload.
Some information about the server cannot be changed during upgrade. Even though the fields are editable in the upgrade form, changing these values from the original settings causes upgrade problems. This includes information like the server name. If any of that information changes, then one of the results is NPEs in the server logs.
For example, the process to register the upgraded server plug-ins fails with a null pointer exception. For example:
2012-03-08 20:33:34,523 ERROR [org.rhq.enterprise.server.core.plugin.ServerPluginScanner] Failed to register server plugin file [/home/hudson/jon-server-3.0.1.GA/jbossas/server/default/deploy/rhq.ear/rhq-serverplugins/rhq-serverplugin-ant-bundle-4.2.0.JON.3.0.1.GA.jar] java.lang.NullPointerException at org.rhq.enterprise.server.core.plugin.ServerPluginScanner.registerServerPlugin(ServerPluginScanner.java:212) ...
This error only occurs if a different server name was entered in the configuration page when the server was upgraded. Changing the Server Name field is not supported for upgrades.
If the installer is not bound to 0.0.0.0 when setting up a server, then it does not set all of the required connection properties. Specifically, the installer does not set the
java.rmi.server.hostname parameter to the real value, and it uses the default of 0.0.0.0. This parameter must be set to the real IP address of the server by manually editing the rhq-server.properties file.
Automatic upgrades of agents on Windows appear to fail. Two conditions must be true for automatic upgrade of agents to work on Windows:
- The agent must be installed and running as a Windows service.
- No Windows Explorer window can be open to any agent subdirectory. Windows sets a lock on directories when they are open, which blocks the upgrade process.
When starting a server while agents are running, the server may log servlet errors in the logs. This is because the remoting (communications) classes are loaded early in the startup sequence, before the server is completely started. This causes some communications interruptions until the server is completely started. These errors can be ignored.
If a JBoss ON server was added to the resource inventory, the old JBoss ON server resource must be deleted from the inventory after upgrade. The old resource entry is unusable after upgrade.
Null pointer exceptions for the
org.apache.catalina.connector.CoyoteAdapter service are returned when the JBoss ON 3.1 server is first installed. These errors are harmless and can be ignored. Installation will complete successfully, and both the server and the GUI will start and run properly.
The installer shows non-replaced {} properties.
5.2. JBoss EAP 6 Resources
The tracking bug for all resolved and open issues related to the EAP 6 plug-in for JBoss ON 3.1 is Bugzilla 707223.
- An edit icon appears next to module names in the Configuration > Current > Extensions area of the EAP 6 server configuration. These properties are read-only and cannot, in fact, be edited even though an edit box pops up.
- If an EAP 6 server is reset to run insecurely, then the "Install RHQ User" operation for the EAP 6 server fails because it has no way to connect to the (secure) management realm.
- Bundle provisioning is only available for EAP 6 standalone servers. Standalone servers use a filesystem-based deployment directory, which is used by the bundle system. Domain resources use a different, distributed style of deployment. The domain style of content distribution is not supported for bundles.
- When editing a network interface for a JBoss EAP 6 server, the network configuration requires either a specific IP address or a setting to use any IP address for IPv4, for IPv6, or any address. However, the UI does not enforce that an IP address selection is made, which can cause the interface to be unreachable.
- Attempting to deploy a child resource such as a WAR application fails with a malformed URL exception if the EAP 6 server is listening on IPv6.
- All child elements for an embedded HornetQ resource — connection factories, JMS queues, and JMS topics — all require a JNDI name in the configuration. Optionally, multiple JNDI names can be set, in a comma-separated list, to support failover.However, the UI does not force the JNDI name to be given when creating the connection factory, JMS queue, and JMS topic children. This means that the creation operation fails if the JNDI name is not included.
- The JBossWeb connector configuration has a section for SSL parameters. Any information in those fields is only read if SSL is enabled (the secure option is selected). However, these fields are still active even if SSL is not enabled. They should be grayed out.Also, if SSL is enabled and then disabled, all of the values for the SSL configuration fields are blanked and unset.
- When manually adding a standalone server in the JBoss ON UI, rather than importing it from the discovery queue, there is an intermittent error:
The original product type discovered for this AS7 server was null, but the server is now reporting its product type is [JBoss EAP 6]
This is shown in a red bar at the top of the UI. Despite the error, the resource is successfully imported and available. This error can be ignored. - Some path elements defined in the configuration of a domain controller or standalone server are computed by the server itself. These computed paths are all prefixed with
java.*,jboss.*, anduser.*.While the configuration values are viewable, they are not editable. However, the Configuration tab for the server shows both edit and delete options for the path elements. If a user attempts to change or delete the path, the path simply reverts to its previous state. - A JVM definition can be created at the host, server group, and managed server levels. There can be multiple JVM definitions at the host level; these are used as templates or base definitions for the server group and managed server definitions. However, server groups and managed servers can have only a single JVM definition.The UI allows a user to attempt to create a second JVM definition as a child to a managed server or server group. If a JVM definition already exists for a managed server or server group, then an error is returned if a user attempts to create another JVM definition. The server throws an exception and there is an error message that the JVM definition is a singleton and that the first resource should be deleted before creating another child resource of that type.
- When a child JVM definition is created for a managed resource, the name of the JVM definition must be the same as the host JVM definition which it is based on. However, the JBoss ON UI creation wizard does not detect or enforce a pre-computed name, so it allows a user to enter any name.
- The UI will not let users enter a driver name when manually adding a datasource, so it is not possible to deploy a module-loaded driver.
- The agent properly detects an EAP 6 server running over an IPv6 address, but it incorrectly reports it as offline.
- It can take several minutes to be able to detect a child resource on a JBoss EAP 6 server.
- If both an EAP 6 server and a JDG server are running on the same system, the EAP 6 child resources are detected and added as (unavailable) child resources of the JDG instance.
- A user can edit configuration in JBoss ON that requires a reboot of EAP 6 to take effect, but there is no indication that the reboot is required.
5.3. Server Configuration
The password mechanism in the server installer doesn't properly hash Unicode characters. The Unicode characters are translated into the HTML entities, and then the HTML characters are hashed and stored as the password.
For example, the string
你好 is treated as the string 你好 in the installer, and the second string is the one that is hashed and stored in the rhq-server.properties file.
After configuring the server, but before logging in, manually update the password in the
rhq-server.properties file:
- Manually generate a hashed password based on the Unicode password:
serverRoot/jon-server-3.1.0.GA/bin/generate-db-password.sh 你好 Encoded password: 68f725778bb36d3b
- Replace the database password value with the new hashed password.
rhq.server.database.password=68f725778bb36d3b
5.4. Agents
When a plug-in, effectively a resource type, is deleted, the agent's plug-in container is automatically rebooted, to reload the new configuration. If the agent itself is running as a daemon, then rebooting the plug-in container also kills the agent thread. That means that if a plug-in (with resources of that type in the inventory) is deleted while the agent is running in the background, the agent process is stopped.
The agent can be restarted without any errors, and the plug-in is successfully removed.
Attempting to use the
config import command in the agent command prompt fails with an InvalidPreferencesFormatException exception.
java.util.prefs.InvalidPreferencesFormatException: org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 41; Document root element "preferences", must match DOCTYPE root "null". ...
The JVM can sometimes cause the agent to crash when Augeas is running.
A fatal error has been detected by the Java Runtime Environment ...
Attempting to ignore a resource in the discovery queue returns a ConcurrentModificationException on the agent.
5.5. Databases
The agent assumes that PostgreSQL uses a default password. If a different password is used, the PostgreSQL resource is shown as unavailable in the UI even if it is running.
5.6. Groups
A dynagroup definition with a short recalculation interval (less than two minutes) can cause duplicate groups to be created.
Changing the recursive setting on a group disassociates the group from any roles it was (previously) associated with.
5.7. Provisioning
Hot deployment fails if the file changes during provisioning.
If a bundle is being deployed in a hot deployment and one of the underlying bundle files, like a WAR or JAR, is changed while the bundle is being uploaded, then provisioning will fail.
Using the Recipe option to upload a recipe file separately than the bundle archive files requires that the XML in the recipe be well-formed and that all tags are explicitly closed, using the format
<tag></tag>, not the abbreviated format <tag />.
Additionally, the Recipe option's upload button does not work on Internet Explorer. To add a recipe file using this option with Internet Explorer, copy the entire recipe file and paste it directly into the text box.
The
<rhq:system-service> Ant tag only works on Red Hat Enterprise Linux systems. Using the tag and then deploying the bundle to other operating systems causes the Ant task to fail to operate and can cause the deployment to fail.
5.8. Content and Repositories
The installation data for content deployed through JBoss ON can be very inaccurate, off by thousands of days.
A repository can only be used for one resource, the one it was initially defined for.
Attempting to delete a repository with a space in the name throws an exception and the deletion fails.
Deploying WARs from a content repository is not supported. WARs and EARs should be deployed using bundles.
Deploying an exploded archive on JBoss EAP 4 results in deploying a zipped archive, which makes the deployed application unavailable.
5.9. Remote API and the CLI
The JBoss ON remote API cannot be used to connect to the JBoss ON server when the client is running inside an application server. For example, the remote API cannot be run from a client inside an EAP instance; it fails with errors like the following:
Caused by: java.lang.IllegalArgumentException: interface org.rhq.enterprise.server.auth.SubjectManagerRemote is not visible from class loader at java.lang.reflect.Proxy.getProxyClass(Proxy.java:353) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:581) at org.rhq.enterprise.client.RemoteClientProxy.getProcessor(RemoteClientProxy.java:69)
ResourceFactoryManager does not throw any exceptions when attempting to create duplicate or invalid resources in the CLI.
Running a search with the *Criteria API using unlimited PageControl, as well as one or more fetches, returns duplicate results.
5.10. UI
Problems with Sessions and Multiple Tabs
If there are multiple tabs open to the JBoss ON UI and one of those sessions times out, then the other tabs will register errors. This can also occur if multiple people are logged into JBoss ON as the same user.
If there are more than 200 children for a resource, not all of the children are displayed in the resource tree. Additionally, refreshing the tree may display a different subset of child resources.
Searching for a resource with a backslash (\) in the name returned no results, even if the resource exists.
Bugzilla 736732: Bookmarked UI Pages May Display the Old UI
The page names used for different pages in the JBoss ON UI for version 3.0 are different than the ones used in JBoss ON 2.4. However, because JBoss ON 3.0 retains some of the elements used in the older JSF-based UI, these pages still exist. This means that bookmarks will go to the old style pages.
For example, the Dashboard page in the old JSF UI had the URL http://localhost:7080/Dashboard.do/. In the new UI, the location is http://localhost:7080/coregui/#Dashboards.
All of the new pages have the
coregui/ element in them. Remove any old bookmarks and replace them with the new GWT-based pages in the coregui/ area.
When the results for a dynamic search are long, the list is truncated but no scrollbar is presented, so it is not possible to scroll to the end of the list.
5.11. Measurements and Monitoring
A user-defined baseline value for a metric is ignored and overwritten by the server-calculated baseline.
5.12. Alerts
Attempting to upload a CLI script directly to a repository to use for alerting fails because it says the SHA-256 checksum for the script is not an OSGi string. Uploading a CLI script through the alert configuration UI works successfully.
In Oracle, the alert notification table is created with the NOLOGGING option. This means that users are required to review the alert notification configuration manually for every alert to verify that the configuration is correct.
Alerts with compound AND conditions can incorrectly fire when one of the conditions goes from true to false within 30 seconds of the other condition going from false to true.
Alert conditions and alert notifications in an alert definition cannot be edited. The condition or notification must be deleted and a new item added.
Recovery alerts are not triggered by an availability condition.
5.13. Drift
Drift definitions can be created for resources which do not support drift, using the remote API to create a script.
Drift monitoring is not supported for embedded web applications, only for standalone applications.
5.14. Platform Resources
The inventory for a Linux platform may discover non-existent network adapters, which are added and then shown as unavailable in the inventory.
5.15. Tomcat Resources
If a WAR file is configured to be deployed to a non-standard location on a Tomcat application server, an invalid deployment name is added to the inventory. The
unpackWARs option extracts the web app to a second, invalid directory. For example, if it should be extracted to /myapps/helloworld, the web app is extracted both there and to /myapps#helloworld.
The agent cannot discover a Tomcat instance if its Java process is running as a user other than the system user that the agent is running as.
5.16. Apache Resources
Apache directives are case-insensitive, but the Apache plug-in treats them as if they were case-sensitive. This makes the plug-in disregard directives with non-default case.
If the
/tmp directory is configured a noexec in the fstab file, the agent throws exceptions because it cannot properly initialize the Augeas lens. In that case, the Configuration tab is unavailable for the Apache resource.
To work around this, make sure that the
/tmp directory does not have noexec set as an option.
# # /etc/fstab # tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0
noexec is one security method available to that directory to prevent unauthorized or malicious scripts from running. Disabling noexec for the directory is not ideal, but it is also not a security issue by itself because other filesystem security settings can be used.
5.17. JBoss and JVM Resources
If EAR, WAR, or script resources are deleted outside of JBoss ON, the agent throws an exception during discovery, and the server is not informed that the resource was deleted.
Discovering a JBoss AS instance fails if the configuration directory is a URL.
The discovery process fails to determine the JNP URL if the data directory is not located at ${jboss.server.home.dir}/data.
Shutting down a JBoss AS 5 resource can fail with java.rmi.NoSuchObjectException: no such object in table.
It is not possible to add some system properties for a JBoss AS resource.
The JBoss AS plug-in is unable to load resource configuration for JMS queues that are deployed using a SAR file.
The discovery process attempts to discover an embedded JVM process through its JMX remoting rather than using the EMSConnection of its parent Tomcat or JBoss resource.
5.19. Performance
Attempting to update the members in a compatible group with thousands of members times out and fails with 'Failed to fetch Resource Group.'
Viewing configuration, alerts, or other data for groups with more than 1000 members can fail with the SQL exception maximum number of expressions in a list is 1000.
5.20. Plug-in Writing
If an embedded plug-in extension is defined and the parent resource type has subcategories, those subcategories cannot be changed or it causes a database deadline.
6. Document History
| Revision History | |||
|---|---|---|---|
| Revision 3.1-3.400 | 2013-10-31 | ||
| |||
| Revision 3.1-3 | June 19, 2012 | ||
| |||
| Revision 3.1-1 | June 12, 2012 | ||
| |||
Legal Notice
Copyright © 2012 Red Hat, Inc..
This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
