Red Hat Training

A Red Hat training course is available for Red Hat JBoss Operations Network

3.6. Server Tuning for Large Numbers of Agents

When the JBoss ON server has a large number of agents in its inventory (depending on resource counts and configuration like monitoring schedules, this can be as few as 100 or over 1,000 agents), the default settings may not allow it to do an upgrade. The size of the data is enough that the server cannot load all of it cleanly.
While less likely, there could be similar performance issues during regular operation.
One of the most apparent symptoms is frequent timeouts for agent requests.
The problem is not memory related; rather, it is a threading problem. The number of agent requests overwhelms the JBoss ON server, which slows overall process.
There are three parts of the server configuration which can be adjusted to improve performance: increasing the storage node memory settings; increasing the EJB pool; and resetting the concurrency limits to increase the number of allowed agent connections.
  1. Increase the default size of the storage node memory usage. This is only necessary for a large number of nodes, around 1,000 or more.
    1. Open the rhq-storage.properties file. 
      [root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-storage.properties
    2. Uncomment the heap-size parameter and set the heap size to 5GB. 
      rhq.storage.heap-size=5120
  2. Increase the EJB pool.
    1. Open the server's standalone-full.xml profile. 
      [root@server ~]# vim /opt/jon/jon-server-3.2.GA/jbossas/standalone/configuration/standalone-full.xml
    2. Change the strict-max-pool key to increase the pool size. The default is 20. For example: 
      <strict-max-pool name="slsb-strict-max-pool" max-pool-size="2000" instance-acquisition-timeout="1" instance-acquisition-timeout-unit="MINUTES"/>
  3. Increase the concurrency limits to increase how many agents can communicate with the server simultaneously.
    1. Open the rhq-server.properties file. 
      [root@server ~]# vim serverRoot/jon-server-3.2.GA/bin/rhq-server.properties
    2. There is a block of communication-related parameters. Concurrency limits are set in the concurrency-limit parameters and the rhq.communications.global-concurrency-limit parameter. There are other communication limits for web UI connections and downloads. The different communication parameters are covered in Section 6.3.3, “Setting Concurrency Limits”.
      For example: 
      rhq.server.startup.web.max-connections=1000
rhq.server.agent-downloads-limit=45
rhq.server.client-downloads-limit=5
rhq.communications.global-concurrency-limit=200
rhq.server.concurrency-limit.inventory-report=25
rhq.server.concurrency-limit.availability-report=25
rhq.server.concurrency-limit.inventory-sync=25
rhq.server.concurrency-limit.content-report=25
rhq.server.concurrency-limit.content-download=25
rhq.server.concurrency-limit.measurement-report=25
rhq.server.concurrency-limit.measurement-schedule-request=25
rhq.server.concurrency-limit.configuration-update=25
  4. Restart the server to load the new settings. 
    [root@server ~]# serverRoot/jon-server-3.2.GA/bin/rhqctl restart --server

Note

The server will not use the new configuration until it is restarted.