9.8. Scaling the Cassandra Storage Node for JBoss ON server

Regardless of the load or demand on a JBoss ON installation, the JBoss ON server will go into maintenance mode while:
  • the storage node processes a backlog of storage requests (during times of peak demand),
  • transfers data from memory to disk, or
  • performs data aggregation or compaction.
In most instances, the system is designed to recover without interruption. However, as the demand or load on a system increases, the length of these outages will also increase. Once the length of these outages begins to interrupt operations, additional storage nodes should be installed to distribute the load. Additional storage nodes are also required when the storage node has insufficient storage space. For details on installing additional storage nodes, see Section 9.2.4, “Installing Additional Nodes”.
Scaling the Cassandra-based JBoss ON Storage Node helps improve performance and reliability, including improving the capacity of the storage node to respond to metric read and write requests. There are several ways the storage node can be scaled. These are:
  • Increase the available memory in the JBoss ON Storage Node's JVM by increasing the heap_max and heap_new properties in RHQ_SERVER_HOME/rhq-storage/conf/cassandra-jvm.properties. For example:
    The heap size can also be changed in the JBoss ON UI, as described by Section 9.7, “Managing the Storage Node Heap”.
    Ensure the host machine has enough physical memory to handle the value set for maximum heap, otherwise the host machine uses swapping memory, slowing the processing speed of the storage node.
  • Increase the length of time objects survive in the new generation space by increasing the max_tenuring_threshold property in RHQ_SERVER_HOME/rhq-storage/conf/cassandra-jvm.properties. For example:
    If the max tenuring threshold of the JVM exceeds 15, the JVM garbage collection does not function correctly.
  • Add additional JBoss ON Storage Nodes to the storage cluster to distribute processing across multiple machines and disks as described in Section 9.2.4, “Installing Additional Nodes”.
  • Relocate the rhq-data directory of the JBoss ON Storage Node to a dedicated disk to improve performance.
  • Move the JBoss ON Storage Node's rhq-data and commit_log directories to dedicated physical disks.
  • Increase the write timeout value by setting the write_request_timeout_in_ms property in RHQ_SERVER_HOME/rhq-storage/conf/cassandra.yaml. For example:
    write_request_timeout_in_ms: 40000
    Increasing the timeout for write operations is not recommended.