Red Hat Training

A Red Hat training course is available for Red Hat JBoss Operations Network

9.8. Backing Up and Restoring the Metrics Storage Database

The storage nodes have a defined flow to manage and archive data. Scheduled snapshots back up all node data (both metrics and node configuration). This allows data to be restored if the cluster is ever corrupted.

9.8.1. About Storage Data Files

All of the database files for the node are stored in the rhq-data/ directory, which is in the same root directory are the JBoss ON server (such as /opt/jon). This directory is divided into three subdirectories:
  • A commit log for write data that has yet to be written to the database
  • A cache directory for archives
  • A data directory which contains both all monitoring data (aggregated and raw), snapshot archives of monitoring data, and node (system) data
             /opt/jon
                |
             rhq-data/
                |
      ---------------------
      |            |      |          
     commit_log/  data/  saved_caches/
The commit_log/ directory contains binary commit logs which store the writes before they are written to the database. Each commit log is named by the timestamp of when it was created.
--- commit_log
   |
   |-- CommitLog-2-timestamp.log
The saved_caches/ directory contains archives for the major tables within the storage node, both the metrics tables and the storage node configuration.
--- saved_caches/
   |
   |-- rhq-metrics_index-KeyCache-b.db
   |
   |-- rhq-one_hour_metrics-KeyCache-b.db
   |
   |-- rhq-raw_metrics-KeyCache-b.db
   |
   |-- rhq-six_hour_metrics-KeyCache-b.db
   |
   |-- system-local-KeyCache-b.db
   |
   |-- system-schema_columnfamilies-KeyCache-b.db
   |
   |-- system-schema_columns-KeyCache-b.db
   |
   |-- system_auth-credentials-KeyCache-b.db
   |
   |-- system_auth-users-KeyCache-b.db
The data/ directory contains the most critical data. This directory is organized by keyspace and then table. The keyspaces are rhq (monitoring data), system_auth, system_traces, and system.
The rhq/ directory contains all of the metrics data. This is covered in the Using JBoss Operations Network for Monitoring, Deploying, and Managing Resources, but all collected monitoring data is aggregated into averages and trends at periodic intervals. The is raw monitoring data which has recently been collected, and then rolling aggregations for the past hour, past six hours, and past 24 hours. Each time period has its own metrics table. snapshot directory is a UNIX timestamp in milliseconds.
There is an additional table that contains an index for all metrics data and all snapshots (backups) of the metrics data. Snapshots are stored in a directory named for the UNIX timestamp of when it was taken.
                     rootDirectory
                           |
                       rhq-data/
                           |
                          data/
                           |
               ------------------------
               |                      |
              system databases...    rhq/
                                      |
  -------------------------------------------------------------------------------------------
  |                 |            |              |                 |                         | 
one_hour_metrics  raw_metrics  schema_version  six_hour_metrics  twenty_four_hour_metrics  metrics_index
                                                                                                 |
                                                                        ------------------------------------------------
                                                                        |                                              |
                                                                        |-- rhq-metrics_index-ic-1-Data.db           snapshots
                                                                        |-- rhq-metrics_index-ic-1-Digest.sha1         |
                                                                        |-- rhq-metrics_index-ic-1-Filter.db           |-- timestamp
                                                                        |-- rhq-metrics_index-ic-1-Index.db            |-- timestamp
                                                                        |-- rhq-metrics_index-ic-1-Statistics.db
                                                                        |-- rhq-metrics_index-ic-1-Summary.db
                                                                        |-- rhq-metrics_index-ic-1-TOC.txt
                                                                        |-- rhq-metrics_index-ic-2-Data.db
                                                                        |-- rhq-metrics_index-ic-2-Digest.sha1
                                                                        |-- rhq-metrics_index-ic-2-Filter.db
                                                                        |-- rhq-metrics_index-ic-2-Index.db
                                                                        |-- rhq-metrics_index-ic-2-Statistics.db
                                                                        |-- rhq-metrics_index-ic-2-Summary.db
                                                                        |-- rhq-metrics_index-ic-2-TOC.txt
                                                                        |-- rhq-metrics_index-ic-3-Data.db
                                                                        |-- rhq-metrics_index-ic-3-Digest.sha1
                                                                        |-- rhq-metrics_index-ic-3-Filter.db
                                                                        |-- rhq-metrics_index-ic-3-Index.db
                                                                        |-- rhq-metrics_index-ic-3-Statistics.db
                                                                        |-- rhq-metrics_index-ic-3-Summary.db
                                                                        |-- rhq-metrics_index-ic-3-TOC.txt
The rest of the data/ directory is comprised of database files for different node areas, including users and permissions (system_auth/), database sessions (system_traces), and schema and configuration (system).
                                      rootDirectory
                                            |
                                        rhq-data/
                                            |
                                          data/
                                            |
                    ------------------------------------------------------------
                    |                            |                   |         |
              system_auth/                system_traces/          system/    rhq/
                    |                            |                   |        |
      ---------------------------           ------------             |       ........
      |             |           |           |          |             |
    credentials/  permissions/ users/     events/    sessions/       |-- HintsColumnFamily
                                                                     |-- IndexInfo
                                                                     |-- LocationInfo
                                                                     |-- Migrations
                                                                     |-- NodeIdInfo
                                                                     |-- Schema
                                                                     |-- batchlog
                                                                     |-- hints
                                                                     |-- local
                                                                     |-- peer_events
                                                                     |-- peers
                                                                     |-- range_xfers
                                                                     |-- schema_columnfamilies
                                                                     |-- schema_columns
                                                                     |-- schema_keyspaces