Managing storage node snapshots in JON

Solution Verified - Updated -

Environment

  • Red Hat JBoss Operations Network (JON)
    • 3.2
    • 3.3
  • Snapshots generated by the JBoss ON storage node

Issue

  • Tools for managing snapshots
  • Storage node snapshots are taking a significant percentage of disk space.
  • Is there a tool for managing the snapshots?
  • How can I clear the data on the storage nodes manually?
  • Can I delete storage node snapshots directly?
  • Please advise how to deal with this incremental disk space usage issue
  • Disk Issue
  • How to archive rhq-data raw_metrics
  • We are planning to archive the raw_metrics folder which is having the rhq-raw_metrics-ic-xxx--Data.db files and snapshots folder which is consuming 105G of Disk space.Please advice how to clean the snapshots and how to clean them up to get the free disk space.
  • How do we purge the local metrics on the server, and are these not going to be moved to the database? I have about 200gb of data stored locally and we only monitor 6 servers.

Resolution

For JBoss ON 3.2 snapshots must be manually archived or deleted on a frequent basis. The JBoss ON system does not currently handle archiving or deleting old storage node snapshots. An enhancement request has been captured in Red Hat Bugzilla 1074633 asking that the snapshot creation process take into consideration old snapshots.

Additionally, a request for updated documentation has been captured in Red Hat Bugzilla 1098071. This request asks for improvements in the product documentation to instruct users of the snapshot process and how to manage them.

The following options are available to manually handle snapshots.

  • move snapshots older than a specific number of days to an alternate or archive location;
  • move all snapshots to an alternate or archive location;
  • delete snapshots older than a specific number of days;
  • move N number of snapshots -- from oldest to youngest -- to an alternate or archive location;
  • delete N number of snapshots -- from oldest to youngest;

The snapshot folder will contain one or more sub-folders with a name that represents the time stamp -- time in milliseconds since the epoch -- that they were created.

For JBoss ON 3.3, it supports the ability to schedule snapshots at desired intervals for storage nodes as well as configure the snapshot retention strategy right from the JBoss ON UI . For more information on how to schedule snapshots and configure retention strategies, please see the Storage Node Snapshots section of the Administration and Configuration Guide

Root Cause

Storage node snapshots are stored indefinitely until they are moved or deleted. The data represents a backup of the storage data at the time the snapshot was taken. In many cases, the snapshot files do not use additional disk space because they utilize file system hard links to their original content. However, over time, these files change due to data purge. This change will result in the snapshot becoming it own copy of the data and using up disk space.

Snapshots are created when weekly storage node and storage cluster maintenance is executed. Therefore, new snapshots will appear every week for each keyspace and its column data.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.