SUMMARY Simple method to automate cleanup of forgotten LVM snapshots

Latest response

Hi,

This is shared with the forum members so that someone might benefit if they have similar problem.

Scenario: IT Operators often forget to clean up obsolete LVM snapshots.

To avoid it, I set up simple method with system timer:

Create systemd service that operates like cron job but with much more flexibility and logging capabilities. I wanted to run LVM snapshot checks every day at 08:35 hours…

Full configuration:

a) Create Shell script /usr/local/bin/delete-lvm-snapshots.sh

#!/bin/sh

PATH=/sbin:/usr/bin:/usr/sbin:/bin; export PATH
VGNAME="myos"

cutoff=$(date -d '4 days ago' +%s)

lvs --nosuffix -o lv_name,lv_time --noheadings os | egrep _snap | while read -r lvname lvcrdate 
do
   age=$(date -d "$lvcrdate" +%s)
   echo "INFO: Snapshot $lvname was created on $lvcrdate"
   logger "INFO: Snapshot $lvname was created on $lvcrdate"
   if (($age < $cutoff))
   then
      echo "WARN: Snapshot $lvname is older than 4 days"
      logger "WARN: Snapshot $lvname is older than 4 days"
      lvremove -f ${VGNAME}/${lvname}
      if [ $? -eq 0 ]
      then
         echo "INFO: Snapshot $lvname removed successfully"
         logger "INFO: Snapshot $lvname removed successfully"
      else
         echo "INFO: Removal of snapshot $lvname failed"
         logger "INFO: Removal of snapshot $lvname failed"
      fi
   else
      echo "INFO: Snapshot $lvname not removed because it is not older than 4 days "
      logger "INFO: Snapshot $lvname not removed because it is not older than 4 days "
   fi
done

exit 0

b) Create systemd service /etc/systemd/system/lvm-snapshots.service

[Unit]
Description=Check status of LVM snapshots daily (cron-like service)

[Service]
Type=simple
ExecStart=/usr/local/bin/delete-lvm-snapshots.sh

[Install]
WantedBy=multi-user.target

c) Create systemd timer /etc/systemd/system/lvm-snapshots.timer

[Unit]
Description=Execute LVM snapshot checks every day at 08:35 hours 

[Timer]
OnCalendar=*-*-* 08:35:00
Unit=lvm-snapshots.service

[Install]
WantedBy=multi-user.target

d) Enable systemd service and timer:

# systemctl enable lvm-snapshots.service && systemctl start lvm-snapshots.service

# systemctl enable lvm-snapshots.timer && systemctl start lvm-snapshots.timer

e) Verify the status of the timer (run at around four minutes before 08:30 hours):

# systemctl list-timers lvm-snapshot*
NEXT                          LEFT          LAST PASSED UNIT                ACTIVATES
Thu 2020-01-30 08:35:00 AEDT  3min 37s left n/a  n/a    lvm-snapshots.timer lvm-snapshots.service

1 timers listed.

f) Check the execution of the timer soon after 08:35 hours:

# systemctl status lvm-snapshots.service -l
   lvm-snapshots.service - Check status of LVM snapshots daily (cron-like service)
   Loaded: loaded (/etc/systemd/system/lvm-snapshots.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Thu 2020-01-30 08:35:00 AEDT; 7min ago
  Process: 104271 ExecStart=/usr/local/bin/delete-lvm-snapshots.sh (code=exited, status=0/SUCCESS)
Main PID: 104271 (code=exited, status=0/SUCCESS)

Jan 30 08:35:00 myhost.domain.dom systemd[1]: Started Check status of LVM snapshots daily (cron-like service).
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh[104271]: INFO: Snapshot home_snap was created on 2020-01-30 08:22:21 +1100
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh[104271]: INFO: Snapshot home_snap not removed because it is not older than 4 days
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh[104271]: INFO: Snapshot var_snap was created on 2020-01-29 12:15:01 +1100
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh[104271]: INFO: Snapshot var_snap not removed because it is not older than 4 days

In addition, syslog also reports the event:

Jan 30 08:30:13 myhost.domain.dom systemd: Started Execute LVM snapshot checks every day at 08:35 hours.
Jan 30 08:35:00 myhost.domain.dom systemd: Started Check status of LVM snapshots daily (cron-like service).
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh: INFO: Snapshot home_snap was created on 2020-01-30 08:22:21 +1100
Jan 30 08:35:00 myhost.domain.dom root: INFO: Snapshot home_snap was created on 2020-01-30 08:22:21 +1100
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh: INFO: Snapshot home_snap not removed because it is not older than 4 days
Jan 30 08:35:00 myhost.domain.dom root: INFO: Snapshot home_snap not removed because it is not older than 4 days
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh: INFO: Snapshot var_snap was created on 2020-01-29 12:15:01 +1100
Jan 30 08:35:00 myhost.domain.dom root: INFO: Snapshot var_snap was created on 2020-01-29 12:15:01 +1100
Jan 30 08:35:00 myhost.domain.dom delete-lvm-snapshots.sh: INFO: Snapshot var_snap not removed because it is not older than 4 days
Jan 30 08:35:00 myhost.domain.dom root: INFO: Snapshot var_snap not removed because it is not older than 4 days

g) Check when the next LVM snapshot clean-up will run:

# systemctl list-timers lvm-snapshot*
NEXT                          LEFT     LAST                          PASSED  UNIT                ACTIVATES
Fri 2020-01-31 08:35:00 AEDT  23h left Thu 2020-01-30 08:35:00 AEDT  49s ago lvm-snapshots.timer lvm-snapshots.service

1 timers listed.

Best wishes,

Dusan Baljevic (amateur radio VK2COT)

Dusan Baljevic's picture

Responses