DIrector root disk usage 97%

Issue

• Root partition of the director is at 97% and the highest usage is at /var/lib/containers/storage/overlay. We need to know how to safely delete any files to reclaim some space.

• Here we see 135GB are consumed by /var:

[root@director01 user]# du -hl / --max-depth=1
218M    /boot
84K     /dev
5.0G    /home
0       /proc
8.6M    /run
0       /sys
33M     /etc
294M    /root
**135G    /var**
0       /tmp
6.1G    /usr
0       /media
16G     /mnt
20K     /opt
10G     /srv
0       /log-files
187G    /backup
358G    /

• Going down the rabbit hole we can see 126GB are in /var/lib:

[root@director01 user]# du -hl /var --max-depth=1
0       /var/tmp
**126G    /var/lib**
6.7G    /var/log
0       /var/adm
1.5G    /var/cache
8.0K    /var/db
0       /var/empty
0       /var/games
0       /var/gopher
0       /var/local
0       /var/nis
4.0K    /var/opt
0       /var/preserve
9.2M    /var/spool
0       /var/yp
0       /var/kerberos
0       /var/crash
52K     /var/www
0       /var/agentx
0       /var/ftp
135G    /var

• If we look at file sizes, there're many huge files:

[root@director01 var]# du -a  | sort -n -r | head -n 20
132438608       .
124088596       ./lib
90135816        ./lib/containers
90135644        ./lib/containers/storage
90096984        ./lib/containers/storage/overlay
38500208        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63
20574052        ./lib/mysql
19587352        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/merged
18975808        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/merged/var
18913204        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/merged/var/log
18912848        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/diff
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/merged/var/log/ceilometer/ceilometer-polling.log
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/merged/var/log/ceilometer
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/diff/var/log/ceilometer/ceilometer-polling.log
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/diff/var/log/ceilometer
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/diff/var/log
18912800        ./lib/containers/storage/overlay/a88672cf53b6ce34c69649685578e7cfcc5930b5e4ec348c24b90bc7d0563e63/diff/var
13426056        ./lib/mysql/ceilometer
11628556        ./lib/mysql/ceilometer/sample.ibd
6815204 ./log