How to tell when Docker volume group is low on space

Solution Verified - Updated -

Environment

  • OpenShift Container Platform
  • Docker Container Runtime

Issue

  • It is possible that the volume group used to house docker images is filling up, but there's a need to verify that this is the issue.
  • We are using docker thinpool storage, and are unsure of how to read its usage as its mechanics are different than other filesystems.
  • We don't have Prometheus, Grafana, or similar monitoring installed in our cluster, and are looking to manually check usage.

Resolution

  • Use "docker info" command. This is the quickest, and should be run on each node with Docker runtime enabled to verify its accuracy.

  • Use the output from "lvs" command, and observe Data% and Meta% values:

  LV           VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  docker-pool  docker_vg  twi-aot--- 242.63g             21.92  14.22
  lv_openshift vg_ose     -wi-ao---- 100.00g
  lv_audit     vg_rh7ctho -wi-ao----   5.00g
  lv_home      vg_rh7ctho -wi-ao----  10.00g
  lv_log       vg_rh7ctho -wi-ao----  10.00g
  lv_root      vg_rh7ctho -wi-ao---- <60.36g
  lv_swap      vg_rh7ctho -wi-a-----   4.00g
  lv_tmp       vg_rh7ctho -wi-ao----  20.00g
  lv_var       vg_rh7ctho -wi-ao----  20.00g

Note that only the Docker thinpool has these values. Adding these percentages together will give you an approximate usage percentage.

Root Cause

"Thin pool" storage refers to a backing storage used by thin provisioning, allowing docker to allocate virtual storage for storing images. This is different than traditional fat provisioning, which uses a traditional storage model.

Using "vgs" to determine thin pool usage can be deceiving. For example:

  VG         #PV #LV #SN Attr   VSize    VFree
  docker_vg    1   1   0 wz--n- <245.00g    1.87g
  vg_ose       1   1   0 wz--n- <250.00g <150.00g
  vg_rh7ctho   1   7   0 wz--n- <129.50g  140.00m

At first glance, VFree appears to be the free space within the docker_vg above, but it's actually the free space outside of the docker_vg. The autoextend feature can obscure this.

If you are unsure of how to set up docker thin pool storage, please see this document.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments