- OpenShift Container Platform
- Docker Container Runtime
- It is possible that the volume group used to house docker images is filling up, but there's a need to verify that this is the issue.
- We are using docker thinpool storage, and are unsure of how to read its usage as its mechanics are different than other filesystems.
- We don't have Prometheus, Grafana, or similar monitoring installed in our cluster, and are looking to manually check usage.
Use "docker info" command. This is the quickest, and should be run on each node with Docker runtime enabled to verify its accuracy.
Use the output from "lvs" command, and observe Data% and Meta% values:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert docker-pool docker_vg twi-aot--- 242.63g 21.92 14.22 lv_openshift vg_ose -wi-ao---- 100.00g lv_audit vg_rh7ctho -wi-ao---- 5.00g lv_home vg_rh7ctho -wi-ao---- 10.00g lv_log vg_rh7ctho -wi-ao---- 10.00g lv_root vg_rh7ctho -wi-ao---- <60.36g lv_swap vg_rh7ctho -wi-a----- 4.00g lv_tmp vg_rh7ctho -wi-ao---- 20.00g lv_var vg_rh7ctho -wi-ao---- 20.00g
Note that only the Docker thinpool has these values. Adding these percentages together will give you an approximate usage percentage.
"Thin pool" storage refers to a backing storage used by thin provisioning, allowing docker to allocate virtual storage for storing images. This is different than traditional fat provisioning, which uses a traditional storage model.
Using "vgs" to determine thin pool usage can be deceiving. For example:
VG #PV #LV #SN Attr VSize VFree docker_vg 1 1 0 wz--n- <245.00g 1.87g vg_ose 1 1 0 wz--n- <250.00g <150.00g vg_rh7ctho 1 7 0 wz--n- <129.50g 140.00m
At first glance, VFree appears to be the free space within the docker_vg above, but it's actually the free space outside of the docker_vg. The autoextend feature can obscure this.
If you are unsure of how to set up docker thin pool storage, please see this document.
- Red Hat OpenShift Container Platform
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.