Set container log limits when Docker is filling up /var/lib/docker with large log files.

Solution Verified - Updated -

Environment

Red Hat Enterprise Atomic Host 7
Red Hat Enterprise Linux 7
Red Hat OpenShift 3.X

Issue

  • How do I set log limits for container logs.
  • Docker is configured to use a thin pool logical volume for storage but is still filling up /var/lib/docker.
  • Running docker info shows plenty of data space available but my root file system is filling up with most space taken up in /var/lib/docker.
  • With OpenShift 3 I am seeing that docker is filling up space on /var/lib/docker.

Resolution

  • Container logs by default are not configured to be rotated or limited to a max size by docker. A log file can grow so large that it fills up the disk space if the container runs for long enough and generate enough logs.

See Docker Logging Documentation for more information on what can be set.

  • To set log limits for containers on a host --log-opt can be configured with max-size and max-file so that a containers logs are rolled over when they reach a max limit and only a certain number of files are saved before being discarded.
# cat /etc/sysconfig/docker 

OPTIONS='--insecure-registry=172.30.0.0/16 --signature-verification=false --selinux-enabled --log-opt max-size=50m --log-opt max-file=5'
  • Restart docker service for the changes to take effect.
# systemctl restart docker

If there is already a log file already present on the host that needs to be removed, run the following to clear its contents and reduce its size.

# cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log

OR

# cat /dev/null >  $(docker inspect --format='{{.LogPath}}'  CONTAINER_ID)

This will remove all logs for the given container

Diagnostic Steps

  • Generate a list of the largest files to confirm that the log files are using a large percent of the disk space.
# find /var/lib/docker/ -name "*.log" -exec ls -sh {} \; | sort -h -r | head -20
# du -aSh /var/lib/docker/ | sort -h -r | head -n 10

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments