Set container log limits when Docker is filling up /var/lib/docker with large log files.

Solution Verified - Updated -

Environment

Red Hat Enterprise Atomic Host 7
Red Hat Enterprise Linux 7
Red Hat OpenShift 3.X

Issue

  • How do I set log limits for container logs.
  • Docker is configured to use a thin pool logical volume for storage but is still filling up /var/lib/docker.
  • Running docker info shows plenty of data space available but my root file system is filling up with most space taken up in /var/lib/docker.
  • With OpenShift 3 I am seeing that docker is filling up space on /var/lib/docker.

Resolution

  • Container logs by default are not configured to be rotated or limited to a max size by docker. A log file can grow so large that it fills up the disk space if the container runs for long enough and generate enough logs.

See Docker Logging Documentation for more information on what can be set.

  • To set log limits for containers on a host --log-opt can be configured with max-size and max-file so that a containers logs are rolled over when they reach a max limit and only a certain number of files are saved before being discarded.
# cat /etc/sysconfig/docker 

OPTIONS='--insecure-registry=172.30.0.0/16 --selinux-enabled --log-opt max-size=50m --log-opt max-file=5'
  • Restart docker service for the changes to take effect.
# systemctl restart docker 

If there is already a log file already present on the host that needs to be removed, run the following to clear its contents and reduce its size.

# cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log

OR

# cat /dev/null >  $(docker inspect --format='{{.LogPath}}'  CONTAINER_ID)

This will remove all logs for the given container

Diagnostic Steps

  • Generate a list of the largest files to confirm that the log files are using a large percent of the disk space.
# find /var/lib/docker/ -name "*.log" -exec ls -sh {} \; | sort -n -r | head -20
# du -aSh /var/lib/docker/ | sort -n -r | head -n 10

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

2 Comments

The commands should be issued without the "-h" human readable flag otherwise log files that are GiB in size will not appear at the top sorted list

# find /var/lib/docker/ -name "*.log" -exec ls -s {} \; | sort -n -r | head -20
# du -aS /var/lib/docker/ | sort -n -r | head -n 10

On a CentOS Atomic Host (CAH), the maximum size is not honored by Docker.

Environment:

[root@atomic1 log]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@atomic1 log]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@atomic1 log]# docker version Client: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-59.el7.centos.x86_64 Go version: go1.6.3 Git commit: 3999ccb-unsupported Built: Thu Dec 15 17:24:43 2016 OS/Arch: linux/amd64

Server: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-59.el7.centos.x86_64 Go version: go1.6.3 Git commit: 3999ccb-unsupported Built: Thu Dec 15 17:24:43 2016 OS/Arch: linux/amd64

cat /etc/sysconfig/docker

/etc/sysconfig/docker Modify these options if you want to change the way the docker daemon runs

OPTIONS='--selinux-enabled --log-driver=journald --log-opt max-size=50m --log-opt max-file=5' if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi

If you want to add your own registry to be used for docker search and docker pull use the ADD_REGISTRY option to list a set of registries, each prepended with --add-registry flag. The first registry added will be the first registry searched.

ADD_REGISTRY='--add-registry registry.access.redhat.com --add-registry registry.centos.org --add-registry atomic1.eupraxialabs.com:5000'

If you want to block registries from being used, uncomment the BLOCK_REGISTRY option and give it a set of registries, each prepended with --block-registry flag. For example adding docker.io will stop users from downloading images from docker.io BLOCK_REGISTRY='--block-registry' If you have a registry secured with https but do not have proper certs distributed, you can tell docker to not look for full authorization by adding the registry to the INSECURE_REGISTRY line and uncommenting it. INSECURE_REGISTRY='--insecure-registry' On an SELinux system, if you remove the --selinux-enabled option, you also need to turn on the docker_transition_unconfined boolean. setsebool -P docker_transition_unconfined 1 Location used for temporary files, such as those created by docker load and build operations. Default is /var/lib/docker/tmp Can be overriden by setting the following environment variable. DOCKER_TMPDIR=/var/tmp Controls the /etc/cron.daily/docker-logrotate cron job status. To disable, uncomment the line below. LOGROTATE=false

#

docker-latest daemon can be used by starting the docker-latest unitfile. To use docker-latest client, uncomment below line DOCKERBINARY=/usr/bin/docker-latest

The offending container (389ds in debug mode) is still growing without bound:

[root@atomic1 log]# ls -lash /var/lib/docker/containers/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e-json.log 129M -rw-r-----. 1 root root 74M Mar 12 10:09 /var/lib/docker/containers/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e-json.log