Set container log limits when Docker is filling up /var/lib/docker with large log files.
Environment
Red Hat Enterprise Atomic Host 7
Red Hat Enterprise Linux 7
Red Hat OpenShift 3.X
Issue
- How do I set log limits for container logs.
- Docker is configured to use a thin pool logical volume for storage but is still filling up
/var/lib/docker. - Running
docker infoshows plenty of data space available but my root file system is filling up with most space taken up in/var/lib/docker. - With OpenShift 3 I am seeing that docker is filling up space on
/var/lib/docker.
Resolution
- Container logs by default are not configured to be rotated or limited to a max size by docker. A log file can grow so large that it fills up the disk space if the container runs for long enough and generate enough logs.
See Docker Logging Documentation for more information on what can be set.
- To set log limits for containers on a host
--log-optcan be configured withmax-sizeandmax-fileso that a containers logs are rolled over when they reach a max limit and only a certain number of files are saved before being discarded.
# cat /etc/sysconfig/docker
OPTIONS='--insecure-registry=172.30.0.0/16 --signature-verification=false --selinux-enabled --log-opt max-size=50m --log-opt max-file=5'
- Restart docker service for the changes to take effect.
# systemctl restart docker
If there is already a log file already present on the host that needs to be removed, run the following to clear its contents and reduce its size.
# cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log
OR
# cat /dev/null > $(docker inspect --format='{{.LogPath}}' CONTAINER_ID)
This will remove all logs for the given container
Diagnostic Steps
- Generate a list of the largest files to confirm that the log files are using a large percent of the disk space.
# find /var/lib/docker/ -name "*.log" -exec ls -sh {} \; | sort -h -r | head -20
# du -aSh /var/lib/docker/ | sort -h -r | head -n 10
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
6 Comments
The commands should be issued without the "-h" human readable flag otherwise log files that are GiB in size will not appear at the top sorted list
On RHEL 7, you can use the option
-hfor the commandsort. From man page:On a CentOS Atomic Host (CAH), the maximum size is not honored by Docker.
Environment:
[root@atomic1 log]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@atomic1 log]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) [root@atomic1 log]# docker version Client: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-59.el7.centos.x86_64 Go version: go1.6.3 Git commit: 3999ccb-unsupported Built: Thu Dec 15 17:24:43 2016 OS/Arch: linux/amd64
Server: Version: 1.10.3 API version: 1.22 Package version: docker-common-1.10.3-59.el7.centos.x86_64 Go version: go1.6.3 Git commit: 3999ccb-unsupported Built: Thu Dec 15 17:24:43 2016 OS/Arch: linux/amd64
cat /etc/sysconfig/docker
/etc/sysconfig/docker Modify these options if you want to change the way the docker daemon runsOPTIONS='--selinux-enabled --log-driver=journald --log-opt max-size=50m --log-opt max-file=5' if [ -z "${DOCKER_CERT_PATH}" ]; then DOCKER_CERT_PATH=/etc/docker fi
If you want to add your own registry to be used for docker search and docker pull use the ADD_REGISTRY option to list a set of registries, each prepended with --add-registry flag. The first registry added will be the first registry searched.ADD_REGISTRY='--add-registry registry.access.redhat.com --add-registry registry.centos.org --add-registry atomic1.eupraxialabs.com:5000'
If you want to block registries from being used, uncomment the BLOCK_REGISTRY option and give it a set of registries, each prepended with --block-registry flag. For example adding docker.io will stop users from downloading images from docker.io BLOCK_REGISTRY='--block-registry' If you have a registry secured with https but do not have proper certs distributed, you can tell docker to not look for full authorization by adding the registry to the INSECURE_REGISTRY line and uncommenting it. INSECURE_REGISTRY='--insecure-registry' On an SELinux system, if you remove the --selinux-enabled option, you also need to turn on the docker_transition_unconfined boolean. setsebool -P docker_transition_unconfined 1 Location used for temporary files, such as those created by docker load and build operations. Default is /var/lib/docker/tmp Can be overriden by setting the following environment variable. DOCKER_TMPDIR=/var/tmp Controls the /etc/cron.daily/docker-logrotate cron job status. To disable, uncomment the line below. LOGROTATE=false#
docker-latest daemon can be used by starting the docker-latest unitfile. To use docker-latest client, uncomment below line DOCKERBINARY=/usr/bin/docker-latestThe offending container (389ds in debug mode) is still growing without bound:
[root@atomic1 log]# ls -lash /var/lib/docker/containers/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e-json.log 129M -rw-r-----. 1 root root 74M Mar 12 10:09 /var/lib/docker/containers/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e/f3ad756523c80c6c396aa5efc32c7647d678b81743e9c05ccb1032921301a97e-json.log
This article should be changed to set --log-opt max-file=2 See https://access.redhat.com/solutions/4260051 for details.
Alternativley:
The truncate command can be used to zero size a file
Setting just:
in /etc/sysconfig/docker would have unexpected consequences, as the following options seem to set by default - but are lost when explictly settings OPTIONS
So I believe the setting should be made as
have asked RedHat to confirm
Hi Paul - did RH get back to you about these settings?