multiple compute node docker service stopped after server reboot
Issue
-
We have rebooted the overcloud computes and observed that hypervisor state was down.
-
After logging into compute observed that docker services are not running :
[root@overcloud-compute-14 tmp]# systemctl | grep docker
var-lib-docker-containers.mount loaded active mounted /var/lib/docker/containers
var-lib-docker-overlay2.mount loaded active mounted /var/lib/docker/overlay2
● docker-storage-setup.service loaded failed failed Docker Storage Setup
docker-cleanup.timer loaded active waiting Run docker-cleanup every hour
[root@overcloud-compute-14 tmp]# systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─99-unset-mountflags.conf
Active: inactive (dead) (Result: signal) since Thu 2019-11-28 20:00:07 UTC; 2min 48s ago
Docs: http://docs.docker.com
Process: 20842 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=killed, signal=ABRT)
Main PID: 20842 (code=killed, signal=ABRT)
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc42022a000, 0...4208deae0)
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...c=0xd12c6b
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: runtime.goexit()
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42031d7c0 sp=0...c=0x46e231
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...122 +0x1e4
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: docker.service holdoff time over, scheduling restart.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Stopped Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Dependency failed for Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Job docker.service/start failed with result 'dependency'.
Hint: Some lines were ellipsized, use -l to show in full.
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.