multiple compute node docker service stopped after server reboot

Solution In Progress - Updated -

Issue

  • We have rebooted the overcloud computes and observed that hypervisor state was down.

  • After logging into compute observed that docker services are not running :

[root@overcloud-compute-14 tmp]# systemctl | grep docker 
  var-lib-docker-containers.mount                                                                  loaded active mounted   /var/lib/docker/containers
  var-lib-docker-overlay2.mount                                                                    loaded active mounted   /var/lib/docker/overlay2
● docker-storage-setup.service                                                                     loaded failed failed    Docker Storage Setup
  docker-cleanup.timer                                                                             loaded active waiting   Run docker-cleanup every hour
[root@overcloud-compute-14 tmp]# systemctl status docker.service 
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─99-unset-mountflags.conf
   Active: inactive (dead) (Result: signal) since Thu 2019-11-28 20:00:07 UTC; 2min 48s ago
     Docs: http://docs.docker.com
  Process: 20842 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=killed, signal=ABRT)
 Main PID: 20842 (code=killed, signal=ABRT)

Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc42022a000, 0...4208deae0)
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...c=0xd12c6b
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: runtime.goexit()
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42031d7c0 sp=0...c=0x46e231
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...122 +0x1e4
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: docker.service holdoff time over, scheduling restart.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Stopped Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Dependency failed for Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Job docker.service/start failed with result 'dependency'.
Hint: Some lines were ellipsized, use -l to show in full.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In