multiple compute node docker service stopped after server reboot

Solution In Progress - Updated -

Issue

  • We have rebooted the overcloud computes and observed that hypervisor state was down.

  • After logging into compute observed that docker services are not running :

[root@overcloud-compute-14 tmp]# systemctl | grep docker 
  var-lib-docker-containers.mount                                                                  loaded active mounted   /var/lib/docker/containers
  var-lib-docker-overlay2.mount                                                                    loaded active mounted   /var/lib/docker/overlay2
● docker-storage-setup.service                                                                     loaded failed failed    Docker Storage Setup
  docker-cleanup.timer                                                                             loaded active waiting   Run docker-cleanup every hour
[root@overcloud-compute-14 tmp]# systemctl status docker.service 
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─99-unset-mountflags.conf
   Active: inactive (dead) (Result: signal) since Thu 2019-11-28 20:00:07 UTC; 2min 48s ago
     Docs: http://docs.docker.com
  Process: 20842 ExecStart=/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --authorization-plugin=rhel-push-plugin --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_NETWORK_OPTIONS $ADD_REGISTRY $BLOCK_REGISTRY $INSECURE_REGISTRY $REGISTRIES (code=killed, signal=ABRT)
 Main PID: 20842 (code=killed, signal=ABRT)

Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc42022a000, 0...4208deae0)
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...c=0xd12c6b
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: runtime.goexit()
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42031d7c0 sp=0...c=0x46e231
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
Nov 28 20:00:07 overcloud-compute-14 dockerd-current[20842]: /builddir/build/BUILD/docker-7f2769b9e0572f62730d91e79e674efd59b7e234/_build/src/github.com/docker/docker/vendor/g...122 +0x1e4
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: docker.service holdoff time over, scheduling restart.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Stopped Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Dependency failed for Docker Application Container Engine.
Nov 28 20:00:07 overcloud-compute-14 systemd[1]: Job docker.service/start failed with result 'dependency'.
Hint: Some lines were ellipsized, use -l to show in full.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content