Docker service crash

Issue

• We've seen that in one of the nodes the docker service crashed and we need to login to the master itself and start the service in order to have the environment running properly.

The following messages were in the logs:

Oct 23 15:06:29 example docker: time="2015-10-23T15:06:29.905266859+02:00" level=info msg="GET /v1.19/containers/4ec3f115f1f47ba7e67b5c43b092eba66ea87b5ce210e13a03f2178926bfde9d/json"
Oct 23 15:06:29 example docker: time="2015-10-23T15:06:29.953866522+02:00" level=info msg="GET /v1.19/containers/abd2ac835ae42cbb04098b199adc609af10fe6dbdecb8b133742ff51408b030a/json"
Oct 23 15:06:31 example kernel: XFS (dm-10): Mounting V4 Filesystem
Oct 23 15:06:31 example kernel: XFS (dm-10): Ending clean mount
Oct 23 15:06:31 example docker: time="2015-10-23T15:06:31.949628542+02:00" level=error msg="Error from V2 registry: Driver devicemapper failed to create image rootfs f53919fe149ae85103e916a7e698706762b3ba2fb1a8cb58e656996668fcaec8: Unknown device 59a090c1706ed374f0e156e3414a39cd1c74dbf53b61328b1f2a0a58143261ce"
Oct 23 15:06:31 example openshift-node: W1023 15:06:31.950184   72602 manager.go:1618] Failed to pull image "library/mongo@sha256:703623552c5477d9eb7427ca4e4db65609618f8fc17841b814c547df6b113502" from pod "mongo-1-eecwy_isbanarqtec-dev" and container "mongo": pulling with digest reference failed from v2 registry
Oct 23 15:06:32 example docker: time="2015-10-23T15:06:31.978595647+02:00" level=info msg="GET /containers/json?all=1"
Oct 23 15:06:32 example docker: time="2015-10-23T15:06:32.099155690+02:00" level=info msg="GET /containers/92961652129bf0e29f03c9244ec7ff3761024b767a8a4070e9e0a2af116c68c7/json"
Oct 23 15:06:32 example docker: panic: runtime error: invalid memory address or nil pointer dereference
Oct 23 15:06:32 example docker: [signal 0xb code=0x1 addr=0x20 pc=0x6386fa]
Oct 23 15:06:32 example docker: goroutine 83495057 [running]:
Oct 23 15:06:32 example docker: runtime.gopanic(0xb9a440, 0xc20802a050)
Oct 23 15:06:32 example docker: /usr/lib/golang/src/runtime/panic.go:425 +0x2a3 fp=0xc22a9e78f8 sp=0xc22a9e7890
Oct 23 15:06:32 example docker: runtime.panicmem()
Oct 23 15:06:32 example docker: /usr/lib/golang/src/runtime/panic.go:42 +0x4e fp=0xc22a9e7920 sp=0xc22a9e78f8
Oct 23 15:06:32 example docker: runtime.sigpanic()
Oct 23 15:06:32 example docker: /usr/lib/golang/src/runtime/sigpanic_unix.go:26 +0x274 fp=0xc22a9e7970 sp=0xc22a9e7920
Oct 23 15:06:32 example docker: bufio.(*Writer).flush(0xc22aad4dc0, 0x0, 0x0)
Oct 23 15:06:32 example docker: /usr/lib/golang/src/bufio/bufio.go:530 +0xda fp=0xc22a9e7a08 sp=0xc22a9e7970
Oct 23 15:06:32 example docker: bufio.(*Writer).Flush(0xc22aad4dc0, 0x0, 0x0)
Oct 23 15:06:32 example docker: /usr/lib/golang/src/bufio/bufio.go:519 +0x3a fp=0xc22a9e7a28 sp=0xc22a9e7a08
Oct 23 15:06:32 example docker: net/http.(*response).Flush(0xc219b9f860)

If the Docker service crashed on a node, it did not automatically restart. As a result, we would need to log in to the master itself and manually start the service in order to have the environment running properly. The docker service should be automatically restarted after a failure.