Openshift Upgrade Playbook fails when restarting node

Solution Verified - Updated -

Issue

  • When starting the upgrade playbook, the play restart node fails with an error message.
  • At this time, the atomic-openshift-node service on the first master has been tried to restart but fails to come up.
  • The logs shows:
Oct 18 11:41:46 node.example.com atomic-openshift-node[105325]: E1018 11:41:46.091066  105325 raw.go:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice/var-lib-docker-devicemapper-mnt-876aecbd6f58a637905da37ab34076ca55bd0950dbe0b5f91323e5d8937955c8.mount": inotify_add_watch /sys/fs/cgroup/memory/system.slice/var-lib-docker-devicemapper-mnt-876aecbd6f58a637905da37ab34076ca55bd0950dbe0b5f91323e5d8937955c8.mount: no space left on device
Oct 18 11:41:46 node.example.com atomic-openshift-node[105325]: E1018 11:41:46.091166  105325 raw.go:140] Failed to watch directory "/sys/fs/cgroup/memory/system.slice": inotify_add_watch /sys/fs/cgroup/memory/system.slice/var-lib-docker-devicemapper-mnt-876aecbd6f58a637905da37ab34076ca55bd0950dbe0b5f91323e5d8937955c8.mount: no space left on device
Oct 18 11:41:46 node.example.com atomic-openshift-node[105325]: F1018 11:41:46.091211  105325 kubelet.go:1352] Failed to start cAdvisor inotify_add_watch /sys/fs/cgroup/memory/system.slice/var-lib-docker-devicemapper-mnt-876aecbd6f58a637905da37ab34076ca55bd0950dbe0b5f91323e5d8937955c8.mount: no space left on device
Oct 18 11:41:46 node.example.com systemd[1]: atomic-openshift-node.service: main process exited, code=exited, status=255/n/a
Oct 18 11:41:46 node.example.com systemd[1]: Unit atomic-openshift-node.service entered failed state.
Oct 18 11:41:46 node.example.com systemd[1]: atomic-openshift-node.service failed.

Environment

  • OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.