Chapter 12. Rebooting Nodes

Some situations require a reboot of nodes in the undercloud and overcloud. The following procedures show how to reboot different node types. Be aware of the following notes:

  • If rebooting all nodes in one role, it is advisable to reboot each node individually. This helps retain services for that role during the reboot.
  • If rebooting all nodes in your OpenStack Platform environment, use the following list to guide the reboot order:

Recommended Node Reboot Order

  1. Reboot the director
  2. Reboot Controller and other composable nodes
  3. Reboot Ceph Storage nodes
  4. Reboot Compute nodes

12.1. Rebooting the undercloud node

The following procedure reboots the undercloud node.

Procedure

  1. Log into the undercloud as the stack user.

  2. Reboot the undercloud: 

    $ sudo reboot
  3. Wait until the node boots.

12.2. Rebooting controller and composable nodes

The following procedure reboots controller nodes and standalone nodes based on composable roles. This excludes Compute nodes and Ceph Storage nodes.

Procedure

  1. Select a node to reboot. Log into it and stop the cluster before rebooting: 

    [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

  2. Reboot the node: 

    [heat-admin@overcloud-controller-0 ~]$ sudo reboot
  3. Wait until the node boots.

  4. Re-enable the cluster for the node: 

    [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster start

  5. Log into the node and check the services. For example:

    1. If the node uses Pacemaker services, check the node has rejoined the cluster: 

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

    2. If the node uses Systemd services, check all services are enabled: 

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

12.3. Rebooting a Ceph Storage (OSD) cluster

The following procedure reboots a cluster of Ceph Storage (OSD) nodes.

Procedure

  1. Log into a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily: 

    $ sudo ceph osd set noout
$ sudo ceph osd set norebalance
  2. Select the first Ceph Storage node to reboot and log into it.

  3. Reboot the node: 

    $ sudo reboot
  4. Wait until the node boots.

  5. Log into the node and check the cluster status: 

    $ sudo ceph -s

    Check that the pgmap reports all pgs as normal (active+clean).

  6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

  7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again: 

    $ sudo ceph osd unset noout
$ sudo ceph osd unset norebalance

  8. Perform a final status check to verify the cluster reports HEALTH_OK: 

    $ sudo ceph status

12.4. Rebooting compute nodes

The following procedure reboots Compute nodes. To ensure minimal downtime of instances in your OpenStack Platform environment, this procedure also includes instructions on migrating instances from the chosen Compute node. This involves the following workflow:

  • Select a Compute node to reboot and disable it so that it does not provision new instances
  • Migrate the instances to another Compute node
  • Reboot the empty Compute node and enable it

Procedure

  1. Log into the undercloud as the stack user.

  2. List all Compute nodes and their UUIDs: 

    $ source ~/stackrc
(undercloud) $ openstack server list --name compute

    Identify the UUID of the Compute node you aim to reboot.

  3. From the undercloud, select a Compute Node and disable it: 

    $ source ~/overcloudrc
(overcloud) $ openstack compute service list
(overcloud) $ openstack compute service set [hostname] nova-compute --disable

  4. List all instances on the Compute node: 

    (overcloud) $ openstack server list --host [hostname] --all-projects

  5. Use one of the following commands to migrate your instances:

    1. Migrate the instance to a specific host of your choice: 

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

    2. Let nova-scheduler automatically select the target host: 

      (overcloud) $ nova live-migration [instance-id]

    3. Live migrate all instances at once: 

      $ nova host-evacuate-live [hostname]
      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

  6. Wait until migration completes.

  7. Confirm the migration was successful: 

    (overcloud) $ openstack server list --host [hostname] --all-projects
  8. Continue migrating instances until none remain on the chosen Compute Node.

  9. Log into the Compute Node and reboot it: 

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot
  10. Wait until the node boots.

  11. Enable the Compute Node again: 

    $ source ~/overcloudrc
(overcloud) $ openstack compute service set [hostname] nova-compute --enable

  12. Check whether the Compute node is enabled: 

    (overcloud) $ openstack compute service list