Chapter 4. Rebooting the overcloud

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

2. Select the first Ceph Storage node to reboot and log into the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Check the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node you want to reboot.

3. From the undercloud, select a Compute Node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

5. If you decided not to migrate instances, skip to this step.

6. If you decided to migrate the instances to another Compute node, use one of the following commands:

   1. Migrate the instance to a different host:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Continue migrating instances until none remain on the chosen Compute Node.

10. Log in to the Compute Node. Reboot the node:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

11. Wait until the node boots.

12. Enable the Compute Node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

13. Check whether the Compute node is enabled:

    (overcloud) $ openstack compute service list

Procedure

1. Log in to a Ceph MON or a Controller node and identify the name of the Ceph MON container:

   $ sudo podman ps | grep -i ceph | grep -i mon
   
   45fe68d340e5  docker-registry.upshift.redhat.com/ceph/rhceph-4.0-rhel8:latest

2. Set the CEPH_MON_CONTAINER variable to the name of the container:

   $ CEPH_MON_CONTAINER=ceph-mon-controller-0

3. Verify that you can use the CEPH_MON_CONTAINER variable to run Ceph commands:

   $ sudo podman exec $CEPH_MON_CONTAINER ceph -s

4. From the Ceph MON or Controller node, disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec $CEPH_MON_CONTAINER ceph osd set noout
   $ sudo podman exec $CEPH_MON_CONTAINER ceph osd set norebalance

5. Log in to the undercloud as the stack user.

6. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node you aim to reboot.

7. From the undercloud, select a Compute node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

8. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Use one of the following commands to migrate your instances:

   1. Migrate the instance to a specific host of your choice:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

10. Wait until the migration completes.

11. Confirm that the migration was successful:

    (overcloud) $ openstack server list --host [hostname] --all-projects

12. Continue migrating instances until none remain on the chosen Compute node.

13. Log in to a Ceph MON or a Controller node and check the cluster status:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph -s

    Check that the pgmap reports all pgs as normal (active+clean).

14. Reboot the Compute HCI node:

    $ sudo reboot

15. Wait until the node boots.

16. Enable the Compute node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

17. Verify that the Compute node is enabled:

    (overcloud) $ openstack compute service list

18. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

19. When complete, log in to a Ceph MON or Controller node and enable cluster rebalancing again:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph osd unset noout
    $ sudo podman exec $CEPH_MON_CONTAINER ceph osd unset norebalance

20. Perform a final status check to verify the cluster reports HEALTH_OK:

    $ sudo podman exec $CEPH_MON_CONTAINER ceph status