Chapter 19. Shutting down and starting up the undercloud and overcloud

If you must perform maintenance on the undercloud and overcloud, you must shut down and start up the undercloud and overcloud nodes in a specific order to ensure minimal issues when your start your overcloud.

19.1. Prerequisites

  • A running undercloud and overcloud

19.2. Undercloud and overcloud shutdown order

To shut down the Red Hat OpenStack Platform environment, you must shut down the overcloud and undercloud in the following order:

  1. Shut down instances on overcloud Compute nodes
  2. Shut down Compute nodes
  3. Stop all high availability and OpenStack Platform services on Controller nodes
  4. Shut down Ceph Storage nodes
  5. Shut down Controller nodes
  6. Shut down the undercloud

19.3. Shutting down instances on overcloud Compute nodes

As a part of shutting down the Red Hat OpenStack Platform environment, shut down all instances on Compute nodes before shutting down the Compute nodes.

Prerequisites

  • An overcloud with active Compute services

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the credentials file for your overcloud:

    $ source ~/overcloudrc
  3. View running instances in the overcloud:

    $ openstack server list --all-projects
  4. Stop each instance in the overcloud:

    $ openstack server stop <INSTANCE>

    Repeat this step for each instance until you stop all instances in the overcloud.

19.4. Shutting down Compute nodes

As a part of shutting down the Red Hat OpenStack Platform environment, log in to and shut down each Compute node.

Prerequisites

  • Shut down all instances on the Compute nodes

Procedure

  1. Log in as the root user to a Compute node.
  2. Shut down the node:

    # shutdown -h now
  3. Perform these steps for each Compute node until you shut down all Compute nodes.

19.5. Stopping services on Controller nodes

As a part of shutting down the Red Hat OpenStack Platform environment, stop services on the Controller nodes before shutting down the nodes. This includes Pacemaker and systemd services.

Prerequisites

  • An overcloud with active Pacemaker services

Procedure

  1. Log in as the root user to a Controller node.
  2. Stop the Pacemaker cluster.

    # pcs cluster stop --all

    This command stops the cluster on all nodes.

  3. Wait until the Pacemaker services stop and check that the services stopped.

    1. Check the Pacemaker status:

      # pcs status
    2. Check that no Pacemaker services are running in Podman:

      # podman ps --filter "name=.*-bundle.*"
  4. Stop the Red Hat OpenStack Platform services:

    # systemctl stop 'tripleo_*'
  5. Wait until the services stop and check that services are no longer running in Podman:

    # podman ps

19.6. Shutting down Ceph Storage nodes

As a part of shutting down the Red Hat OpenStack Platform environment, disable Ceph Storage services then log in to and shut down each Ceph Storage node.

Prerequisites

  • A healthy Ceph Storage cluster
  • Ceph MON services are running on standalone Ceph MON nodes or on Controller nodes

Procedure

  1. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.
  2. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

    # sudo podman exec -it ceph-mon-controller-0 ceph status

    Ensure that the status is HEALTH_OK.

  3. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags for the cluster. In the following example, the podman commands set these flags through a Ceph MON container on a Controller node:

    # sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
    # sudo podman exec -it ceph-mon-controller-0 ceph osd set norecover
    # sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance
    # sudo podman exec -it ceph-mon-controller-0 ceph osd set nobackfill
    # sudo podman exec -it ceph-mon-controller-0 ceph osd set nodown
    # sudo podman exec -it ceph-mon-controller-0 ceph osd set pause
  4. Shut down each Ceph Storage node:

    1. Log in as the root user to a Ceph Storage node.
    2. Shut down the node:

      # shutdown -h now
    3. Perform these steps for each Ceph Storage node until you shut down all Ceph Storage nodes.
  5. Shut down any standalone Ceph MON nodes:

    1. Log in as the root user to a standalone Ceph MON node.
    2. Shut down the node:

      # shutdown -h now
    3. Perform these steps for each standalone Ceph MON node until you shut down all standalone Ceph MON nodes.

19.7. Shutting down Controller nodes

As a part of shutting down the Red Hat OpenStack Platform environment, log in to and shut down each Controller node.

Prerequisites

  • Stop the Pacemaker cluster
  • Stop all Red Hat OpenStack Platform services on the Controller nodes

Procedure

  1. Log in as the root user to a Controller node.
  2. Shut down the node:

    # shutdown -h now
  3. Perform these steps for each Controller node until you shut down all Controller nodes.

19.8. Shutting down the undercloud

As a part of shutting down the Red Hat OpenStack Platform environment, log in to the undercloud node and shut down the undercloud.

Prerequisites

  • A running undercloud

Procedure

  1. Log in to the undercloud as the stack user.
  2. Shut down the undercloud:

    $ sudo shutdown -h now

19.9. Performing system maintenance

After you completely shut down the undercloud and overcloud, perform any maintenance to the systems in your environment and then start up the undercloud and overcloud.

19.10. Undercloud and overcloud startup order

To start the Red Hat OpenStack Platform environment, you must start the undercloud and overcloud in the following order:

  1. Start the undercloud
  2. Start Controller nodes
  3. Start Ceph Storage nodes
  4. Start high availability services on the Controller nodes
  5. Start Compute nodes
  6. Start instances on overcloud Compute nodes

19.11. Starting the undercloud

As a part of starting the Red Hat OpenStack Platform environment, power on the undercloud node, log in to the undercloud, and check the undercloud services.

Prerequisites

  • A powered down undercloud

Procedure

  1. Power on the undercloud and wait until the undercloud boots.

Verification

  1. Log in to the undercloud as the stack user.
  2. Check the services on the undercloud:

    $ systemctl list-units 'tripleo_*'
  3. Source the credential file for your undercloud and run the validation command to check that all services and containers are active and healthy.

    $ source stackrc
    $ openstack tripleo validator run --validation service-status --limit undercloud

Additional resources

19.12. Starting Controller nodes

As a part of starting the Red Hat OpenStack Platform environment, power on each Controller node and check the non-Pacemaker services on the node.

Prerequisites

  • Powered down Controller nodes

Procedure

  1. Power on each Controller node.

Verification

  1. Log in to each Controller as the root user.
  2. Check the services on the Controller node:

    $ systemctl -t service

    Only non-Pacemaker based services are running.

19.13. Starting Ceph Storage nodes

As a part of starting the Red Hat OpenStack Platform environment, power on the Ceph MON and Ceph Storage nodes and enable Ceph Storage services.

Prerequisites

  • A powered down Ceph Storage cluster
  • Ceph MON services are enabled on powered down standalone Ceph MON nodes or on powered on Controller nodes

Procedure

  1. If your environment has standalone Ceph MON nodes, power on each Ceph MON node.
  2. Power on each Ceph Storage node.
  3. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.
  4. Check the status of the cluster nodes. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

    # sudo podman exec -it ceph-mon-controller-0 ceph status

    Ensure that each node is powered on and connected.

  5. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags for the cluster. In the following example, the podman commands unset these flags through a Ceph MON container on a Controller node:

    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norecover
    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance
    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nobackfill
    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nodown
    # sudo podman exec -it ceph-mon-controller-0 ceph osd unset pause

Verification

  1. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

    # sudo podman exec -it ceph-mon-controller-0 ceph status

    Ensure the status is HEALTH_OK.

19.14. Starting the Pacemaker cluster on Controller nodes

As a part of starting the Red Hat OpenStack Platform environment, start the Pacemaker cluster on the Controller nodes. This process enables high availability services, such as the database, messaging, load balancing, and VIP assignment.

Prerequisites

  • An overcloud with running Controller nodes

Procedure

  1. Log in as the root user to a Controller node.
  2. Start the Pacemaker cluster.

    # pcs cluster start --all

    This command starts the cluster on all nodes.

  3. Wait until the Pacemaker services start and check that the services started.

    1. Check the Pacemaker status:

      # pcs status
    2. Check that the Pacemaker services are running in Podman:

      # podman ps --filter "name=.*-bundle.*"

19.15. Starting Compute nodes

As a part of starting the Red Hat OpenStack Platform environment, power on each Compute node and check the services on the node.

Prerequisites

  • Powered down Compute nodes

Procedure

  1. Power on each Compute node.

Verification

  1. Log in to each Compute as the root user.
  2. Check the services on the Compute node:

    $ systemctl -t service

19.16. Starting instances on overcloud Compute nodes

As a part of starting the Red Hat OpenStack Platform environment, start the instances on on Compute nodes.

Prerequisites

  • An active overcloud with active nodes

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the credentials file for your overcloud:

    $ source ~/overcloudrc
  3. View running instances in the overcloud:

    $ openstack server list --all-projects
  4. Start an instance in the overcloud:

    $ openstack server start <INSTANCE>