Procedure

1. To pull the images directly from registry.redhat.io in your overcloud deployment, an environment file is required to specify the image parameters. Run the following command to generate the container image environment file:

   (undercloud) $ sudo openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

2. Modify the overcloud_images.yaml file and include the following parameters to authenticate with registry.redhat.io during deployment:

   ContainerImageRegistryLogin: true
   ContainerImageRegistryCredentials:
     registry.redhat.io:
       <USERNAME>: <PASSWORD>

   • Replace <USERNAME> and <PASSWORD> with your credentials for registry.redhat.io.

     The overcloud_images.yaml file contains the image locations on the undercloud. Include this file with your deployment.

     Note

     Before you run the openstack overcloud deploy command, you must log in to the remote registry:

     (undercloud) $ sudo docker login registry.redhat.io

Procedure

1. Find the address of the local undercloud registry. The address uses the following pattern:

   <REGISTRY_IP_ADDRESS>:8787

   Use the IP address of your undercloud, which you previously set with the local_ip parameter in your undercloud.conf file. For the commands below, the address is assumed to be 192.168.24.1:8787.

2. Log in to registry.redhat.io:

   (undercloud) $ docker login registry.redhat.io --username $RH_USER --password $RH_PASSWD

3. Create a template to upload the images to the local registry, and the environment file to refer to those images:

   (undercloud) $ openstack overcloud container image prepare \
     --namespace=registry.redhat.io/rhosp13 \
     --push-destination=192.168.24.1:8787 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml \
     --output-images-file /home/stack/local_registry_images.yaml

   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.

4. Verify that the following two files have been created:

   • local_registry_images.yaml, which contains container image information from the remote source. Use this file to pull the images from the Red Hat Container Registry (registry.redhat.io) to the undercloud.
   • overcloud_images.yaml, which contains the eventual image locations on the undercloud. You include this file with your deployment.

5. Pull the container images from the remote registry and push them to the undercloud registry:

   (undercloud) $ openstack overcloud container image upload \
     --config-file  /home/stack/local_registry_images.yaml \
     --verbose

   Pulling the required images might take some time depending on the speed of your network and your undercloud disk.

   Note

   The container images consume approximately 10 GB of disk space.

6. The images are now stored on the undercloud’s docker-distribution registry. To view the list of images on the undercloud’s docker-distribution registry, run the following command:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog | jq .repositories[]

   Note

   The _catalog resource by itself displays only 100 images. To display more images, use the ?n=<interger> query string with the _catalog resource to display a larger number of images:

   (undercloud) $  curl http://192.168.24.1:8787/v2/_catalog?n=150 | jq .repositories[]

   To view a list of tags for a specific image, use the skopeo command:

   (undercloud) $ curl -s http://192.168.24.1:8787/v2/rhosp13/openstack-keystone/tags/list | jq .tags

   To verify a tagged image, use the skopeo command:

   (undercloud) $ skopeo inspect --tls-verify=false docker://192.168.24.1:8787/rhosp13/openstack-keystone:13.0-44

Procedure

1. Create a template to pull images to the local registry:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud container image prepare \
     --namespace=rhosp13 \
     --prefix=openstack- \
     --output-images-file /home/stack/satellite_images

   • Use the -e option to include any environment files for optional services.
   • Use the -r option to include a custom roles file.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.
   Note

   This version of the openstack overcloud container image prepare command targets the registry on the registry.redhat.io to generate an image list. It uses different values than the openstack overcloud container image prepare command used in a later step.

2. This creates a file called satellite_images with your container image information. You will use this file to synchronize container images to your Satellite 6 server.

3. Remove the YAML-specific information from the satellite_images file and convert it into a flat file containing only the list of images. The following sed commands accomplish this:

   (undercloud) $ awk -F ':' '{if (NR!=1) {gsub("[[:space:]]", ""); print $2}}' ~/satellite_images > ~/satellite_images_names

   This provides a list of images that you pull into the Satellite server.

4. Copy the satellite_images_names file to a system that contains the Satellite 6 hammer tool. Alternatively, use the instructions in the Hammer CLI Guide to install the hammer tool to the undercloud.

5. Run the following hammer command to create a new product (OSP13 Containers) to your Satellite organization:

   $ hammer product create \
     --organization "ACME" \
     --name "OSP13 Containers"

   This custom product will contain our images.

6. Add the base container image to the product:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhosp13/openstack-base \
     --name base

7. Add the overcloud container images from the satellite_images file.

   $ while read IMAGE; do \
     IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \
     hammer repository create \
     --organization "ACME" \
     --product "OSP13 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name $IMAGE \
     --name $IMAGENAME ; done < satellite_images_names

8. Synchronize the container images:

   $ hammer product synchronize \
     --organization "ACME" \
     --name "OSP13 Containers"

   Wait for the Satellite server to complete synchronization.

   Note

   Depending on your configuration, hammer might ask for your Satellite server username and password. You can configure hammer to automatically login using a configuration file. See the "Authentication" section in the Hammer CLI Guide.

9. If your Satellite 6 server uses content views, create a new content view version to incorporate the images.

10. Check the tags available for the base image:

    $ hammer docker tag list --repository "base" \
      --organization "ACME" \
      --product "OSP13 Containers"

    This displays tags for the OpenStack Platform container images.

11. Return to the undercloud and generate an environment file for the images on your Satellite server. The following is an example command for generating the environment file:

    (undercloud) $ openstack overcloud container image prepare \
      --namespace=satellite6.example.com:5000 \
      --prefix=acme-osp13_containers- \
      --tag-from-label {version}-{release} \
      --output-env-file=/home/stack/templates/overcloud_images.yaml

    Note

    This version of the openstack overcloud container image prepare command targets the Satellite server. It uses different values than the openstack overcloud container image prepare command used in a previous step.

    When running this command, include the following data:

    • --namespace - The URL and port of the registry on the Satellite server. The registry port on Red Hat Satellite is 5000. For example, --namespace=satellite6.example.com:5000.

      Note

      If you are using Red Hat Satellite version 6.10, you do not need to specify a port. The default port of 443 is used. For more information, see "How can we adapt RHOSP13 deployment to Red Hat Satellite 6.10?".

    • --prefix= - The prefix is based on a Satellite 6 convention for labels, which uses lower case characters and substitutes spaces for underscores. The prefix differs depending on whether you use content views:

      • If you use content views, the structure is [org]-[environment]-[content view]-[product]-. For example: acme-production-myosp13-osp13_containers-.
      • If you do not use content views, the structure is [org]-[product]-. For example: acme-osp13_containers-.
    • --tag-from-label {version}-{release} - Identifies the latest tag for each image.
    • -e - Include any environment files for optional services.
    • -r - Include a custom roles file.

    • --set ceph_namespace, --set ceph_image, --set ceph_tag - If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location. Note that ceph_image now includes a Satellite-specific prefix. This prefix is the same value as the --prefix option. For example:

      --set ceph_image=acme-osp13_containers-rhceph-3-rhel7

      This ensures the overcloud uses the Ceph container image using the Satellite naming convention.

12. The overcloud_images.yaml file contains the image locations on the Satellite server. Include this file with your deployment.

Procedure

1. Log into the director as the stack user.

2. Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update:

   $ sudo yum update -y python-tripleoclient

3. The director uses the openstack undercloud upgrade command to update the Undercloud environment. Run the command:

   $ openstack undercloud upgrade

4. Wait until the undercloud upgrade process completes.

5. Reboot the undercloud to update the operating system’s kernel and other system packages:

   $ sudo reboot

6. Wait until the node boots.

Procedure

1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

2. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-13.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-13.0.tar; do tar -xvf $i; done
   $ cd ~

3. On the undercloud node, source the undercloud credentials:

   $ source ~/stackrc

4. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

5. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

6. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   $ source ~/stackrc

3. To increase the number of concurrent processes that container-puppet uses to generate configuration files, you must configure the DockerPuppetProcessCount parameter.

   1. Create an environment file called updates-environment.yaml in your templates directory:

      $ touch ~/templates/updates-environment.yaml

   2. Edit the file and add the following content:

      parameter_defaults:
          DockerPuppetProcessCount: 8

   3. Use the -e option to include this environment file when you run the openstack overcloud update prepare, openstack overcloud ceph-upgrade run, and openstack overcloud update converge commands.

4. On a Controller node, archive your deleted database entries:

   1. From the overcloud, list all instances of the Controller nodes:

      $ source ~/overcloudrc
      $ openstack server list

   2. Log on to a Controller node that is running the nova_api_cron container:

      ssh heat-admin@<controller_ip>

      • Replace <controller name or IP> with the IP address of a Controller node.

   3. Archive deleted database entries:

      $ sudo docker exec -u 42436 -ti nova_api_cron bash
      $ nova-manage db archive_deleted_rows --max_rows 1000
      $ exit

5. To download all packages that are required for an update on all the overcloud nodes, complete the following steps:

   1. Create a static inventory file of your overcloud:

      $ tripleo-ansible-inventory \
      --ansible_ssh_user heat-admin \
      --static-yaml-inventory ~/inventory.yaml

   2. Create the following Ansible playbook:

      $ cat > ~/yum-download-only.yaml <<'EOF'
      - hosts: all
        gather_facts: false
        tasks:
          - name: Pre-download all packages on all overcloud nodes
            shell:
              yum upgrade -y --downloadonly
            become: true
      EOF

   3. Run the yum-download-only.yaml Ansible playbook:

      $ ansible-playbook \
      -i ~/inventory.yaml \
      -f 20 ~/yum-download-only.yaml \
      --limit Controller,Compute,CephStorage

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the update preparation command:

   $ openstack overcloud update prepare \
       --templates \
       -r <ROLES DATA FILE> \
       -n <NETWORK DATA FILE> \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /home/stack/templates/updates-environment.yaml \
       -e <ENVIRONMENT FILE> \
       -e <ENVIRONMENT FILE> \
       --stack <STACK_NAME>
       ...

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e).
   • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
   • If you use your own custom roles, include your custom roles (roles_data) file (-r).
   • If you use custom networks, include your composable network (network_data) file (-n).
   • If you deploy a high availability cluster, include the --ntp-server option in the update preparation command, or include the NtpServer parameter and value in your environment file.
   • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your stack.
3. Wait until the update preparation completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the Ceph Storage update command. For example:

   $ openstack overcloud ceph-upgrade run \
       --templates \
       -e <ENVIRONMENT FILE> \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /home/stack/templates/updates-environment.yaml

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e)
   • The environment file with your container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning, as this option is deprecated in favor of using -e for the container image environment file.
   • If applicable, your custom roles (roles_data) file (--roles-file)
   • If applicable, your composable network (network_data) file (--networks-file)
3. Wait until the Ceph Storage node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the update command:

   $ openstack overcloud update run --nodes Controller

3. Wait until the Controller node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the update command:

   $ openstack overcloud update run --nodes Compute

3. Wait until the Compute node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the update command:

   $ openstack overcloud update run --nodes ComputeHCI

3. Wait until the node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Update group nodes.

   To update all nodes in a group:

   $ openstack overcloud update run --nodes <GROUP_NAME>

   To update a single node in a group:

   $ openstack overcloud update run --nodes <GROUP_NAME> [NODE_INDEX]

   Note

   Ensure that you update all nodes if you choose to update nodes individually.

   The index of the first node in a group is zero (0). For example, to update the first node in a group named CephStorage, the command is:

   openstack overcloud update run --nodes CephStorage[0]

3. Wait until the node update completes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the update finalization command:

   $ openstack overcloud update converge \
       --templates \
       --stack <STACK_NAME> \
       -e /home/stack/templates/overcloud_images.yaml \
       -e /home/stack/templates/updates-environment.yaml \
       -e <ENVIRONMENT FILE>
       ...

   Include the following options relevant to your environment:

   • Custom configuration environment files (-e)
   • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
   • If applicable, your custom roles (roles_data) file (--roles-file)
   • If applicable, your composable network (network_data) file (--networks-file)
   • If the name of your overcloud stack is different than the default name overcloud, you must include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your overcloud stack.
3. Wait until the update finalization completes.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. Repeat these steps for all Controller and composable nodes.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

2. Select the first Ceph Storage node to reboot and log into it.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to a Ceph MON or Controller node and check the cluster status:

   $ sudo ceph -s

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the Ceph MON or Controller node, reboot the next Ceph Storage node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo ceph osd unset noout
   $ sudo ceph osd unset norebalance

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo ceph status

Procedure

1. Log in to the undercloud as the stack user.

2. To identify the Compute node that you intend to reboot, list all Compute nodes:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

3. From the overcloud, select a Compute Node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host <hostname> --all-projects

5. Migrate your instances. For more information on migration strategies, see Migrating virtual machines between Compute nodes.

6. Log into the Compute Node and reboot it:

   [heat-admin@overcloud-compute-0 ~]$ sudo reboot

7. Wait until the node boots.

8. Enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname> nova-compute --enable

9. Verify that the Compute node is enabled:

   (overcloud) $ openstack compute service list

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo ceph osd set noout
   $ sudo ceph osd set norebalance

2. Log in to the undercloud as the stack user.

3. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node you aim to reboot.

4. From the undercloud, select a Compute node and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

5. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

6. Use one of the following commands to migrate your instances:

   1. Migrate the instance to a specific host of your choice:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until the migration completes.

8. Confirm that the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Continue migrating instances until none remain on the chosen Compute node.

10. Log in to a Ceph MON or a Controller node and check the cluster status:

    $ sudo ceph -s

    Check that the pgmap reports all pgs as normal (active+clean).

11. Reboot the Compute HCI node:

    $ sudo reboot

12. Wait until the node boots.

13. Enable the Compute node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

14. Verify that the Compute node is enabled:

    (overcloud) $ openstack compute service list

15. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

16. When complete, log in to a Ceph MON or Controller node and enable cluster rebalancing again:

    $ sudo ceph osd unset noout
    $ sudo ceph osd unset norebalance

17. Perform a final status check to verify the cluster reports HEALTH_OK:

    $ sudo ceph status