The undercloud is the main director node. It is a single-system OpenStack installation that includes components for provisioning and managing the OpenStack nodes that form your OpenStack environment (the overcloud). The components that form the undercloud provide the multiple functions:

The overcloud is the resulting Red Hat OpenStack Platform environment created using the undercloud. This includes different nodes roles which you define based on the OpenStack Platform environment you aim to create. The undercloud includes a default set of overcloud node roles, which include:

The Red Hat OpenStack Platform director uses a Controller node cluster to provide high availability services to your OpenStack Platform environment. The director installs a duplicate set of components on each Controller node and manages them together as a single service. This type of cluster configuration provides a fallback in the event of operational failures on a single Controller node; this provides OpenStack users with a certain degree of continuous operation.

The OpenStack Platform director uses some key pieces of software to manage components on the Controller node:

It is common for large organizations using OpenStack to serve thousands of clients or more. Each OpenStack client is likely to have their own unique needs when consuming block storage resources. Deploying glance (images), cinder (volumes) and/or nova (Compute) on a single node can become impossible to manage in large deployments with thousands of clients. Scaling OpenStack externally resolves this challenge.

However, there is also a practical requirement to virtualize the storage layer with a solution like Red Hat Ceph Storage so that you can scale the Red Hat OpenStack Platform storage layer from tens of terabytes to petabytes (or even exabytes) of storage. Red Hat Ceph Storage provides this storage virtualization layer with high availability and high performance while running on commodity hardware. While virtualization might seem like it comes with a performance penalty, Ceph stripes block device images as objects across the cluster; this means large Ceph Block Device images have better performance than a standalone disk. Ceph Block devices also support caching, copy-on-write cloning, and copy-on-read cloning for enhanced performance.

See Red Hat Ceph Storage for additional information about Red Hat Ceph Storage.

Note the following:

The undercloud system hosting the director provides provisioning and management for all nodes in the overcloud.

$ yum install libvirt-client libvirt-daemon qemu-kvm libvirt-daemon-driver-qemu libvirt-daemon-kvm virt-install bridge-utils rsync

$ virt-install --name undercloud --memory=16384 --vcpus=4 --location /var/lib/libvirt/images/rhel-server-7.4-x86_64-dvd.iso --disk size=100 --network bridge=br-ex --network bridge=br-ctlplane --graphics=vnc --hvm --os-variant=rhel7

--initrd-inject=/root/ks.cfg --extra-args "ks=file:/ks.cfg"

$ virsh snapshot-create-as --domain undercloud --disk-only --atomic --quiesce

$ rsync --sparse -avh --progress /var/lib/libvirt/images/undercloud.qcow2 1.qcow2

$ virsh blockcommit undercloud vda --active --verbose --pivot

The undercloud host requires at least two networks:

This represents the minimum number of networks required. However, the director can isolate other Red Hat OpenStack Platform network traffic into other networks. Red Hat OpenStack Platform supports both physical interfaces and tagged VLANs for network isolation.

Note the following:

The following sections detail the requirements for individual systems and nodes in the overcloud installation.

Both the undercloud and overcloud require access to Red Hat repositories either through the Red Hat Content Delivery Network, or through Red Hat Satellite 5 or 6. If using a Red Hat Satellite Server, synchronize the required repositories to your OpenStack Platform environment. Use the following list of CDN channel names as a guide:

The director provides multiple default node types for building your overcloud. These node types are:

The following table provides some example of different overclouds and defines the node types for each scenario.

In addition, consider whether to split individual services into custom roles. For more information on the composable roles architecture, see "Composable Services and Custom Roles" in the Advanced Overcloud Customization guide.

It is important to plan your environment’s networking topology and subnets so that you can properly map roles and services to correctly communicate with each other. Red Hat OpenStack Platform uses the neutron networking service, which operates autonomously and manages software-based networks, static and floating IP addresses, and DHCP. The director deploys this service on each Controller node in an overcloud environment.

Red Hat OpenStack Platform maps the different services onto separate network traffic types, which are assigned to the various subnets in your environments. These network traffic types include:

In a typical Red Hat OpenStack Platform installation, the number of network types often exceeds the number of physical network links. In order to connect all the networks to the proper hosts, the overcloud uses VLAN tagging to deliver more than one network per interface. Most of the networks are isolated subnets but some require a Layer 3 gateway to provide routing for Internet access or infrastructure network connectivity.

The director provides a method for mapping six of these traffic types to certain subnets or VLANs. These traffic types include:

Any unassigned networks are automatically assigned to the same subnet as the Provisioning network.

The diagram below provides an example of a network topology where the networks are isolated on separate VLANs. Each overcloud node uses two interfaces (nic2 and nic3) in a bond to deliver these networks over their respective VLANs. Meanwhile, each overcloud node communicates with the undercloud over the Provisioning network through a native VLAN using nic1.

The following table provides examples of network traffic mappings different network layouts:

The director provides different storage options for the overcloud environment. This includes:

The director installation process requires a non-root user to execute commands. Use the following commands to create the user named stack and set a password:

[root@director ~]# useradd stack
[root@director ~]# passwd stack  # specify a password

Disable password requirements for this user when using sudo:

[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
[root@director ~]# chmod 0440 /etc/sudoers.d/stack

Switch to the new stack user:

[root@director ~]# su - stack
[stack@director ~]$

Continue the director installation as the stack user.

The director uses system images and Heat templates to create the overcloud environment. To keep these files organized, we recommend creating directories for images and templates:

[stack@director ~]$ mkdir ~/images
[stack@director ~]$ mkdir ~/templates

Other sections in this guide use these two directories to store certain files.

The director requires a fully qualified domain name for its installation and configuration process. This means you may need to set the hostname of your director’s host. Check the hostname of your host:

[stack@director ~]$ hostname    # Checks the base hostname
[stack@director ~]$ hostname -f # Checks the long hostname (FQDN)

If either commands do not report the correct hostname or report an error, use hostnamectl to set a hostname:

[stack@director ~]$ sudo hostnamectl set-hostname manager.example.com
[stack@director ~]$ sudo hostnamectl set-hostname --transient manager.example.com

The director also requires an entry for the system’s hostname and base name in /etc/hosts. For example, if the system is named manager.example.com, then /etc/hosts requires an entry like:

127.0.0.1   manager.example.com manager localhost localhost.localdomain localhost4 localhost4.localdomain4

To install the Red Hat OpenStack Platform director, first register the host system using Red Hat Subscription Manager, and subscribe to the required channels.

These repositories contain packages the director installation requires.

Perform an update on your system to make sure you have the latest base system packages:

[stack@director ~]$ sudo yum update -y
[stack@director ~]$ sudo reboot

The system is now ready for the director installation.

Use the following command to install the required command line tools for director installation and configuration:

[stack@director ~]$ sudo yum install -y python-tripleoclient

This installs all packages required for the director installation.

If you aim to create an overcloud with Ceph Storage nodes, install the additional ceph-ansible package:

[stack@director ~]$ sudo yum install -y ceph-ansible

The director installation process requires certain settings to determine your network configurations. The settings are stored in a template located in the stack user’s home directory as undercloud.conf.

Red Hat provides a basic template to help determine the required settings for your installation. Copy this template to the stack user’s home directory:

[stack@director ~]$ cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf

The undercloud.conf file contains settings to configure your undercloud. If you omit or comment out a parameter, the undercloud installation uses the default value.

The template contains two sections: [DEFAULT] and [auth]. The [DEFAULT] section contains the following parameters:

The [auth] section contains the following parameters:

Modify the values for these parameters to suit your network. When complete, save the file and run the following command:

[stack@director ~]$ openstack undercloud install

This launches the director’s configuration script. The director installs additional packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

The configuration script generates two files when complete:

The configuration also starts all OpenStack Platform services automatically. Check the enabled services using the following command:

[stack@director ~]$ sudo systemctl list-units openstack-*

The installation also adds the stack user to the docker group so the stack user has access to container management commands. Refresh the stack user’s permissions with the following command:

[stack@director ~]$ exec su -l stack

The command prompts you to log in again. Enter the stack user’s password.

To initialize the stack user to use the command line tools, run the following command:

[stack@director ~]$ source ~/stackrc

The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

(undercloud) [stack@director ~]$

You can now use the director’s command line tools.

The director requires several disk images for provisioning overcloud nodes. This includes:

Obtain these images from the rhosp-director-images and rhosp-director-images-ipa packages:

(undercloud) [stack@director ~]$ sudo yum install rhosp-director-images rhosp-director-images-ipa

Extract the archives to the images directory on the stack user’s home (/home/stack/images):

(undercloud) [stack@director ~]$ cd ~/images
(undercloud) [stack@director images]$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-12.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-12.0.tar; do tar -xvf $i; done

Import these images into the director:

(undercloud) [stack@director images]$ openstack overcloud image upload --image-path /home/stack/images/

This uploads the following images into the director: bm-deploy-kernel, bm-deploy-ramdisk, overcloud-full, overcloud-full-initrd, overcloud-full-vmlinuz. These are the images for deployment and the overcloud. The script also installs the introspection images on the director’s PXE server.

View a list of the images in the CLI:

(undercloud) [stack@director images]$ openstack image list
+--------------------------------------+------------------------+
| ID                                   | Name                   |
+--------------------------------------+------------------------+
| 765a46af-4417-4592-91e5-a300ead3faf6 | bm-deploy-ramdisk      |
| 09b40e3d-0382-4925-a356-3a4b4f36b514 | bm-deploy-kernel       |
| ef793cd0-e65c-456a-a675-63cd57610bd5 | overcloud-full         |
| 9a51a6cb-4670-40de-b64b-b70f4dd44152 | overcloud-full-initrd  |
| 4f7e33f4-d617-47c1-b36f-cbe90f132e5d | overcloud-full-vmlinuz |
+--------------------------------------+------------------------+

This list will not show the introspection PXE images. The director copies these files to /httpboot.

(undercloud) [stack@director images]$ ls -l /httpboot
total 341460
-rwxr-xr-x. 1 root              root                5153184 Mar 31 06:58 agent.kernel
-rw-r--r--. 1 root              root              344491465 Mar 31 06:59 agent.ramdisk
-rw-r--r--. 1 ironic-inspector  ironic-inspector        337 Mar 31 06:23 inspector.ipxe

If you intend for the overcloud to resolve external hostnames, such as cdn.redhat.com, it is recommended to set a nameserver on the overcloud nodes. For a standard overcloud without network isolation, the nameserver is defined using the undercloud’s neutron subnet. Use the following commands to define nameservers for the environment:

(undercloud) [stack@director images]$ openstack subnet list
(undercloud) [stack@director images]$ openstack subnet set --dns-nameserver [nameserver1-ip] --dns-nameserver [nameserver2-ip] [subnet-uuid]

View the subnet to verify the nameserver:

(undercloud) [stack@director images]$ openstack subnet show [subnet-uuid]
+-------------------+-----------------------------------------------+
| Field             | Value                                         |
+-------------------+-----------------------------------------------+
| ...               |                                               |
| dns_nameservers   | 8.8.8.8                                       |
| ...               |                                               |
+-------------------+-----------------------------------------------+

Red Hat provides a process to back up important data from the undercloud host and the Red Hat OpenStack Platform director. For more information about undercloud backups, see the "Back Up and Restore the Director Undercloud" guide.

This completes the undercloud configuration. The next chapter explores basic overcloud configuration, including registering nodes, inspecting them, and then tagging them into various node roles.

This section provides an overview on how to use the openstack overcloud container image prepare command, including conceptual information on the command’s various options. You can find proper use cases and end-to-end procedures for using this command in Selecting a Registry Method.

The following snippet is an example of this file’s contents:

parameter_defaults:
  DockerAodhApiImage: registry.access.redhat.com/rhosp12/openstack-aodh-api:latest
  DockerAodhConfigImage: registry.access.redhat.com/rhosp12/openstack-aodh-api:latest
...

The openstack overcloud container image prepare command uses the following options for this function:

The following is an example of this file’s contents:

container_images:
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-api:latest
- imagename: registry.access.redhat.com/rhosp12/openstack-aodh-evaluator:latest
...

As a result, the director generates the image names using the following format:

You can also discover the latest versioned tag for an image using the openstack overcloud container image tag discover command. For example:

$ sudo openstack overcloud container image tag discover \
  --image registry.access.redhat.com/rhosp12/openstack-base:latest \
  --tag-from-label version-release

This checks the tags available for the openstack-base image using the image’s version-release label as a basis and outputs the latest versioned tag.

The director only prepares container images for core OpenStack Platform Services ^[1]. Some additional features use services that require additional container images. You enable these services with environment files. The openstack overcloud container image prepare command uses the following option to include environment files and their respective container images:

The following table provides a sample list of additional services that use container images and their respective environment file locations within the /usr/share/openstack-tripleo-heat-templates directory.

The next few sections provide examples of including additional services.

In addition to this environment file, you also need to define the Ceph Storage container location, which is different from the OpenStack Platform services. Use the --set option to set the following parameters specific to Ceph Storage:

The following snippet is an example that includes Ceph Storage in your container image files:

$ openstack overcloud container image prepare \
  ...
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  --set ceph_namespace=registry.access.redhat.com/rhceph \
  --set ceph_image=rhceph-2-rhel7 \
  --set ceph_tag=<CEPH_TAG> \
  ...

$ openstack overcloud container image prepare \
  ...
  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml \
  ...

$ openstack overcloud container image prepare \
  ...
  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sahara.yaml \
  ...

Red Hat hosts the overcloud container images on registry.access.redhat.com. Pulling the images from a remote registry is the simplest method because the registry is already setup and all you require is the URL and namespace of the image you aim to pull. However, during overcloud creation, the overcloud nodes all pull images from the remote repository, which can congest your external connection. If that is a problem, you can either:

To pull the images directly from registry.access.redhat.com in your overcloud deployment, an environment file is required to specify the image parameters. The following commands automatically create this environment file:

The registry configuration is ready. Continue with the instructions in Chapter 6, Configuring a Basic Overcloud with the CLI Tools.

You can configure a local registry on the undercloud to store overcloud container images. This method involves the following:

This keeps network traffic for container images within your internal network, which does not congest your external network connection and can speed the deployment process.

To pull the images from registry.access.redhat.com to our local registry, use the following process:

The registry configuration is ready. Continue with the instructions in Chapter 6, Configuring a Basic Overcloud with the CLI Tools.

Red Hat Satellite 6 offers registry synchronization capabilities. This provides a method to pull multiple images into a Satellite server and manage them as part of an application life cycle. The Satellite also acts as a registry for other container-enabled systems to use. For more details information on managing container images, see "Managing Container Images" in the Red Hat Satellite 6 Content Management Guide.

The examples in this procedure use the hammer command line tool for Red Hat Satellite 6 and an example organization called ACME. Substitute this organization for your own Satellite 6 organization.

To pull the images from registry.access.redhat.com to our local registry, use the following process:

The registry configuration is ready. Continue with the instructions in Chapter 6, Configuring a Basic Overcloud with the CLI Tools.

The director requires a node definition template, which you create manually. This file (instackenv.json) uses the JSON format file, and contains the hardware and power management details for your nodes. For example, a template for registering two nodes might look like this:

{
    "nodes":[
        {
            "mac":[
                "bb:bb:bb:bb:bb:bb"
            ],
            "name":"node01",
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.168.24.205"
        },
        {
            "mac":[
                "cc:cc:cc:cc:cc:cc"
            ],
            "name":"node02",
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.168.24.206"
        }
    ]
}

This template uses the following attributes:

After creating the template, save the file to the stack user’s home directory (/home/stack/instackenv.json), then import it into the director using the following commands:

$ source ~/stackrc
(undercloud) $ openstack overcloud node import ~/instackenv.json

This imports the template and registers each node from the template into the director.

After the node registration and configuration completes, view a list of these nodes in the CLI:

(undercloud) $ openstack baremetal node list

The director can run an introspection process on each node. This process causes each node to boot an introspection agent over PXE. This agent collects hardware data from the node and sends it back to the director. The director then stores this introspection data in the OpenStack Object Storage (swift) service running on the director. The director uses hardware information for various purposes such as profile tagging, benchmarking, and manual root disk assignment.

Run the following command to inspect the hardware attributes of each node:

(undercloud) $ openstack overcloud node introspect --all-manageable --provide

Monitor the progress of the introspection using the following command in a separate terminal window:

(undercloud) $ sudo journalctl -l -u openstack-ironic-inspector -u openstack-ironic-inspector-dnsmasq -u openstack-ironic-conductor -f

After the introspection completes, all nodes change to an available state.

(undercloud) $ openstack baremetal node manage [NODE UUID]
(undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

After the introspection completes, the nodes changes to an available state.

(undercloud) $ for node in $(openstack baremetal node list --fields uuid -f value) ; do openstack baremetal node manage $node ; done
(undercloud) $ openstack overcloud node introspect --all-manageable --provide

After the introspection completes, all nodes change to an available state.

To get a list of interfaces on a node:

(undercloud) $ openstack baremetal introspection interface list [NODE UUID]

For example:

(undercloud) $ openstack baremetal introspection interface list c89397b7-a326-41a0-907d-79f8b86c7cd9
+-----------+-------------------+------------------------+-------------------+----------------+
| Interface | MAC Address       | Switch Port VLAN IDs   | Switch Chassis ID | Switch Port ID |
+-----------+-------------------+------------------------+-------------------+----------------+
| p2p2      | 00:0a:f7:79:93:19 | [103, 102, 18, 20, 42] | 64:64:9b:31:12:00 | 510            |
| p2p1      | 00:0a:f7:79:93:18 | [101]                  | 64:64:9b:31:12:00 | 507            |
| em1       | c8:1f:66:c7:e8:2f | [162]                  | 08:81:f4:a6:b3:80 | 515            |
| em2       | c8:1f:66:c7:e8:30 | [182, 183]             | 08:81:f4:a6:b3:80 | 559            |
+-----------+-------------------+------------------------+-------------------+----------------+

To see interface data and switch port information:

(undercloud) $ openstack baremetal introspection interface show [NODE UUID] [INTERFACE]

For example:

(undercloud) $ openstack baremetal introspection interface show c89397b7-a326-41a0-907d-79f8b86c7cd9 p2p1
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| Field                                | Value                                                                                                                  |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+
| interface                            | p2p1                                                                                                                   |
| mac                                  | 00:0a:f7:79:93:18                                                                                                      |
| node_ident                           | c89397b7-a326-41a0-907d-79f8b86c7cd9                                                                                   |
| switch_capabilities_enabled          | [u'Bridge', u'Router']                                                                                                 |
| switch_capabilities_support          | [u'Bridge', u'Router']                                                                                                 |
| switch_chassis_id                    | 64:64:9b:31:12:00                                                                                                      |
| switch_port_autonegotiation_enabled  | True                                                                                                                   |
| switch_port_autonegotiation_support  | True                                                                                                                   |
| switch_port_description              | ge-0/0/2.0                                                                                                             |
| switch_port_id                       | 507                                                                                                                    |
| switch_port_link_aggregation_enabled | False                                                                                                                  |
| switch_port_link_aggregation_id      | 0                                                                                                                      |
| switch_port_link_aggregation_support | True                                                                                                                   |
| switch_port_management_vlan_id       | None                                                                                                                   |
| switch_port_mau_type                 | Unknown                                                                                                                |
| switch_port_mtu                      | 1514                                                                                                                   |
| switch_port_physical_capabilities    | [u'1000BASE-T fdx', u'100BASE-TX fdx', u'100BASE-TX hdx', u'10BASE-T fdx', u'10BASE-T hdx', u'Asym and Sym PAUSE fdx'] |
| switch_port_protocol_vlan_enabled    | None                                                                                                                   |
| switch_port_protocol_vlan_ids        | None                                                                                                                   |
| switch_port_protocol_vlan_support    | None                                                                                                                   |
| switch_port_untagged_vlan_id         | 101                                                                                                                    |
| switch_port_vlan_ids                 | [101]                                                                                                                  |
| switch_port_vlans                    | [{u'name': u'RHOS13-PXE', u'id': 101}]                                                                                 |
| switch_protocol_identities           | None                                                                                                                   |
| switch_system_name                   | rhos-compute-node-sw1                                                                                                  |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------+

For example, the numa_topology collector is part of these hardware inspection extras and includes the following information for each NUMA node:

Use the openstack baremetal introspection data save _UUID_ | jq .numa_topology command to retrieve this information, with the UUID of the bare-metal node.

The following example shows the retrieved NUMA information for a bare-metal node:

{
  "cpus": [
    {
      "cpu": 1,
      "thread_siblings": [
        1,
        17
      ],
      "numa_node": 0
    },
    {
      "cpu": 2,
      "thread_siblings": [
        10,
        26
      ],
      "numa_node": 1
    },
    {
      "cpu": 0,
      "thread_siblings": [
        0,
        16
      ],
      "numa_node": 0
    },
    {
      "cpu": 5,
      "thread_siblings": [
        13,
        29
      ],
      "numa_node": 1
    },
    {
      "cpu": 7,
      "thread_siblings": [
        15,
        31
      ],
      "numa_node": 1
    },
    {
      "cpu": 7,
      "thread_siblings": [
        7,
        23
      ],
      "numa_node": 0
    },
    {
      "cpu": 1,
      "thread_siblings": [
        9,
        25
      ],
      "numa_node": 1
    },
    {
      "cpu": 6,
      "thread_siblings": [
        6,
        22
      ],
      "numa_node": 0
    },
    {
      "cpu": 3,
      "thread_siblings": [
        11,
        27
      ],
      "numa_node": 1
    },
    {
      "cpu": 5,
      "thread_siblings": [
        5,
        21
      ],
      "numa_node": 0
    },
    {
      "cpu": 4,
      "thread_siblings": [
        12,
        28
      ],
      "numa_node": 1
    },
    {
      "cpu": 4,
      "thread_siblings": [
        4,
        20
      ],
      "numa_node": 0
    },
    {
      "cpu": 0,
      "thread_siblings": [
        8,
        24
      ],
      "numa_node": 1
    },
    {
      "cpu": 6,
      "thread_siblings": [
        14,
        30
      ],
      "numa_node": 1
    },
    {
      "cpu": 3,
      "thread_siblings": [
        3,
        19
      ],
      "numa_node": 0
    },
    {
      "cpu": 2,
      "thread_siblings": [
        2,
        18
      ],
      "numa_node": 0
    }
  ],
  "ram": [
    {
      "size_kb": 66980172,
      "numa_node": 0
    },
    {
      "size_kb": 67108864,
      "numa_node": 1
    }
  ],
  "nics": [
    {
      "name": "ens3f1",
      "numa_node": 1
    },
    {
      "name": "ens3f0",
      "numa_node": 1
    },
    {
      "name": "ens2f0",
      "numa_node": 0
    },
    {
      "name": "ens2f1",
      "numa_node": 0
    },
    {
      "name": "ens1f1",
      "numa_node": 0
    },
    {
      "name": "ens1f0",
      "numa_node": 0
    },
    {
      "name": "eno4",
      "numa_node": 0
    },
    {
      "name": "eno1",
      "numa_node": 0
    },
    {
      "name": "eno3",
      "numa_node": 0
    },
    {
      "name": "eno2",
      "numa_node": 0
    }
  ]
}

You can use auto-discovery to register undercloud nodes and generate their metadata, without first having to create an instackenv.json file. This improvement can help reduce the time spent initially collecting the node’s information, for example, removing the need to collate the IPMI IP addresses and subsequently create the instackenv.json.

After registering and inspecting the hardware of each node, you will tag them into specific profiles. These profile tags match your nodes to flavors, and in turn the flavors are assigned to a deployment role. The following example shows the relationship across roles, flavors, profiles, and nodes for Controller nodes:

Default profile flavors compute, control, swift-storage, ceph-storage, and block-storage are created during undercloud installation and are usable without modification in most environments.

To tag a node into a specific profile, add a profile option to the properties/capabilities parameter for each node. For example, to tag your nodes to use Controller and Compute profiles respectively, use the following commands:

(undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13
(undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

The addition of the profile:compute and profile:control options tag the two nodes into each respective profiles.

These commands also set the boot_option:local parameter, which defines how each node boots. Depending on your hardware, you might also need to add the boot_mode parameter to uefi so that nodes boot using UEFI instead of the default BIOS mode. For more information, see Section D.2, “UEFI Boot Mode”.

After completing node tagging, check the assigned profiles or possible profiles:

(undercloud) $ openstack overcloud profiles list

(undercloud) $ openstack flavor create --id auto --ram 4096 --disk 40 --vcpus 1 networker
(undercloud) $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="networker" networker

Assign nodes with this new profile:

(undercloud) $ openstack baremetal node set --property capabilities='profile:networker,boot_option:local' dad05b82-0c74-40bf-9d12-193184bfc72d

Some nodes might use multiple disks. This means the director needs to identify the disk to use for the root disk during provisioning.

There are several properties you can use to help the director identify the root disk:

In this example, you specify the drive to deploy the overcloud image using the serial number of the disk to determine the root device.

Check the disk information from each node’s hardware introspection. The following command displays the disk information from a node:

(undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

For example, the data for one node might show three disks:

[
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sda",
    "wwn_vendor_extension": "0x1ea4dcc412a9632b",
    "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f380700",
    "serial": "61866da04f3807001ea4dcc412a9632b"
  }
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sdb",
    "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
    "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f380d00",
    "serial": "61866da04f380d001ea4e13c12e36ad6"
  }
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sdc",
    "wwn_vendor_extension": "0x1ea4e31e121cfb45",
    "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f37fc00",
    "serial": "61866da04f37fc001ea4e31e121cfb45"
  }
]

For this example, set the root device to disk 2, which has 61866da04f380d001ea4e13c12e36ad6 as the serial number. This requires a change to the root_device parameter for the node definition:

(undercloud) $ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

This helps the director identify the specific disk to use as the root disk. When we initiate our overcloud creation, the director provisions this node and writes the overcloud image to this disk.

You will need to follow the following procedure if your undercloud uses TLS, and the CA is not publicly trusted. The undercloud operates its own Certificate Authority (CA) for SSL endpoint encryption; to make the undercloud endpoints accessible to the rest of your deployment, you will need to configure your overcloud nodes to trust the undercloud CA. This procedure is required for new Red Hat OpenStack Platform 12 deployments, and for upgrades from 11 to 12.

The CA certificate is copied to each overcloud node during the overcloud deployment, causing it to trust the encryption presented by the undercloud’s SSL endpoints. For more information on including environment files, see Section 6.9, “Including Environment Files in Overcloud Creation”.

The undercloud includes a set of Heat templates that acts as a plan for your overcloud creation. You can customize aspects of the overcloud using environment files, which are YAML-formatted files that override parameters and resources in the core Heat template collection. You can include as many environment files as necessary. However, the order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

It is recommended to keep your custom environment files organized in a separate directory, such as the templates directory.

You can customize advanced features for your overcloud using the Advanced Overcloud Customization guide.

The final stage in creating your OpenStack environment is to run the openstack overcloud deploy command to create it. Before running this command, you should familiarize yourself with key options and how to include custom environment files.

Some command line parameters are outdated or deprecated in favor of using Heat template parameters, which you include in the parameter_defaults section on an environment file. The following table maps deprecated parameters to their Heat Template equivalents.

These parameters are scheduled for removal in a future version of Red Hat OpenStack Platform.

(undercloud) $ openstack help overcloud deploy

The -e includes an environment file to customize your overcloud. You can include as many environment files as necessary. However, the order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

Any environment files added to the overcloud using the -e option become part of your overcloud’s stack definition. The following command is an example of how to start the overcloud creation with custom environment files included:

(undercloud) $ openstack overcloud deploy --templates \
  -e /home/stack/templates/node-info.yaml\
  -e /home/stack/templates/overcloud_images.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  -e /home/stack/templates/ceph-custom-config.yaml \
  -e /home/stack/inject-trust-anchor-hiera.yaml \
  --ntp-server pool.ntp.org \

This command contains the following additional options:

The director requires these environment files for re-deployment and post-deployment functions in Chapter 9, Performing Tasks after Overcloud Creation. Failure to include these files can result in damage to your overcloud.

If you aim to later modify the overcloud configuration, you should:

Do not edit the overcloud configuration directly as such manual configuration gets overridden by the director’s configuration when updating the overcloud stack with the director.

(undercloud) $ ls -1 ~/templates
00-node-info.yaml
10-network-isolation.yaml
20-network-environment.yaml
30-storage-environment.yaml
40-rhel-registration.yaml

Run the following deployment command to include the directory:

(undercloud) $ openstack overcloud deploy --templates --environment-directory ~/templates

For example, an answers file might contain the following:

templates: /usr/share/openstack-tripleo-heat-templates/
environments:
  - ~/templates/00-node-info.yaml
  - ~/templates/10-network-isolation.yaml
  - ~/templates/20-network-environment.yaml
  - ~/templates/30-storage-environment.yaml
  - ~/templates/40-rhel-registration.yaml

Run the following deployment command to include the answers file:

(undercloud) $ openstack overcloud deploy --answers-file ~/answers.yaml

As an alternative to using the openstack overcloud deploy command, the director can also manage imported plans.

To create a new plan, run the following command as the stack user:

(undercloud) $ openstack overcloud plan create --templates /usr/share/openstack-tripleo-heat-templates my-overcloud

This creates a plan from the core Heat template collection in /usr/share/openstack-tripleo-heat-templates. The director names the plan based on your input. In this example, it is my-overcloud. The director uses this name as a label for the object storage container, the workflow environment, and overcloud stack names.

Add parameters from environment files using the following command:

(undercloud) $ openstack overcloud parameters set my-overcloud ~/templates/my-environment.yaml

Deploy your plans using the following command:

(undercloud) $ openstack overcloud plan deploy my-overcloud

Delete existing plans using the following command:

(undercloud) $ openstack overcloud plan delete my-overcloud

Before executing an overcloud creation or stack update, validate your Heat templates and environment files for any errors.

(undercloud) $ openstack overcloud plan create --templates /usr/share/openstack-tripleo-heat-templates overcloud-validation
(undercloud) $ mkdir ~/overcloud-validation
(undercloud) $ cd ~/overcloud-validation
(undercloud) $ openstack container save overcloud-validation

Use the rendered template in ~/overcloud-validation for the validation tests that follow.

(undercloud) $ openstack orchestration template validate --show-nested --template ~/overcloud-validation/overcloud.yaml -e ~/overcloud-validation/overcloud-resource-registry-puppet.yaml -e [ENVIRONMENT FILE] -e [ENVIRONMENT FILE]

This command identifies any syntax errors in the template. If the template syntax validates successfully, the output shows a preview of the resulting overcloud template.

The overcloud creation process begins and the director provisions your nodes. This process takes some time to complete. To view the status of the overcloud creation, open a separate terminal as the stack user and run:

(undercloud) $ source ~/stackrc
(undercloud) $ openstack stack list --nested

The openstack stack list --nested command shows the current stage of the overcloud creation.

The director generates a script to configure and help authenticate interactions with your overcloud from the director host. The director saves this file, overcloudrc, in your stack user’s home director. Run the following command to use this file:

(undercloud) $ source ~/overcloudrc

This loads the necessary environment variables to interact with your overcloud from the director host’s CLI. The command prompt changes to indicate this:

(overcloud) $

To return to interacting with the director’s host, run the following command:

(overcloud) $ source ~/stackrc
(undercloud) $

Each node in the overcloud also contains a user called heat-admin. The stack user has SSH access to this user on each node. To access a node over SSH, find the IP address of the desired node:

(undercloud) $ openstack server list

Then connect to the node using the heat-admin user and the node’s IP address:

(undercloud) $ ssh heat-admin@192.168.24.23

This concludes the creation of the overcloud using the command line tools. For post-creation functions, see Chapter 9, Performing Tasks after Overcloud Creation.

Users access the director’s web UI through SSL. For example, if the IP address of your undercloud is 192.168.24.1, then the address to access the UI is https://192.168.24.1. The web UI initially presents a login screen with fields for the following:

When logging in to the UI, the UI accesses the OpenStack Identity Public API and obtains the endpoints for the other Public API services. These services include

The UI interacts directly with these Public APIs, which is why your client system requires access to their endpoints. The director exposes these endpoints through SSL/TLS encrypted paths on the Public VIP (undercloud_public_host in your undercloud.conf file). Each path corresponds to the service. For example, https://192.168.24.2:443/keystone maps to the OpenStack Identity Public API.

If you aim to change the endpoints or use a different IP for endpoint access, the director UI reads settings from the /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js file. This file uses the following parameters:

The following is an example tripleo_ui_config.js file where 192.168.24.2 is the Public VIP for the undercloud:

window.tripleOUiConfig = {
  'keystone': 'https://192.168.24.2:443/keystone/v2.0',
  'heat': 'https://192.168.24.2:443/heat/v1/%(tenant_id)s',
  'ironic': 'https://192.168.24.2:443/ironic',
  'mistral': 'https://192.168.24.2:443/mistral/v2',
  'swift': 'https://192.168.24.2:443/swift/v1/AUTH_%(tenant_id)s',
  'zaqar-websocket': 'wss://192.168.24.2:443/zaqar',
  "zaqar_default_queue": "tripleo",
  'excludedLanguages': [],
  'loggers': ["console","zaqar"]
};

The UI provides three main sections:

These validation tasks run automatically at certain points in the deployment. However, you can also run them manually. Click the Play button for a validation task you want to run. Click the title of each validation task to run it, or click a validation title to view more information about it.

The director UI requires a plan before configuring the overcloud. This plan is usually a Heat template collection, like the one on your undercloud at /usr/share/openstack-tripleo-heat-templates. In addition, you can customize the plan to suit your hardware and environment requirements. For more information about customizing the overcloud, see the Advanced Overcloud Customization guide.

The plan displays four main steps to configuring your overcloud:

The undercloud installation and configuration automatically uploads a plan. You can also import multiple plans in the web UI. Click on the All Plans breadcrumb on the Plan screen. This displays the current Plans listing. Change between multiple plans by clicking on a card.

Click Import Plan and a window appears asking you for the following information:

If you need to copy the director’s Heat template collection to a client machine, archive the files and copy them:

$ cd /usr/share/openstack-tripleo-heat-templates/
$ tar -cf ~/overcloud.tar *
$ scp ~/overcloud.tar user@10.0.0.55:~/.

Once the director UI uploads the plan, the plan appears in the Plans listing and you can now configure it. Click on the plan card of your choice.

The first step in configuring the overcloud is to register your nodes. Start the node registration process either through:

This displays the Register Nodes window.

The director requires a list of nodes for registration, which you can supply using one of two methods:

The details you need to provide for manual registration include the following:

After entering your node information, click Register Nodes at the bottom of the window.

The director registers the nodes. Once complete, you can use the UI to perform introspection on the nodes.

The director UI can run an introspection process on each node. This process causes each node to boot an introspection agent over PXE. This agent collects hardware data from the node and sends it back to the director. The director then stores this introspection data in the OpenStack Object Storage (swift) service running on the director. The director uses hardware information for various purposes such as profile tagging, benchmarking, and manual root disk assignment.

To start the introspection process:

Once the introspection process completes, select all nodes with the Provision State set to manageable then click the Provide Nodes button. Wait until the Provision State changes to available.

The nodes are now ready to tag and provision.

You can assign a set of profiles to each node. Each profile corresponds to a respective flavor and roles (see Section 6.4, “Tagging Nodes into Profiles” for more information).

The Nodes screen includes an additional menu toggle that provides extra node management actions, such as Tag Nodes.

To tag a set of nodes:

The Plan screen provides a method to customize your uploaded plan. Under 2 Specify Deployment Configuration, click the Edit Configuration link to modify your base overcloud configuration.

A window appears with two main tabs:

After registering and inspecting the hardware of each node, you assign them into roles from your plan.

To assign nodes to a role, scroll to the 3 Configure Roles and Assign Nodes section on the Plan screen. Each role uses a spinner widget to assign the number of nodes to a role. The available nodes per roles are based on the tagged nodes in Section 7.6, “Tagging Nodes into Profiles in the Web UI”.

This changes the *Count parameter for each role. For example, if you change the number of nodes in the Controller role to 3, this sets the ControllerCount parameter to 3. You can also view and edit these count values in the Parameters tab of the deployment configuration. See Section 7.7, “Editing Overcloud Plan Parameters in the Web UI” for more information.

Each node role provides a method for configuring role-specific parameters. Scroll to 3 Configure Roles and Assign Nodes roles on the Plan screen. Click the Edit Role Parameters icon (pencil icon) next to the role name.

A window appears that shows two main tabs:

Once the overcloud plan is configured, you can start the overcloud deployment. This involves scrolling to the 4 Deploy section and clicking Validate and Deploy.

If you have not run or passed all the validations for the undercloud, a warning message appears. Make sure that your undercloud host satisfies the requirements before running a deployment.

When you are ready to deploy, click Deploy.

The UI regularly monitors the progress of the overcloud’s creation and display a progress bar indicating the current percentage of progress. The View detailed information link displays a log of the current OpenStack Orchestration stacks in your overcloud.

Wait until the overcloud deployment completes.

After the overcloud creation process completes, the 4 Deploy section displays the current overcloud status and the following details:

Use this information to access your overcloud.

This concludes the creation of the overcloud through the director’s UI. For post-creation functions, see Chapter 9, Performing Tasks after Overcloud Creation.

At a later stage in this process, the director requires SSH access to the overcloud nodes as the stack user.

Each node requires access to a Red Hat subscription. The following procedure shows how to register each node to the Red Hat Content Delivery Network. Perform these steps on each node:

The node is now ready to use for your overcloud.

Each pre-provisioned node uses the OpenStack Orchestration (heat) agent to communicate with the director. The agent on each node polls the director and obtains metadata tailored to each node. This metadata allows the agent to configure each node.

Install the initial packages for the orchestration agent on each node:

[root@controller ~]# sudo yum -y install python-heat-agent*

If the director uses SSL/TLS, the pre-provisioned nodes require the certificate authority file used to sign the director’s SSL/TLS certificates. If using your own certificate authority, perform the following on each overcloud node:

This ensures the overcloud nodes can access the director’s Public API over SSL/TLS.

The pre-provisioned overcloud nodes obtain metadata from the director using standard HTTP requests. This means all overcloud nodes require L3 access to either:

The director uses a Control Plane network to manage and configure a standard overcloud. For an overcloud with pre-provisioned nodes, your network configuration might require some modification to accommodate how the director communicates with the pre-provisioned nodes.

During standard overcloud creation, the director creates OpenStack Networking (neutron) ports to automatically assigns IP addresses to the overcloud nodes on the Provisioning / Control Plane network. However, this can cause the director to assign different IP addresses to the ones manually configured for each node. In this situation, use a predictable IP address strategy to force the director to use the pre-provisioned IP assignments on the Control Plane.

An example of a predictable IP strategy is to use an environment file (ctlplane-assignments.yaml) with the following IP assignments:

resource_registry:
  OS::TripleO::DeployedServer::ControlPlanePort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml

parameter_defaults:
  DeployedServerPortMap:
    controller-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.2
      subnets:
        - cidr: 24
    compute-ctlplane:
      fixed_ips:
        - ip_address: 192.168.24.3
      subnets:
        - cidr: 24

In this example, the OS::TripleO::DeployedServer::ControlPlanePort resource passes a set of parameters to the director and defines the IP assignments of our pre-provisioned nodes. The DeployedServerPortMap parameter defines the IP addresses and subnet CIDRs that correspond to each overcloud node. The mapping defines:

A later step in this chapter uses the resulting environment file (ctlplane-assignments.yaml) as part of the openstack overcloud deploy command.

By default, the director uses the Provisioning network as the overcloud Control Plane. However, if this network is isolated and non-routable, nodes cannot communicate with the director’s Internal API during configuration. In this situation, you might need to define a separate network for the nodes and configure them to communicate with the director over the Public API.

There are several requirements for this scenario:

The examples in this section use IP address assignments that differ from the main scenario:

The following sections provide additional configuration for situations that require a separate network for overcloud nodes.

The hieradata_override in your undercloud.conf allows you to specify additional Puppet hieradata for undercloud configuration. Use the following steps to modify hieradata relevant to OpenStack Orchestration:

This switches the orchestration metadata server to use URLs on the director’s Public API.

resource_registry:
  OS::TripleO::DeployedServer::ControlPlanePort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::ControlPlaneVipPort: /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-neutron-port.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml 1

parameter_defaults:
  NeutronPublicInterface: eth1
  EC2MetadataIp: 192.168.100.1 2
  ControlPlaneDefaultRoute: 192.168.100.1
  DeployedServerPortMap:
    control_virtual_ip:
      fixed_ips:
        - ip_address: 192.168.100.1
      subnets:
        - cidr: 24
    controller0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.100.2
      subnets:
        - cidr: 24
    compute0-ctlplane:
      fixed_ips:
        - ip_address: 192.168.100.3
      subnets:
        - cidr: 24

The overcloud deployment uses the standard CLI methods from Section 6.8, “Creating the Overcloud with the CLI Tools”. For pre-provisioned nodes, the deployment command requires some additional options and environment files from the core Heat template collection:

The following is an example overcloud deployment command with the environment files specific to the pre-provisioned architecture:

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy \
  [other arguments] \
  --disable-validations \
  -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-bootstrap-environment-rhel.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-pacemaker-environment.yaml \
  -r /usr/share/openstack-tripleo-heat-templates/deployed-server/deployed-server-roles-data.yaml

This begins the overcloud configuration. However, the deployment stack pauses when the overcloud node resources enter the CREATE_IN_PROGRESS stage:

2017-01-14 13:25:13Z [overcloud.Compute.0.Compute]: CREATE_IN_PROGRESS  state changed
2017-01-14 13:25:14Z [overcloud.Controller.0.Controller]: CREATE_IN_PROGRESS  state changed

This pause is due to the director waiting for the orchestration agent on the overcloud nodes to poll the metadata server. The next section shows how to configure nodes to start polling the metadata server.

The deployment is now in progress but paused at a CREATE_IN_PROGRESS stage. The next step is to configure the orchestration agent on the overcloud nodes to poll the metadata server on the director. There are two ways to accomplish this:

[stack@director ~]$ source ~/stackrc

In addition, the script also requires some additional environment variables to define the nodes roles and their IP addressess. These environment variables are:

The following commands demonstrate how to set these environment variables:

(undercloud) $ export OVERCLOUD_ROLES="ControllerDeployedServer ComputeDeployedServer"
(undercloud) $ export ControllerDeployedServer_hosts="192.168.100.2"
(undercloud) $ export ComputeDeployedServer_hosts="192.168.100.3"

Run the script to configure the orchestration agent on each overcloud node:

(undercloud) $ /usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/get-occ-config.sh

The script accomplishes the following:

Once the script completes, the overcloud nodes start polling orchestration service on the director. The stack deployment continues.

[stack@director ~]$ source ~/stackrc
(undercloud) $ for STACK in $(openstack stack resource list -n5 --filter name=deployed-server -c stack_name -f value overcloud) ; do STACKID=$(echo $STACK | cut -d '-' -f2,4 --output-delimiter " ") ; echo "== Metadata URL for $STACKID ==" ; openstack stack resource metadata $STACK deployed-server | jq -r '.["os-collect-config"].request.metadata_url' ; echo ; done

This displays the stack name and metadata URL for each node:

== Metadata URL for ControllerDeployedServer 0 ==
http://192.168.24.1:8080/v1/AUTH_6fce4e6019264a5b8283e7125f05b764/ov-edServer-ts6lr4tm5p44-deployed-server-td42md2tap4g/43d302fa-d4c2-40df-b3ac-624d6075ef27?temp_url_sig=58313e577a93de8f8d2367f8ce92dd7be7aac3a1&temp_url_expires=2147483586

== Metadata URL for ComputeDeployedServer 0 ==
http://192.168.24.1:8080/v1/AUTH_6fce4e6019264a5b8283e7125f05b764/ov-edServer-wdpk7upmz3eh-deployed-server-ghv7ptfikz2j/0a43e94b-fe02-427b-9bfe-71d2b7bb3126?temp_url_sig=8a50d8ed6502969f0063e79bb32592f4203a136e&temp_url_expires=2147483586

On each overcloud node:

After you have configured and restarted them, the orchestration agents poll the director’s orchestration service for overcloud configuration. The deployment stack continues its creation and the stack for each node eventually changes to CREATE_COMPLETE.

The overcloud configuration process begins. This process takes some time to complete. To view the status of the overcloud creation, open a separate terminal as the stack user and run:

[stack@director ~]$ source ~/stackrc
(undercloud) $ heat stack-list --show-nested

The heat stack-list --show-nested command shows the current stage of the overcloud creation.

The director generates a script to configure and help authenticate interactions with your overcloud from the director host. The director saves this file, overcloudrc, in your stack user’s home director. Run the following command to use this file:

(undercloud) $ source ~/overcloudrc

This loads the necessary environment variables to interact with your overcloud from the director host’s CLI. The command prompt changes to indicate this:

(overcloud) $

To return to interacting with the director’s host, run the following command:

(overcloud) $ source ~/stackrc
(undercloud) $

The process for scaling pre-provisioned nodes is similar to the standard scaling procedures in Chapter 10, Scaling the Overcloud. However, the process for adding new pre-provisioned nodes differs since pre-provisioned nodes do not use the standard registration and management process from OpenStack Bare Metal (ironic) and OpenStack Compute (nova).

The general process for scaling up the nodes is:

In most scaling operations, you need to obtain the UUID value of the node to pass to openstack overcloud node delete. To obtain this UUID, list the resources for the specific role:

$ openstack stack resource list overcloud -c physical_resource_id -c stack_name -n5 --filter type=OS::TripleO::<RoleName>Server

Replace <RoleName> in the above command with the actual name of the role that you are scaling down. For example, for the ComputeDeployedServer role:

$ openstack stack resource list overcloud -c physical_resource_id -c stack_name -n5 --filter type=OS::TripleO::ComputeDeployedServerServer

Use the stack_name column in the command output to identify the UUID associated with each node. The stack_name includes the integer value of the index of the node in the Heat resource group. For example, in the following sample output:

+------------------------------------+----------------------------------+
| physical_resource_id               | stack_name                       |
+------------------------------------+----------------------------------+
| 294d4e4d-66a6-4e4e-9a8b-           | overcloud-ComputeDeployedServer- |
| 03ec80beda41                       | no7yfgnh3z7e-1-ytfqdeclwvcg      |
| d8de016d-                          | overcloud-ComputeDeployedServer- |
| 8ff9-4f29-bc63-21884619abe5        | no7yfgnh3z7e-0-p4vb3meacxwn      |
| 8c59f7b1-2675-42a9-ae2c-           | overcloud-ComputeDeployedServer- |
| 2de4a066f2a9                       | no7yfgnh3z7e-2-mmmaayxqnf3o      |
+------------------------------------+----------------------------------+

The indices 0, 1, or 2 in the stack_name column correspond to the node order in the Heat resource group. Pass the corresponding UUID value from the physical_resource_id column to openstack overcloud node delete command.

Once you have removed overcloud nodes from the stack, power off these nodes. Under a standard deployment, the bare metal services on the director control this function. However, with pre-provisioned nodes, you should either manually shutdown these nodes or use the power management control for each physical system. If you do not power off the nodes after removing them from the stack, they might remain operational and reconnect as part of the overcloud environment.

After powering down the removed nodes, reprovision them back to a base operating system configuration so that they do not unintentionally join the overcloud in the future

Removing an entire overcloud that uses pre-provisioned nodes uses the same procedure as a standard overcloud. See Section 9.13, “Removing the Overcloud” for more details.

After removing the overcloud, power off all nodes and reprovision them back to a base operating system configuration.

This concludes the creation of the overcloud using pre-provisioned nodes. For post-creation functions, see Chapter 9, Performing Tasks after Overcloud Creation.

The overcloud runs most OpenStack Platform services in containers. In certain situations, you might need to control the individual services on a host. This section provides some common docker commands you can run on an overcloud node to manage containerized services. For more comprehensive information on using docker to manage containers, see "Working with Docker formatted containers" in the Getting Started with Containers guide.

$ sudo docker ps

To also list stopped or failed containers, add the --all option:

$ sudo docker ps --all

To list container images:

$ sudo docker images

$ sudo docker inspect keystone

$ sudo docker restart keystone

To stop a containerized service, use the docker stop command. For example, to stop the keystone container:

$ sudo docker stop keystone

To start a stopped containerized service, use the docker start command. For example, to start the keystone container:

$ sudo docker start keystone

$ sudo docker logs keystone

$ sudo docker exec -it keystone /bin/bash

To enter the shell for the keystone container as the root user:

$ sudo docker exec --user 0 -it <NAME OR ID> /bin/bash

To exit from the container:

# exit

For information about troubleshooting OpenStack Platform containerized services, see Section 12.7.3, “Containerized Service Failures”.

The overcloud requires a Tenant network for instances. Source the overcloud and create an initial Tenant network in Neutron. For example:

$ source ~/overcloudrc
(overcloud) $ openstack network create default
(overcloud) $ openstack subnet create default --network default --gateway 172.20.1.1 --subnet-range 172.20.0.0/16

This creates a basic Neutron network called default. The overcloud automatically assigns IP addresses from this network using an internal DHCP mechanism.

Confirm the created network:

(overcloud) $ openstack network list
+-----------------------+-------------+--------------------------------------+
| id                    | name        | subnets                              |
+-----------------------+-------------+--------------------------------------+
| 95fadaa1-5dda-4777... | default     | 7e060813-35c5-462c-a56a-1c6f8f4f332f |
+-----------------------+-------------+--------------------------------------+

You need to create the External network on the overcloud so that you can assign floating IP addresses to instances.

Source the overcloud and create an External network in Neutron. For example:

$ source ~/overcloudrc
(overcloud) $ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre
(overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

In this example, you create a network with the name public. The overcloud requires this specific name for the default floating IP pool. This is also important for the validation tests in Section 9.7, “Validating the Overcloud”.

This command also maps the network to the datacentre physical network. As a default, datacentre maps to the br-ex bridge. Leave this option as the default unless you have used custom neutron settings during the overcloud creation.

$ source ~/overcloudrc
(overcloud) $ openstack network create public --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 104
(overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

The provider:segmentation_id value defines the VLAN to use. In this case, you can use 104.

Confirm the created network:

(overcloud) $ openstack network list
+-----------------------+-------------+--------------------------------------+
| id                    | name        | subnets                              |
+-----------------------+-------------+--------------------------------------+
| d474fe1f-222d-4e32... | public      | 01c5f621-1e0f-4b9d-9c30-7dc59592a52f |
+-----------------------+-------------+--------------------------------------+

Floating IP networks can use any bridge, not just br-ex, as long as you meet the following conditions:

Create the Floating IP network after creating the overcloud:

$ source ~/overcloudrc
(overcloud) $ openstack network create ext-net --external --provider-physical-network floating --provider-network-type vlan --provider-segment 105
(overcloud) $ openstack subnet create ext-subnet --network ext-net --dhcp --allocation-pool start=10.1.2.51,end=10.1.2.250 --gateway 10.1.2.1 --subnet-range 10.1.2.0/24

A provider network is a network attached physically to a network existing outside of the deployed overcloud. This can be an existing infrastructure network or a network that provides external access directly to instances through routing instead of floating IPs.

When creating a provider network, you associate it with a physical network, which uses a bridge mapping. This is similar to floating IP network creation. You add the provider network to both the Controller and the Compute nodes because the Compute nodes attach VM virtual network interfaces directly to the attached network interface.

For example, if the desired provider network is a VLAN on the br-ex bridge, use the following command to add a provider network on VLAN 201:

$ source ~/overcloudrc
(overcloud) $ openstack network create provider_network --provider-physical-network datacentre --provider-network-type vlan --provider-segment 201 --share

This command creates a shared network. It is also possible to specify a tenant instead of specifying --share. That network will only be available to the specified tenant. If you mark a provider network as external, only the operator may create ports on that network.

Add a subnet to a provider network if you want neutron to provide DHCP services to the tenant instances:

(overcloud) $ openstack subnet create provider-subnet --network  provider_network --dhcp --allocation-pool start=10.9.101.50,end=10.9.101.100 --gateway 10.9.101.254 --subnet-range 10.9.101.0/24

Other networks might require access externally through the provider network. In this situation, create a new router so that other networks can route traffic through the provider network:

(overcloud) $ openstack router create external
(overcloud) $ openstack router set --external-gateway provider_network external

Attach other networks to this router. For example, if you had a subnet called subnet1, you can attach it to the router with the following commands:

(overcloud) $ openstack router add subnet external subnet1

This adds subnet1 to the routing table and allows traffic using subnet1 to route to the provider network.

Validation steps in this guide assume that your installation contains flavors. If you have not already created at least one flavor, use the following commands to create a basic set of default flavors that have a range of storage and processing capability:

$ openstack flavor create m1.tiny --ram 512 --disk 0 --vcpus 1
$ openstack flavor create m1.smaller --ram 1024 --disk 0 --vcpus 1
$ openstack flavor create m1.small --ram 2048 --disk 10 --vcpus 1
$ openstack flavor create m1.medium --ram 3072 --disk 10 --vcpus 2
$ openstack flavor create m1.large --ram 8192 --disk 10 --vcpus 4
$ openstack flavor create m1.xlarge --ram 8192 --disk 10 --vcpus 8

Use $ openstack flavor create --help to learn more about the openstack flavor create command.

The overcloud uses the OpenStack Integration Test Suite (tempest) tool set to conduct a series of integration tests. This section provides information on preparations for running the integration tests. For full instruction on using the OpenStack Integration Test Suite, see the OpenStack Integration Test Suite Guide.

$ source ~/stackrc
(undercloud) $ sudo ovs-vsctl add-port br-ctlplane vlan201 tag=201 -- set interface vlan201 type=internal
(undercloud) $ sudo ip l set dev vlan201 up; sudo ip addr add 172.16.0.201/24 dev vlan201

Before running the OpenStack Integration Test Suite, check that the heat_stack_owner role exists in your overcloud:

$ source ~/overcloudrc
(overcloud) $ openstack role list
+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 6226a517204846d1a26d15aae1af208f | swiftoperator    |
| 7c7eb03955e545dd86bbfeb73692738b | heat_stack_owner |
+----------------------------------+------------------+

If the role does not exist, create it:

(overcloud) $ openstack role create heat_stack_owner

$ source ~/stackrc
(undercloud) $ sudo ovs-vsctl del-port vlan201

Sometimes you might intend to modify the overcloud to add additional features, or change the way it operates. To modify the overcloud, make modifications to your custom environment files and Heat templates, then rerun the openstack overcloud deploy command from your initial overcloud creation. For example, if you created an overcloud using Section 6.8, “Creating the Overcloud with the CLI Tools”, you would rerun the following command:

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates \
  -e ~/templates/node-info.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/network-environment.yaml \
  -e ~/templates/storage-environment.yaml \
  --ntp-server pool.ntp.org

The director checks the overcloud stack in heat, and then updates each item in the stack with the environment files and heat templates. It does not recreate the overcloud, but rather changes the existing overcloud.

If you aim to include a new environment file, add it to the openstack overcloud deploy command with a -e option. For example:

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates \
  -e ~/templates/new-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/network-environment.yaml \
  -e ~/templates/storage-environment.yaml \
  -e ~/templates/node-info.yaml \
  --ntp-server pool.ntp.org

This includes the new parameters and resources from the environment file into the stack.

Use the following procedure if you have an existing OpenStack environment and aim to migrate its virtual machines to your Red Hat OpenStack Platform environment.

Create a new image by taking a snapshot of a running server and download the image.

$ source ~/overcloudrc
(overcloud) $ openstack server image create instance_name --name image_name
(overcloud) $ openstack image save image_name --file exported_vm.qcow2

Upload the exported image into the overcloud and launch a new instance.

(overcloud) $ openstack image create imported_image --file exported_vm.qcow2 --disk-format qcow2 --container-format bare
(overcloud) $ openstack server create  imported_instance --key-name default --flavor m1.demo --image imported_image --nic net-id=net_id

In some situations, you might perform maintenance on an overcloud Compute node. To prevent downtime, migrate the VMs on the Compute node to another Compute node in the overcloud.

The director configures all Compute nodes to provide secure migration. All Compute nodes also require a shared SSH key to provide each host’s nova user with access to other Compute nodes during the migration process. The director creates this key using the OS::TripleO::Services::NovaCompute composable service. This composable service is one of the main services included on all Compute roles by default (see "Composable Services and Custom Roles" in Advanced Overcloud Customization).

To migrate an instance:

This migrates all instances from a Compute node. You can now perform maintenance on the node without any instance downtime. To return the Compute node to an enabled state, run the following command:

$ source ~/overcloudrc
(overcloud) $ openstack compute service set [hostname] nova-compute --enable

The director provides the ability to run Ansible-based automation on your OpenStack Platform environment. The director uses the tripleo-ansible-inventory command to generate a dynamic inventory of nodes in your environment.

To view a dynamic inventory of nodes, run the tripleo-ansible-inventory command after sourcing stackrc:

$ source ~/stackrc
(undercloud) $ tripleo-ansible-inventory --list

The --list option provides details on all hosts.

This outputs the dynamic inventory in a JSON format:

{"overcloud": {"children": ["controller", "compute"], "vars": {"ansible_ssh_user": "heat-admin"}}, "controller": ["192.168.24.2"], "undercloud": {"hosts": ["localhost"], "vars": {"overcloud_horizon_url": "http://192.168.24.4:80/dashboard", "overcloud_admin_password": "abcdefghijklm12345678", "ansible_connection": "local"}}, "compute": ["192.168.24.3"]}

To execute Ansible playbooks on your environment, run the ansible command and include the full path of the dynamic inventory tool using the -i option. For example:

(undercloud) $ ansible [HOSTS] -i /bin/tripleo-ansible-inventory [OTHER OPTIONS]

To avoid accidental removal of the overcloud with the heat stack-delete overcloud command, Heat contains a set of policies to restrict certain actions. Edit the /etc/heat/policy.json and find the following parameter:

"stacks:delete": "rule:deny_stack_user"

Change it to:

"stacks:delete": "rule:deny_everybody"

Save the file.

This prevents removal of the overcloud with the heat client. To allow removal of the overcloud, revert the policy to the original value.

The whole overcloud can be removed when desired.

Delete any existing overcloud:

$ source ~/stackrc
(undercloud) $ openstack overcloud delete overcloud

Confirm the deletion of the overcloud:

(undercloud) $ openstack stack list

Deletion takes a few minutes.

Once the removal completes, follow the standard steps in the deployment scenarios to recreate your overcloud.

The Identity Service (keystone) uses a token-based system for access control against the other OpenStack services. After a certain period, the database will accumulate a large number of unused tokens; a default cron job flushes the token table every day. It is recommended that you monitor your environment and adjust the token flush interval as needed.

For the overcloud, you can adjust the interval using the KeystoneCronToken values. For more information, see the Overcloud Parameters guide.

To add more nodes to the director’s node pool, create a new JSON file (for example, newnodes.json) containing the new node details to register:

{
  "nodes":[
    {
        "mac":[
            "dd:dd:dd:dd:dd:dd"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.168.24.207"
    },
    {
        "mac":[
            "ee:ee:ee:ee:ee:ee"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.168.24.208"
    }
  ]
}

See Section 6.1, “Registering Nodes for the Overcloud” for an explanation of these parameters.

Run the following command to register these nodes:

$ source ~/stackrc
(undercloud) $ openstack overcloud node import newnodes.json

After registering the new nodes, launch the introspection process for them. Use the following commands for each new node:

(undercloud) $ openstack baremetal node manage [NODE UUID]
(undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

This detects and benchmarks the hardware properties of the nodes.

After the introspection process completes, tag each new node for its desired role. For example, for a Compute node, use the following command:

(undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]

Scaling the overcloud requires running the openstack overcloud deploy again with the desired number of nodes for a role. For example, to scale to 5 Compute nodes:

(undercloud) $ openstack overcloud deploy --templates --compute-scale 5 [OTHER_OPTIONS]

This updates the entire overcloud stack. Note that this only updates the stack. It does not delete the overcloud and replace the stack.

There might be situations where you need to remove Compute nodes from the overcloud. For example, you might need to replace a problematic Compute node.

Next, disable the node’s Compute service on the overcloud. This stops the node from scheduling new instances.

$ source ~/overcloudrc
(overcloud) $ openstack compute service list
(overcloud) $ openstack compute service set [hostname] nova-compute --disable

Switch back to the undercloud:

(overcloud) $ source ~/stackrc

Removing overcloud nodes requires an update to the overcloud stack in the director using the local template files. First identify the UUID of the overcloud stack:

(undercloud) $ openstack stack list

Identify the UUIDs of the nodes to delete:

(undercloud) $ openstack server list

Run the following command to delete the nodes from the stack and update the plan accordingly:

(undercloud) $ openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID] [NODE3_UUID]

Finally, remove the node’s Compute service:

(undercloud) $ source ~/overcloudrc
(overcloud) $ openstack compute service list
(overcloud) $ openstack compute service delete [service-id]

And remove the node’s Open vSwitch agent:

(overcloud) $ openstack network agent list
(overcloud) $ openstack network agent delete [openvswitch-agent-id]

You are now free to remove the node from the overcloud and re-provision it for other purposes.

If a Compute node fails, you can replace the node with a working one. Replacing a Compute node uses the following process:

This process ensures that a node can be replaced without affecting the availability of any instances.

In certain circumstances a Controller node in a high availability cluster might fail. In these situations, you must remove the node from the cluster and replace it with a new Controller node. This also includes ensuring the node connects to the other nodes in the cluster.

This section provides instructions on how to replace a Controller node. The process involves running the openstack overcloud deploy command to update the overcloud with a request to replace a controller node. Note that this process is not completely automatic; during the overcloud stack update process, the openstack overcloud deploy command will at some point report a failure and halt the overcloud stack update. At this point, the process requires some manual intervention. Then the openstack overcloud deploy process can continue.

(undercloud) $ openstack server list
+--------------------------------------+------------------------+
| ID                                   | Name                   |
+--------------------------------------+------------------------+
| 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0    |
| 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1    |
| 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2    |
| a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 |
| cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 |
| 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 |
+--------------------------------------+------------------------+

(undercloud) $ openstack baremetal node list
+--------------------------------------+------+--------------------------------------+
| UUID                                 | Name | Instance UUID                        |
+--------------------------------------+------+--------------------------------------+
| 36404147-7c8a-41e6-8c72-a6e90afc7584 | None | 7bee57cf-4a58-4eaf-b851-2a8bf6620e48 |
| 91eb9ac5-7d52-453c-a017-c0e3d823efd0 | None | None                                 |
| 75b25e9a-948d-424a-9b3b-f0ef70a6eacf | None | None                                 |
| 038727da-6a5c-425f-bd45-fda2f4bd145b | None | 763bfec2-9354-466a-ae65-2401c13e07e5 |
| dc2292e6-4056-46e0-8848-d6e96df1f55d | None | 2017b481-706f-44e1-852a-2ee857c303c4 |
| c7eadcea-e377-4392-9fc3-cf2b02b7ec29 | None | 5f73c7d7-4826-49a5-b6be-8bfd558f3b41 |
| da3a8d19-8a59-4e9d-923a-6a336fe10284 | None | cfefaf60-8311-4bc3-9416-6a824a40a9ae |
| 807cb6ce-6b94-4cd1-9969-5c47560c2eee | None | c07c13e6-a845-4791-9628-260110829c3a |
+--------------------------------------+------+--------------------------------------+

(undercloud) $ openstack baremetal node maintenance set da3a8d19-8a59-4e9d-923a-6a336fe10284

(undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 75b25e9a-948d-424a-9b3b-f0ef70a6eacf

(undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs resource unmanage galera"

parameters:
  ControllerRemovalPolicies:
    [{'resource_list': ['1']}]

parameter_defaults:
  CorosyncSettleTries: 5

(undercloud) $ openstack overcloud deploy --templates --control-scale 3 -e ~/templates/remove-controller.yaml [OTHER OPTIONS]

(undercloud) $ openstack stack list --nested

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates --control-scale 3 [OTHER OPTIONS]

[heat-admin@overcloud-controller-0 ~]$ for i in $(sudo pcs property | grep overcloud-controller-0: | cut -d' ' -f 3- | tr  ' ' '\n' | grep role); do sudo pcs property set --node overcloud-controller-3 $i; done

[heat-admin@overcloud-controller-0 ~]$ sudo pcs status

[heat-admin@overcloud-controller-0 ~]$ exit

$ source ~/overcloudrc
(overcloud) $ neutron l3-agent-list-hosting-router r1

(overcloud) $ neutron agent-list | grep "neutron-l3-agent"

(overcloud) $ neutron l3-agent-router-add fd6b3d6e-7d8c-4e1a-831a-4ec1c9ebb965 r1
(overcloud) $ neutron l3-agent-router-remove b40020af-c6dd-4f7a-b426-eba7bac9dbc2 r1

(overcloud) $ neutron l3-agent-list-hosting-router r1

(overcloud) $ neutron agent-list -F id -F host | grep overcloud-controller-1
| ddae8e46-3e8e-4a1b-a8b3-c87f13c294eb | overcloud-controller-1.localdomain |
(overcloud) $ neutron agent-delete ddae8e46-3e8e-4a1b-a8b3-c87f13c294eb

[stack@director ~]$ source ~/overcloudrc
(overcloud) $ openstack compute service list --host overcloud-controller-1.localdomain

(overcloud) $ for SERVICE in $(openstack compute service list --host overcloud-controller-1.localdomain -f value -c ID) ; do openstack compute service delete $SERVICE ; done

The director provides a method to replace Ceph Storage nodes in a director-created cluster. You can find these instructions in the Deploying an Overcloud with Containerized Red Hat Ceph guide.

This section describes how to replace Object Storage nodes while maintaining the integrity of the cluster. In this example, we have a two-node Object Storage cluster where the node overcloud-objectstorage-1 needs to be replaced. Our aim is to add one more node, then remove overcloud-objectstorage-1 (effectively replacing it).

The director deletes the Object Storage node from the overcloud and updates the rest of the nodes on the overcloud to accommodate the node removal.

You can exclude overcloud nodes from receiving an updated deployment. This is useful in scenarios where you aim to scale new node while excluding existing nodes from receiving an updated set of parameters and resources from the core Heat template collection. In other words, the blacklisted nodes are isolated from the effects of the stack operation.

Use the DeploymentServerBlacklist parameter in an environment file to create a blacklist.

parameter_defaults:
  DeploymentServerBlacklist:
    - overcloud-compute-0
    - overcloud-compute-1
    - overcloud-compute-2

Include this environment file with your openstack overcloud deploy command. For example:

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates \
  -e server-blacklist.yaml \
  [OTHER OPTIONS]

Heat blacklists any servers in the list from receiving updated Heat deployments. After the stack operation completes, any blacklisted servers remain unchanged. You can also power off or stop the os-collect-config agents during the operation.

parameter_defaults:
  DeploymentServerBlacklist: []

To reboot the director node, follow this process:

To reboot the Controller nodes, follow this process:

To reboot the Ceph Storage nodes, follow this process:

Reboot each Compute node individually and ensure zero downtime of instances in your OpenStack Platform environment. This involves the following workflow:

List all Compute nodes and their UUIDs:

$ source ~/stackrc
(undercloud) $ openstack server list --name compute

Select a Compute node to reboot and first migrate its instances using the following process:

Reboot the Compute node using the following process

To reboot the Object Storage nodes, follow this process:

Issues with node registration usually arise from issues with incorrect node details. In this case, use ironic to fix problems with node data registered. Here are a few examples:

Find out the assigned port UUID:

$ source ~/stackrc
(undercloud) $ openstack baremetal port list --node [NODE UUID]

Update the MAC address:

(undercloud) $ openstack baremetal port set --address=[NEW MAC] [PORT UUID]

Run the following command:

(undercloud) $ openstack baremetal node set --driver-info ipmi_address=[NEW IPMI ADDRESS] [NODE UUID]

The introspection process must run to completion. However, ironic’s Discovery daemon (ironic-inspector) times out after a default 1 hour period if the discovery ramdisk provides no response. Sometimes this might indicate a bug in the discovery ramdisk but usually it happens due to an environment misconfiguration, particularly BIOS boot settings.

Here are some common scenarios where environment misconfiguration occurs and advice on how to diagnose and resolve them.

$ source ~/stackrc
(undercloud) $ openstack baremetal node manage [NODE UUID]

Then, when discovery completes, change back to AVAILABLE before provisioning:

(undercloud) $ openstack baremetal node provide [NODE UUID]

$ source ~/stackrc
(undercloud) $ openstack baremetal introspection abort [NODE UUID]

You can also wait until the process times out. If necessary, change the timeout setting in /etc/ironic-inspector/inspector.conf to another period in minutes.

$ sudo systemctl list-units openstack-swift*

The OpenStack Workflow (mistral) service groups multiple OpenStack tasks into workflows. Red Hat OpenStack Platform uses a set of these workflow to perform common functions across the CLI and web UI. This includes bare metal node control, validations, plan management, and overcloud deployment.

For example, when running the openstack overcloud deploy command, the OpenStack Workflow service executes two workflows. The first one uploads the deployment plan:

Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: aef1e8c6-a862-42de-8bce-073744ed5e6b
Plan updated

The second one starts the overcloud deployment:

Deploying templates in the directory /tmp/tripleoclient-LhRlHX/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 97b64abe-d8fc-414a-837a-1380631c764d
2016-11-28 06:29:26Z [overcloud]: CREATE_IN_PROGRESS  Stack CREATE started
2016-11-28 06:29:26Z [overcloud.Networks]: CREATE_IN_PROGRESS  state changed
2016-11-28 06:29:26Z [overcloud.HeatAuthEncryptionKey]: CREATE_IN_PROGRESS  state changed
2016-11-28 06:29:26Z [overcloud.ServiceNetMap]: CREATE_IN_PROGRESS  state changed
...

$ source ~/stackrc
(undercloud) $ openstack workflow execution list | grep "ERROR"

Get the UUID of the failed workflow execution (for example, dffa96b0-f679-4cd2-a490-4769a3825262) and view the execution and its output:

(undercloud) $ openstack workflow execution show dffa96b0-f679-4cd2-a490-4769a3825262
(undercloud) $ openstack workflow execution output show dffa96b0-f679-4cd2-a490-4769a3825262

This provides information about the failed task in the execution. The openstack workflow execution show also displays the workflow used for the execution (for example, tripleo.plan_management.v1.publish_ui_logs_to_swift). You can view the full workflow definition using the following command:

(undercloud) $ openstack workflow definition show tripleo.plan_management.v1.publish_ui_logs_to_swift

This is useful for identifying where in the workflow a particular task occurs.

You can also view action executions and their results using a similar command syntax:

(undercloud) $ openstack action execution list
(undercloud) $ openstack action execution show 8a68eba3-0fec-4b2a-adc9-5561b007e886
(undercloud) $ openstack action execution output show 8a68eba3-0fec-4b2a-adc9-5561b007e886

This is useful for identifying a specific action causing issues.

There are three layers where the deployment can fail:

If an overcloud deployment has failed at any of these levels, use the OpenStack clients and service log files to diagnose the failed deployment.

$ source ~/stackrc
(undercloud) $ openstack stack list --nested --property status=FAILED
+-----------------------+------------+--------------------+----------------------+
| id                    | stack_name | stack_status       | creation_time        |
+-----------------------+------------+--------------------+----------------------+
| 7e88af95-535c-4a55... | overcloud  | CREATE_FAILED      | 2015-04-06T17:57:16Z |
+-----------------------+------------+--------------------+----------------------+

$ source ~/stackrc
(undercloud) $ openstack baremetal node list

+----------+------+---------------+-------------+-----------------+-------------+
| UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
+----------+------+---------------+-------------+-----------------+-------------+
| f1e261...| None | None          | power off   | available       | False       |
| f0b8c1...| None | None          | power off   | available       | False       |
+----------+------+---------------+-------------+-----------------+-------------+

$ source ~/stackrc
(undercloud) $ openstack stack resource list overcloud --filter status=FAILED

(undercloud) $ openstack stack resource show overcloud [FAILED RESOURCE]

(undercloud) $ openstack server list

(undercloud) $ ssh heat-admin@192.168.24.14

[heat-admin@overcloud-controller-0 ~]$ sudo journalctl -u os-collect-config

(undercloud) $ openstack server list
(undercloud) $ openstack server show [SERVER ID]

Discovery and deployment tasks will fail if the destination hosts are allocated an IP address which is already in use. To avoid this issue, you can perform a port scan of the Provisioning network to determine whether the discovery IP range and host IP range are free.

Perform the following steps from the undercloud host:

Install nmap:

$ sudo yum install nmap

Use nmap to scan the IP address range for active addresses. This example scans the 192.168.24.0/24 range, replace this with the IP subnet of the Provisioning network (using CIDR bitmask notation):

$ sudo nmap -sn 192.168.24.0/24

Review the output of the nmap scan:

For example, you should see the IP address(es) of the undercloud, and any other hosts that are present on the subnet. If any of the active IP addresses conflict with the IP ranges in undercloud.conf, you will need to either change the IP address ranges or free up the IP addresses before introspecting or deploying the overcloud nodes.

$ sudo nmap -sn 192.168.24.0/24

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-02 15:14 EDT
Nmap scan report for 192.168.24.1
Host is up (0.00057s latency).
Nmap scan report for 192.168.24.2
Host is up (0.00048s latency).
Nmap scan report for 192.168.24.3
Host is up (0.00045s latency).
Nmap scan report for 192.168.24.5
Host is up (0.00040s latency).
Nmap scan report for 192.168.24.9
Host is up (0.00019s latency).
Nmap done: 256 IP addresses (5 hosts up) scanned in 2.45 seconds

Sometimes the /var/log/nova/nova-conductor.log contains the following error:

NoValidHost: No valid host was found. There are not enough hosts available.

This means the nova Scheduler could not find a bare metal node suitable for booting the new instance. This in turn usually means a mismatch between resources that nova expects to find and resources that ironic advertised to nova. Check the following in this case:

After creating your overcloud, you might want to perform certain overcloud operations in the future. For example, you might aim to scale your available nodes, or replace faulty nodes. Certain issues might arise when performing these operations. This section provides some advice to diagnose and troubleshoot failed post-creation operations.

$ sudo docker logs keystone

$ sudo docker inspect keystone

$ sudo docker inspect keystone | jq .[0].Mounts

$ sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone

$ OPTIONS=$( sudo docker inspect --format='{{range .Config.Env}} -e "{{.}}" {{end}} {{range .Mounts}} -v {{.Source}}:{{.Destination}}{{if .Mode}}:{{.Mode}}{{end}}{{end}} -ti {{.Config.Image}}' keystone )
$ sudo docker run --rm $OPTIONS /bin/bash

$ sudo docker exec -ti keystone <COMMAND>

$ sudo docker exec -ti keystone /openstack/healthcheck

$ sudo docker exec -ti keystone /bin/bash

$ sudo docker export keystone -o keystone.tar

The advice in this section aims to help increase the performance of your undercloud. Implement the recommendations as necessary.

If you need to contact Red Hat for support on OpenStack Platform, you might need to generate an sosreport. See the following knowledgebase article for more information on how to create an sosreport:

Use the following logs to find out information about the undercloud and overcloud when troubleshooting.