Chapter 10. Scaling overcloud nodes

Warning

Do not use openstack server delete to remove nodes from the overcloud. Read the procedures defined in this section to properly remove and replace nodes.

There might be situations where you need to add or remove nodes after the creation of the overcloud. For example, you might need to add more Compute nodes to the overcloud. This situation requires updating the overcloud.

Use the following table to determine support for scaling each node type:

Table 10.1. Scale Support for Each Node Type

Node Type

Scale Up?

Scale Down?

Notes

Controller

N

N

You can replace Controller nodes using the procedures in Chapter 11, Replacing Controller Nodes.

Compute

Y

Y

 

Ceph Storage Nodes

Y

N

You must have at least 1 Ceph Storage node from the initial overcloud creation.

Object Storage Nodes

Y

Y

 
Important

Ensure to leave at least 10 GB free space before scaling the overcloud. This free space accommodates image conversion and caching during the node provisioning process.

10.1. Adding nodes to the overcloud

Complete the following steps to add more nodes to the director node pool.

Procedure

  1. Create a new JSON file (newnodes.json) containing the new node details to register:

    {
      "nodes":[
        {
            "mac":[
                "dd:dd:dd:dd:dd:dd"
            ],
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"ipmi",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.168.24.207"
        },
        {
            "mac":[
                "ee:ee:ee:ee:ee:ee"
            ],
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"ipmi",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.168.24.208"
        }
      ]
    }
  2. Run the following command to register the new nodes:

    $ source ~/stackrc
    (undercloud) $ openstack overcloud node import newnodes.json
  3. After registering the new nodes, run the following commands to launch the introspection process for each new node:

    (undercloud) $ openstack baremetal node manage [NODE UUID]
    (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

    This process detects and benchmarks the hardware properties of the nodes.

10.2. Increasing node counts for roles

Complete the following steps to scale overcloud nodes for a specific role, such as a Compute node.

Procedure

  1. Tag each new node with the role you want. For example, to tag a node with the Compute role, run the following command:

    (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]
  2. Scaling the overcloud requires that you edit the environment file that contains your node counts and re-deploy the overcloud. For example, to scale your overcloud to 5 Compute nodes, edit the ComputeCount parameter:

    parameter_defaults:
      ...
      ComputeCount: 5
      ...
  3. Rerun the deployment command with the updated file, which in this example is called node-info.yaml:

    (undercloud) $ openstack overcloud deploy --templates -e /home/stack/templates/node-info.yaml [OTHER_OPTIONS]

    Ensure you include all environment files and options from your initial overcloud creation. This includes the same scale parameters for non-Compute nodes.

  4. Wait until the deployment operation completes.

10.3. Removing Compute nodes

There might be situations where you need to remove Compute nodes from the overcloud. For example, you might need to replace a problematic Compute node.

Important

Before removing a Compute node from the overcloud, migrate the workload from the node to other Compute nodes. See Section 8.12, “Migrating instances from a Compute node” for more details.

Procedure

  1. Source the overcloud configuration:

    $ source ~/overcloudrc
  2. Disable the Compute service on the outgoing node on the overcloud to prevent the node from scheduling new instances:

    (overcloud) $ openstack compute service list
    (overcloud) $ openstack compute service set [hostname] nova-compute --disable
    Tip

    Add a short explanation on why the service is being disabled using the --disable-reason option. This is useful if you intend to redeploy the Compute service at a later point.

  3. Source the undercloud configuration:

    (overcloud) $ source ~/stackrc
  4. When you remove overcloud nodes, you must update the overcloud stack in the director using the local template files. First, identify the UUID of the overcloud stack:

    (undercloud) $ openstack stack list
  5. Identify the UUIDs of the nodes to delete:

    (undercloud) $ openstack server list
  6. Run the following command to delete the nodes from the stack and update the plan accordingly:

    (undercloud) $ openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID] [NODE3_UUID]
    Important

    If you passed any extra environment files when you created the overcloud, pass them here again using the -e or --environment-file option to avoid making undesired manual changes to the overcloud.

  7. Ensure the openstack overcloud node delete command runs to completion before you continue. Use the openstack stack list command and check the overcloud stack has reached an UPDATE_COMPLETE status.

    Important

    If you intend to redeploy the Compute service using the same host name, then you need to use the existing service records for the redeployed node. If this is the case, skip the remaining steps in this procedure, and proceed with the instructions detailed in Redeploying the Compute service using the same host name.

  8. Remove the Compute service from the node:

    (undercloud) $ source ~/overcloudrc
    (overcloud) $ openstack compute service list
    (overcloud) $ openstack compute service delete [service-id]
  9. Remove the Open vSwitch agent from the node:

    (overcloud) $ openstack network agent list
    (overcloud) $ openstack network agent delete [openvswitch-agent-id]
  10. Remove the deleted Compute service as a resource provider from the placement service:

    (overcloud) $ openstack resource provider list
    (overcloud) $ openstack resource provider delete [uuid]
    Note

    To work with the placement service, install the python2-osc-placement package.

You are now free to remove the node from the overcloud and re-provision it for other purposes.

Redeploying the Compute service using the same host name

To redeploy a disabled Compute service, re-enable it after a node with the same host name is up again. For instance:

(overcloud) $ openstack compute service set compute-1.localdomain nova-compute --disable --disable-reason "gets re-provisioned"
(overcloud) $ openstack compute service list --long
...
| 80 | nova-compute | compute-1.localdomain | nova  | disabled | up | 2018-07-13T14:35:04.000000 | gets re-provisioned |
...
(overcloud) $ openstack compute service set compute-1.localdomain nova-compute --enable

10.4. Replacing Ceph Storage nodes

You can use the director to replace Ceph Storage nodes in a director-created cluster. You can find these instructions in the Deploying an Overcloud with Containerized Red Hat Ceph guide.

10.5. Replacing Object Storage nodes

Follow the instructions in this section to understand how to replace Object Storage nodes while maintaining the integrity of the cluster. This example involves a three-node Object Storage cluster in which the node overcloud-objectstorage-1 must be replaced. The goal of the procedure is to add one more node and then remove overcloud-objectstorage-1, effectively replacing it.

Procedure

  1. Increase the Object Storage count using the ObjectStorageCount parameter. This parameter is usually located in node-info.yaml, which is the environment file containing your node counts:

    parameter_defaults:
      ObjectStorageCount: 4

    The ObjectStorageCount parameter defines the quantity of Object Storage nodes in your environment. In this situation, we scale from 3 to 4 nodes.

  2. Run the deployment command with the updated ObjectStorageCount parameter:

    $ source ~/stackrc
    (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES
  3. After the deployment command completes, the overcloud contains an additional Object Storage node.
  4. Replicate data to the new node. Before removing a node (in this case, overcloud-objectstorage-1), wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service should log entries similar to the following example:

    Mar 29 08:49:05 localhost object-server: Object replication complete.
    Mar 29 08:49:11 localhost container-server: Replication run OVER
    Mar 29 08:49:13 localhost account-server: Replication run OVER
  5. To remove the old node from the ring, reduce the ObjectStorageCount parameter to the omit the old node. In this case, reduce it to 3:

    parameter_defaults:
      ObjectStorageCount: 3
  6. Create a new environment file named remove-object-node.yaml. This file identifies and removes the specified Object Storage node. The following content specifies the removal of overcloud-objectstorage-1:

    parameter_defaults:
      ObjectStorageRemovalPolicies:
        [{'resource_list': ['1']}]
  7. Include both the node-info.yaml and remove-object-node.yaml files in the deployment command:

    (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES -e remove-object-node.yaml

The director deletes the Object Storage node from the overcloud and updates the rest of the nodes on the overcloud to accommodate the node removal.

Important

Make sure to include all environment files and options from your initial overcloud creation. This includes the same scale parameters for non-Compute nodes.

10.6. Blacklisting nodes

You can exclude overcloud nodes from receiving an updated deployment. This is useful in scenarios where you aim to scale new nodes while excluding existing nodes from receiving an updated set of parameters and resources from the core Heat template collection. In other words, the blacklisted nodes are isolated from the effects of the stack operation.

Use the DeploymentServerBlacklist parameter in an environment file to create a blacklist.

Setting the Blacklist

The DeploymentServerBlacklist parameter is a list of server names. Write a new environment file, or add the parameter value to an existing custom environment file and pass the file to the deployment command:

parameter_defaults:
  DeploymentServerBlacklist:
    - overcloud-compute-0
    - overcloud-compute-1
    - overcloud-compute-2
Note

The server names in the parameter value are the names according to OpenStack Orchestration (heat), not the actual server hostnames.

Include this environment file with your openstack overcloud deploy command:

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates \
  -e server-blacklist.yaml \
  [OTHER OPTIONS]

Heat blacklists any servers in the list from receiving updated Heat deployments. After the stack operation completes, any blacklisted servers remain unchanged. You can also power off or stop the os-collect-config agents during the operation.

Warning
  • Exercise caution when blacklisting nodes. Only use a blacklist if you fully understand how to apply the requested change with a blacklist in effect. It is possible to create a hung stack or configure the overcloud incorrectly using the blacklist feature. For example, if a cluster configuration changes applies to all members of a Pacemaker cluster, blacklisting a Pacemaker cluster member during this change can cause the cluster to fail.
  • Do not use the blacklist during update or upgrade procedures. Those procedures have their own methods for isolating changes to particular servers. See the Upgrading Red Hat OpenStack Platform documentation for more information.
  • When adding servers to the blacklist, further changes to those nodes are not supported until the server is removed from the blacklist. This includes updates, upgrades, scale up, scale down, and node replacement.

Clearing the Blacklist

To clear the blacklist for subsequent stack operations, edit the DeploymentServerBlacklist to use an empty array:

parameter_defaults:
  DeploymentServerBlacklist: []
Warning

Do not just omit the DeploymentServerBlacklist parameter. If you omit the parameter, the overcloud deployment uses the previously saved value.