Menu Close
Settings Close

Language and Page Formatting Options

Chapter 20. Replacing Networker nodes

In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. (For more information, see Tagging nodes into profiles in the Director Installation and Usage guide.) In these situations, you must remove the node from the cluster and replace it with a new Networker node that runs the Networking service (neutron) agents.

The topics in this section are:

20.1. Preparing to replace network nodes

Replacing a Networker node on a Red Hat OpenStack Platform (RHOSP) overcloud, requires that you perform several preparation steps. Completing all of the required preparation steps helps you to avoid complications during the Networker node replacement process.

Prerequisites

  • Your RHOSP deployment is highly available with three or more Networker nodes.

Procedure

  1. Log in to your undercloud as the stack user.
  2. Source the undercloud credentials file:

    $ source ~/stackrc
  3. Check the current status of the overcloud stack on the undercloud:

    $ openstack stack list --nested

    The overcloud stack and its subsequent child stacks should have a status of either CREATE_COMPLETE or UPDATE_COMPLETE.

  4. Ensure that you have a recent backup image of the undercloud node by running the Relax-and-Recover tool.

    For more information, see the Backing up and restoring the undercloud and control plane nodes guide.

  5. Log in to a Controller node as root.
  6. Open an interactive bash shell on the container and check the status of the Galera cluster:

    # pcs status

    Ensure that the Controller nodes are in Master mode.

    Sample output

    * Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]:
         * galera-bundle-0   (ocf::heartbeat:galera):         Master controller-0
         * galera-bundle-1   (ocf::heartbeat:galera):         Master controller-1
         * galera-bundle-2   (ocf::heartbeat:galera):         Master controller-2

  7. Log on to the RHOSP director node and check the nova-compute service:

    $ sudo systemctl status tripleo_nova_compute
    $ openstack baremetal node list

    The output should show all non-maintenance mode nodes as up.

  8. Make sure all undercloud services are running:

    $ sudo systemctl -t service

20.2. Replacing a Networker node

In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. Replacing a Networker node requires running the openstack overcloud deploy command to update the overcloud with a the new node.

Prerequisites

  • Your RHOSP deployment is highly available with three or more Networker nodes.
  • The node that you add must be able to connect to the other nodes in the cluster over the network.
  • You have performed the steps described in Section 20.1, “Preparing to replace network nodes”

Procedure

  1. Log in to your undercloud as the stack user.
  2. Source the undercloud credentials file:

    Example

    $ source ~/stackrc

  3. Identify the index of the node to remove:

    $ openstack baremetal node list -c UUID -c Name -c "Instance UUID"

    Sample output

    +--------------------------------------+------+--------------------------------------+
    | UUID                                 | Name | Instance UUID                        |
    +--------------------------------------+------+--------------------------------------+
    | 36404147-7c8a-41e6-8c72-6af1e339da2a | None | 7bee57cf-4a58-4eaf-b851-f3203f6e5e05 |
    | 91eb9ac5-7d52-453c-a017-0f2fb289c3cd | None | None                                 |
    | 75b25e9a-948d-424a-9b3b-0f2fb289c3cd | None | None                                 |
    | 038727da-6a5c-425f-bd45-16aa2bc4ba91 | None | 763bfec2-9354-466a-ae65-1fdf45d35c61 |
    | dc2292e6-4056-46e0-8848-165d06fcc948 | None | 2017b481-706f-44e1-852a-57fb03ecef11 |
    | c7eadcea-e377-4392-9fc3-716f1bd57527 | None | 5f73c7d7-4826-49a5-b6be-0a95c6bdd2f8 |
    | da3a8d19-8a59-4e9d-923a-29254d688f6d | None | cfefaf60-8311-4bc3-9416-46852e2cb83f |
    | 807cb6ce-6b94-4cd1-9969-d390650854c7 | None | c07c13e6-a845-4791-9628-c8514585fe27 |
    | 0c245daa-7817-4ae9-a883-fed2e9c68d6c | None | 844c9a88-713a-4ff1-8737-30858d724593 |
    | e6499ef7-3db2-4ab4-bfa7-feb44c6591c6 | None | aef7c27a-f0b4-4814-b0ff-d3f792321212 |
    | 7545385c-bc49-4eb9-b13c-201368ce1c62 | None | c2e40164-c659-4849-a28f-a7b270ed2970 |
    +--------------------------------------+------+--------------------------------------+

  4. Set the node into maintenance mode by using the baremetal node maintenance set command.

    Example

    $ openstack baremetal node maintenance set e6499ef7-3db2-4ab4-bfa7-ef59539bf972

  5. Create a JSON file to add the new node to the node pool that contains RHOSP director.

    Example

    {
      "nodes":[
        {
            "mac":[
                "dd:dd:dd:dd:dd:dd"
            ],
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"ipmi",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.168.24.207"
        }
      ]
    }

    For more information, see Adding nodes to the overcloud in the Director Installation and Usage guide.

  6. Run the openstack overcloud node import command to register the new node.

    Example

    $ openstack overcloud node import newnode.json

  7. After registering the new node, launch the introspection process by using the following commands:

    $ openstack baremetal node manage <node>
    $ openstack overcloud node introspect <node> --provide
  8. Tag the new node with the Networker profile by using the openstack baremetal node set command.

    Example

    $ openstack baremetal node set --property \
        capabilities='profile:networker,boot_option:local' \
        91eb9ac5-7d52-453c-a017-c0e3d823efd0

  9. Create a ~/templates/remove-networker.yaml environment file that defines the index of the node that you intend to remove:

    Example

    parameters:
    NetworkerRemovalPolicies:
       [{'resource_list': ['1']}]

  10. Create a ~/templates/node-count-networker.yaml environment file and set the total count of Networker nodes in the file.

    Example

    parameter_defaults:
     OvercloudNetworkerFlavor: networker
     NetworkerCount: 3

  11. Run the openstack overcloud deploy command and include the core heat templates, environment files, and the environment files that you modified.

    Important

    The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence.

    $ openstack overcloud deploy --templates \
    -e <your_environment_files> \
    -e /home/stack/templates/node-count-networker.yaml \
    -e /home/stack/templates/remove-networker.yaml

    RHOSP director removes the old Networker node, creates a new one, and updates the overcloud stack.

Verification

  1. Check the status of the overcloud stack:

    $ openstack stack list --nested
  2. Verify that the new Networker node is listed, and the old one is removed.

    $ openstack server list -c ID -c Name -c Status

    Sample output

    +--------------------------------------+------------------------+--------+
    | ID                                   | Name                   | Status |
    +--------------------------------------+------------------------+--------+
    | 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0    | ACTIVE |
    | 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1    | ACTIVE |
    | 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2    | ACTIVE |
    | a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 | ACTIVE |
    | cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 | ACTIVE |
    | 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 | ACTIVE |
    | 844c9a88-713a-4ff1-8737-6410bf551d4f | overcloud-networker-0  | ACTIVE |
    | c2e40164-c659-4849-a28f-507eb7edb79f | overcloud-networker-2  | ACTIVE |
    | 425a0828-b42f-43b0-940c-7fb02522753a | overcloud-networker-3  | ACTIVE |
    +--------------------------------------+------------------------+--------+

Additional resources

20.3. Rescheduling nodes and cleaning up the Networking service

As a part of replacing a Red Hat OpenStack Platform (RHOSP) Networker node, remove all Networking service agents on the removed node from the database. Doing so ensures that the Networking service does not identify the agents as out-of-service ("dead"). For ML2/OVS users, removing agents from the removed node enables the DHCP resources to be automatically rescheduled to other Networker nodes.

Prerequisites

  • Your RHOSP deployment is highly available with three or more Networker nodes.

Procedure

  1. Log in to your undercloud as the stack user.
  2. Source the overcloud credentials file:

    Example

    $ source ~/overcloudrc

  3. Verify that the RHOSP Networking service processes exist, and are marked out-of-service (xxx) for the overcloud-networker-1.

    $ openstack network agent list -c ID -c Binary -c Host -c Alive  | grep overcloud-networker-1

    Sample output for ML2/OVN

    +--------------------------------------+-----------------------+-------+-------------------------------+
    | ID                                   | Host                  | Alive | Binary                        |
    +--------------------------------------+-----------------------+-------+-------------------------------+
    | 26316f47-4a30-4baf-ba00-d33c9a9e0844 | overcloud-networker-1 | xxx   | ovn-controller                |
    +--------------------------------------+-----------------------+-------+-------------------------------+

    Sample output for ML2/OVS

    +--------------------------------------+-----------------------+-------+------------------------+
    | ID                                   | Host                  | Alive | Binary                 |
    +--------------------------------------+-----------------------+-------+------------------------+
    | 8377-66d75323e466c-b838-1149e10441ee | overcloud-networker-1 | xxx   | neutron-metadata-agent |
    | b55d-797668c336707-a2cf-cba875eeda21 | overcloud-networker-1 | xxx   | neutron-l3-agent       |
    | 9dcb-00a9e32ecde42-9458-01cfa9742862 | overcloud-networker-1 | xxx   | neutron-ovs-agent      |
    | be83-e4d9329846540-9ae6-1540947b2ffd | overcloud-networker-1 | xxx   | neutron-dhcp-agent     |
    +--------------------------------------+-----------------------+-------+------------------------+

  4. Capture the UUIDs of the agents registered for overcloud-networker-1.

    $ AGENT_UUIDS=$(openstack network agent list -c ID -c Host -c Alive -c Binary -f value | grep overcloud-networker-1 | cut -d\  -f1)
  5. Delete any remaining overcloud-networker-1 agents from the database.

    $ for agent in $AGENT_UUIDS; do neutron agent-delete $agent ; done

    Sample output

    Deleted agent(s): 26316f47-4a30-4baf-ba00-d33c9a9e0844

Additional resources