Chapter 21. Replacing Networker nodes

Procedure

1. Log in to your undercloud as the stack user.

2. Source the undercloud credentials file:

   $ source ~/stackrc

3. Check the current status of the overcloud stack on the undercloud:

   $ openstack stack list --nested

   The overcloud stack and its subsequent child stacks should have a status of either CREATE_COMPLETE or UPDATE_COMPLETE.

4. Ensure that you have a recent backup image of the undercloud node by running the Relax-and-Recover tool.

   For more information, see the Backing up and restoring the undercloud and control plane nodes guide.

5. Log in to a Controller node as root.

6. Open an interactive bash shell on the container and check the status of the Galera cluster:

   # pcs status

   Ensure that the Controller nodes are in Master mode.

   Sample output

   * Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]:
        * galera-bundle-0   (ocf::heartbeat:galera):         Master controller-0
        * galera-bundle-1   (ocf::heartbeat:galera):         Master controller-1
        * galera-bundle-2   (ocf::heartbeat:galera):         Master controller-2

7. Log on to the RHOSP director node and check the nova-compute service:

   $ sudo systemctl status tripleo_nova_compute
   $ openstack baremetal node list

   The output should show all non-maintenance mode nodes as up.

8. Make sure all undercloud services are running:

   $ sudo systemctl -t service

Procedure

1. Log in to your undercloud as the stack user.

2. Source the undercloud credentials file:

   Example

   $ source ~/stackrc

3. Identify the index of the node to remove:

   $ openstack baremetal node list -c UUID -c Name -c "Instance UUID"

   Sample output

   +--------------------------------------+------+--------------------------------------+
   | UUID                                 | Name | Instance UUID                        |
   +--------------------------------------+------+--------------------------------------+
   | 36404147-7c8a-41e6-8c72-6af1e339da2a | None | 7bee57cf-4a58-4eaf-b851-f3203f6e5e05 |
   | 91eb9ac5-7d52-453c-a017-0f2fb289c3cd | None | None                                 |
   | 75b25e9a-948d-424a-9b3b-0f2fb289c3cd | None | None                                 |
   | 038727da-6a5c-425f-bd45-16aa2bc4ba91 | None | 763bfec2-9354-466a-ae65-1fdf45d35c61 |
   | dc2292e6-4056-46e0-8848-165d06fcc948 | None | 2017b481-706f-44e1-852a-57fb03ecef11 |
   | c7eadcea-e377-4392-9fc3-716f1bd57527 | None | 5f73c7d7-4826-49a5-b6be-0a95c6bdd2f8 |
   | da3a8d19-8a59-4e9d-923a-29254d688f6d | None | cfefaf60-8311-4bc3-9416-46852e2cb83f |
   | 807cb6ce-6b94-4cd1-9969-d390650854c7 | None | c07c13e6-a845-4791-9628-c8514585fe27 |
   | 0c245daa-7817-4ae9-a883-fed2e9c68d6c | None | 844c9a88-713a-4ff1-8737-30858d724593 |
   | e6499ef7-3db2-4ab4-bfa7-feb44c6591c6 | None | aef7c27a-f0b4-4814-b0ff-d3f792321212 |
   | 7545385c-bc49-4eb9-b13c-201368ce1c62 | None | c2e40164-c659-4849-a28f-a7b270ed2970 |
   +--------------------------------------+------+--------------------------------------+

4. Set the node into maintenance mode by using the baremetal node maintenance set command.

   Example

   $ openstack baremetal node maintenance set e6499ef7-3db2-4ab4-bfa7-ef59539bf972

5. Create a JSON file to add the new node to the node pool that contains RHOSP director.

   Example

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.207"
       }
     ]
   }

   For more information, see Adding nodes to the overcloud in the Director Installation and Usage guide.

6. Run the openstack overcloud node import command to register the new node.

   Example

   $ openstack overcloud node import newnode.json

7. After registering the new node, launch the introspection process by using the following commands:

   $ openstack baremetal node manage <node>
   $ openstack overcloud node introspect <node> --provide

8. Tag the new node with the Networker profile by using the openstack baremetal node set command.

   Example

   $ openstack baremetal node set --property \
       capabilities='profile:networker,boot_option:local' \
       91eb9ac5-7d52-453c-a017-c0e3d823efd0

9. Create a ~/templates/remove-networker.yaml environment file that defines the index of the node that you intend to remove:

   Example

   parameters:
   NetworkerRemovalPolicies:
      [{'resource_list': ['1']}]

10. Create a ~/templates/node-count-networker.yaml environment file and set the total count of Networker nodes in the file.

    Example

    parameter_defaults:
     OvercloudNetworkerFlavor: networker
     NetworkerCount: 3

11. Run the openstack overcloud deploy command and include the core heat templates, environment files, and the environment files that you modified.

    Important

    The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence.

    $ openstack overcloud deploy --templates \
    -e <your_environment_files> \
    -e /home/stack/templates/node-count-networker.yaml \
    -e /home/stack/templates/remove-networker.yaml

    RHOSP director removes the old Networker node, creates a new one, and updates the overcloud stack.

Verification

1. Check the status of the overcloud stack:

   $ openstack stack list --nested

2. Verify that the new Networker node is listed, and the old one is removed.

   $ openstack server list -c ID -c Name -c Status

   Sample output

   +--------------------------------------+------------------------+--------+
   | ID                                   | Name                   | Status |
   +--------------------------------------+------------------------+--------+
   | 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0    | ACTIVE |
   | 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1    | ACTIVE |
   | 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2    | ACTIVE |
   | a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 | ACTIVE |
   | cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 | ACTIVE |
   | 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 | ACTIVE |
   | 844c9a88-713a-4ff1-8737-6410bf551d4f | overcloud-networker-0  | ACTIVE |
   | c2e40164-c659-4849-a28f-507eb7edb79f | overcloud-networker-2  | ACTIVE |
   | 425a0828-b42f-43b0-940c-7fb02522753a | overcloud-networker-3  | ACTIVE |
   +--------------------------------------+------------------------+--------+

Procedure

1. Log in to your undercloud as the stack user.

2. Source the overcloud credentials file:

   Example

   $ source ~/overcloudrc

3. Verify that the RHOSP Networking service processes exist, and are marked out-of-service (xxx) for the overcloud-networker-1.

   $ openstack network agent list -c ID -c Binary -c Host -c Alive  | grep overcloud-networker-1

   Sample output for ML2/OVN

   +--------------------------------------+-----------------------+-------+-------------------------------+
   | ID                                   | Host                  | Alive | Binary                        |
   +--------------------------------------+-----------------------+-------+-------------------------------+
   | 26316f47-4a30-4baf-ba00-d33c9a9e0844 | overcloud-networker-1 | xxx   | ovn-controller                |
   +--------------------------------------+-----------------------+-------+-------------------------------+

   Sample output for ML2/OVS

   +--------------------------------------+-----------------------+-------+------------------------+
   | ID                                   | Host                  | Alive | Binary                 |
   +--------------------------------------+-----------------------+-------+------------------------+
   | 8377-66d75323e466c-b838-1149e10441ee | overcloud-networker-1 | xxx   | neutron-metadata-agent |
   | b55d-797668c336707-a2cf-cba875eeda21 | overcloud-networker-1 | xxx   | neutron-l3-agent       |
   | 9dcb-00a9e32ecde42-9458-01cfa9742862 | overcloud-networker-1 | xxx   | neutron-ovs-agent      |
   | be83-e4d9329846540-9ae6-1540947b2ffd | overcloud-networker-1 | xxx   | neutron-dhcp-agent     |
   +--------------------------------------+-----------------------+-------+------------------------+

4. Capture the UUIDs of the agents registered for overcloud-networker-1.

   $ AGENT_UUIDS=$(openstack network agent list -c ID -c Host -c Alive -c Binary -f value | grep overcloud-networker-1 | cut -d\  -f1)

5. Delete any remaining overcloud-networker-1 agents from the database.

   $ for agent in $AGENT_UUIDS; do neutron agent-delete $agent ; done

   Sample output

   Deleted agent(s): 26316f47-4a30-4baf-ba00-d33c9a9e0844