Openstack compute scale-up fails when using predictable ips

Solution Verified - Updated -

Environment

  • Red Hat OpenStack Platform 13.0

Issue

  • After scaling down compute-nodes then scaling up compute-nodes when using predictable ips , scaling-up fails with following errors.
overcloud.Compute.#.NetworkDeployment:
  resource_type: OS::TripleO::SoftwareDeployment
  physical_resource_id: 332c4c83-a335-4d2e-a68c-98cc7ed7eab1
  status: CREATE_FAILED
  status_reason: |
    Error: resources.NetworkDeployment: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  • Os-colllect-config error on the compute node, like the following
os-collect-config: Traceback (most recent call last):
os-collect-config: File "/usr/bin/os-net-config", line 10, in <module>
os-collect-config: sys.exit(main())
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 254, in main
os-collect-config: obj = objects.object_from_json(iface_json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 45, in object_from_json
os-collect-config: return Vlan.from_json(json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 424, in from_json
os-collect-config: opts = _BaseOpts.base_opts_from_json(json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 335, in base_opts_from_json
os-collect-config: addresses.append(Address.from_json(address))
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 248, in from_json
os-collect-config: return Address(ip_netmask)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 239, in __init__
os-collect-config: ip_nw = netaddr.IPNetwork(self.ip_netmask)
os-collect-config: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 938, in __init__
os-collect-config: raise AddrFormatError('invalid IPNetwork %s' % addr)
os-collect-config: netaddr.core.AddrFormatError: invalid IPNetwork /25

Resolution

  • Replace the index of the down scaled node with deleted and move index
  • Example when compute 3 is removed
  ComputeIPs:
    tenant: 
    - 192.168.10.1       <-- index 0
    - 192.168.10.2       <-- index 1
    - 192.168.10.3       <-- index 2
    - 192.168.10.4       <-- index 3 compute 3
    - 192.168.10.5       <-- index 4
    internal_api: 
    - 192.168.20.1       <-- index 0
    - 192.168.20.2       <-- index 1
    - 192.168.20.3       <-- index 2
    - 192.168.20.4       <-- index 3 compute 3
    - 192.168.20.5       <-- index 4
    storage: 
    - 192.168.30.1       <-- index 0
    - 192.168.30.2       <-- index 1
    - 192.168.30.3       <-- index 2
    - 192.168.30.4       <-- index 3 compute 3
    - 192.168.30.5       <-- index 4
  • The compute 3 index needs to be replaced
  ComputeIPs:
    tenant: 
    - 192.168.10.1       <-- index 0
    - 192.168.10.2       <-- index 1
    - 192.168.10.3       <-- index 2
    - deleted            <-- index 3
    - 192.168.10.5       <-- index 4
    - 192.168.10.4       <-- index 5
    internal_api: 
    - 192.168.20.1       <-- index 0
    - 192.168.20.2       <-- index 1
    - 192.168.20.3       <-- index 2
    - deleted            <-- index 3
    - 192.168.20.5       <-- index 4
    - 192.168.20.4       <-- index 5
    storage: 
    - 192.168.30.1       <-- index 0
    - 192.168.30.2       <-- index 1
    - 192.168.30.3       <-- index 2
    - deleted            <-- index 3
    - 192.168.30.5       <-- index 4
    - 192.168.30.4       <-- index 5
  • If you are using profiles you will not need to do the next step
  • Now if your nodes are using scheduler hint the node name will also need to be changed using openstack baremetal node set
#compute-3 is being reindex to compute-5
openstack baremetal node show compute-3
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                                                                                                                                                                                                                                                     |
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
..............................
| instance_info          | {u'root_gb': u'57', u'display_name': u'compute-3', u'image_source': u'021d6430-59d1-4d91-8473-cb306fa8f1cc', u'capabilities': u'{"profile": "compute", "boot_option": "local"}', u'memory_mb': u'10240', u'vcpus': u'15', u'local_gb': u'59', u'configdrive': u'******', u'swap_mb': u'0', u'nova_host_id': u'undercloud-0.redhat.local'} |          
..............................                                                                                                                                                                                                                                                               
| properties             | {u'memory_mb': u'10240', u'cpu_arch': u'x86_64', u'local_gb': u'59', u'cpus': u'16', u'capabilities': u'profile:compute,boot_option:local,node:compute-3'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ..............................
| uuid                   | d5897d75-4f6b-4510-a438-43051519d64e                                                                                                                                                                                                                                                                                                      |
| vendor_interface       | ipmitool                                                                                                                                                                                                                                                                                                                                  |
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
openstack baremetal node set --property capabilities='node:compute-5,boot_option:local'

openstack baremetal node show compute-5
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                                                                                                                                                                                                                                                     |
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
..............................
| instance_info          | {u'root_gb': u'57', u'display_name': u'compute-5', u'image_source': u'021d6430-59d1-4d91-8473-cb306fa8f1cc', u'capabilities': u'{"profile": "compute", "boot_option": "local"}', u'memory_mb': u'10240', u'vcpus': u'15', u'local_gb': u'59', u'configdrive': u'******', u'swap_mb': u'0', u'nova_host_id': u'undercloud-0.redhat.local'} |          
..............................                                                                                                                                                                                                                                                               
| properties             | {u'memory_mb': u'10240', u'cpu_arch': u'x86_64', u'local_gb': u'59', u'cpus': u'16', u'capabilities': u'profile:compute,boot_option:local,node:compute-5'}                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ..............................
| uuid                   | d5897d75-4f6b-4510-a438-43051519d64e                                                                                                                                                                                                                                                                                                      |
| vendor_interface       | ipmitool                                                                                                                                                                                                                                                                                                                                  |
+------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Root Cause

  • When the node is removed from the overcloud the index for it's ip is blacklisted
  • So either a new index with a new ip is need
  • Or the old index needs to be replaced and the old ip moved to a new index

Diagnostic Steps

  • When configured with ips scaling up compute-nodes fails with following errors
overcloud.Compute.#.NetworkDeployment:
  resource_type: OS::TripleO::SoftwareDeployment
  physical_resource_id: 332c4c83-a335-4d2e-a68c-98cc7ed7eab1
  status: CREATE_FAILED
  status_reason: |
    Error: resources.NetworkDeployment: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  • Os-colllect-config error on the compute node due to no ip being available, like the following
os-collect-config: Traceback (most recent call last):
os-collect-config: File "/usr/bin/os-net-config", line 10, in <module>
os-collect-config: sys.exit(main())
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 254, in main
os-collect-config: obj = objects.object_from_json(iface_json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 45, in object_from_json
os-collect-config: return Vlan.from_json(json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 424, in from_json
os-collect-config: opts = _BaseOpts.base_opts_from_json(json)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 335, in base_opts_from_json
os-collect-config: addresses.append(Address.from_json(address))
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 248, in from_json
os-collect-config: return Address(ip_netmask)
os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/objects.py", line 239, in __init__
os-collect-config: ip_nw = netaddr.IPNetwork(self.ip_netmask)
os-collect-config: File "/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py", line 938, in __init__
os-collect-config: raise AddrFormatError('invalid IPNetwork %s' % addr)
os-collect-config: netaddr.core.AddrFormatError: invalid IPNetwork /25

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments