Error: Failed to launch instance "rhel7-151102": Please try again later [Error: Unavailable console type rdp-html5.].
Environment
- Red Hat OpenStack 6.0
Issue
- Depoyed OpenStack with Provider network with OpenvSwitch configuration ( see Scenario: Provider networks with Open vSwitch ).
- Intended to have 1 external (provider) network with no VLAN , and another external network with VLAN for tenant.
- After creating those 2 external neutron network and run instance with non-admin tenant on them, failed with
Error: Failed to launch instance "rhel7-151102": Please try again later [Error: Unavailable console type rdp-html5.]error.
Resolution
- This caused by creating neutron network without
--router:external=True( and--shared) option. -–router:external=Trueis required for neutron to create external network with Floating IP address pool.- If this properly is set to
False, Instances launched by non-Admin tenants will not be able to attach external network with Floating IP address.
Here's example.
Creating external network
- Create neutron network as external network ( --router:external=True )
[root@osp6 ~(keystone_admin)]# ip netns
[root@osp6 ~(keystone_admin)]# neutron net-list
[root@osp6 ~(keystone_admin)]# keystone tenant-list
+----------------------------------+----------+---------+
| id | name | enabled |
+----------------------------------+----------+---------+
| 2d4c3f0e3bc3496987e9440de7f6d799 | admin | True |
| 7793dcc0dc054f6693eb6f4173fe04df | project1 | True |
| 8dc587f29de344a9ab9eb23cbb0cb8d5 | services | True |
+----------------------------------+----------+---------+
[root@osp6 ~(keystone_admin)]# neutron net-create ext_net --router:external=True --tenant-id 7793dcc0dc054f6693eb6f4173fe04df --shared --provider:network_type flat --provider:physical_network physnet1
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 4f41b008-8f4c-41bb-b02f-dde337ea1c33 |
| name | ext_net |
| provider:network_type | flat |
| provider:physical_network | physnet1 |
| provider:segmentation_id | |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+---------------------------+--------------------------------------+
- Add subnet
[root@osp6 ~(keystone_admin)]# neutron subnet-create ext_net --allocation-pool start=10.0.3.201,end=10.0.3.210 --gateway 10.0.3.253 10.0.3.0/24 --enable-dhcp --name ext_subnet1 --tenant-id 7793dcc0dc054f6693eb6f4173fe04df
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.0.3.201", "end": "10.0.3.210"} |
| cidr | 10.0.3.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.3.253 |
| host_routes | |
| id | d9e4ba34-f024-4be5-a6fc-b8c1de054163 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | ext_subnet1 |
| network_id | 4f41b008-8f4c-41bb-b02f-dde337ea1c33 |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+-------------------+----------------------------------------------+
- Check if there is external network
[root@osp6 ~(keystone_admin)]# neutron net-external-list
+--------------------------------------+---------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+--------------------------------------------------+
| 4f41b008-8f4c-41bb-b02f-dde337ea1c33 | ext_net | d9e4ba34-f024-4be5-a6fc-b8c1de054163 10.0.3.0/24 |
+--------------------------------------+---------+--------------------------------------------------+
Update existing neutron network as external network
- Create Non-external Network
[root@osp6 ~(keystone_admin)]# neutron net-list
[root@osp6 ~(keystone_admin)]# neutron net-create ext_net --tenant-id 7793dcc0dc054f6693eb6f4173fe04df --provider:network_type flat --provider:physical_network physnet1l
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 8a50be86-3631-47f8-9f76-260a2f1e07c3 |
| name | ext_net |
| provider:network_type | flat |
| provider:physical_network | physnet1l |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+---------------------------+--------------------------------------+
- Add subnet
[root@osp6 ~(keystone_admin)]# neutron subnet-create ext_net --allocation-pool start=10.0.3.201,end=10.0.3.210 --gateway 10.0.3.253 10.0.3.0/24 --enable-dhcp --name ext_subnet1 --tenant-id 7793dcc0dc054f6693eb6f4173fe04df
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.0.3.201", "end": "10.0.3.210"} |
| cidr | 10.0.3.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.3.253 |
| host_routes | |
| id | 6db5876c-6dae-4049-9f79-f94fc4b43676 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | ext_subnet1 |
| network_id | 8a50be86-3631-47f8-9f76-260a2f1e07c3 |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+-------------------+----------------------------------------------+
- Update network with
--sharedand--router:external
[root@osp6 ~(keystone_admin)]# neutron net-update ext_net --shared --router:external
Updated network: ext_net
[root@osp6 ~(keystone_admin)]# neutron net-show ext_net
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 8a50be86-3631-47f8-9f76-260a2f1e07c3 |
| name | ext_net |
| provider:network_type | flat |
| provider:physical_network | physnet1l |
| provider:segmentation_id | |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | 6db5876c-6dae-4049-9f79-f94fc4b43676 |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+---------------------------+--------------------------------------+
- Check if set to external network
[root@osp6 ~(keystone_admin)]# neutron net-external-list
+--------------------------------------+---------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+--------------------------------------------------+
| 8a50be86-3631-47f8-9f76-260a2f1e07c3 | ext_net | 6db5876c-6dae-4049-9f79-f94fc4b43676 10.0.3.0/24 |
+--------------------------------------+---------+--------------------------------------------------+
Root Cause
- Created those 2 neutron networks without
--router:external=True( and--shared) option. - Without them , neutron doesn't treat them as external network , and instance launched by non-admin tenants can not bind to those network.
Here's example;
1) Create external network with admin tenant and no --shared.
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# . ~/keystonerc_admin ; keystone tenant-list
<...>
+----------------------------------+----------+---------+
| id | name | enabled |
+----------------------------------+----------+---------+
| 05c275ecf6a64db6810a3d0c1fc18e89 | admin | True | <==== Admin
| 3c833f0deb1540ceb60e94f18eb60f3f | project1 | True |
| cb6e3ceb53e64274bbe52f58d999f8e3 | services | True |
+----------------------------------+----------+---------+
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# neutron net-show ext_net
+-----------------+--------------------------------------+
| Field | Value |
+-----------------+--------------------------------------+
| admin_state_up | True |
| id | 7d08caf9-cb9a-41b0-8fb7-438c10f78f28 |
| mtu | 0 |
| name | ext_net |
| router:external | True |
| shared | False | <==== Not shared.
| status | ACTIVE |
| subnets | 810c4a15-cef2-4c33-9002-1adeed31f5e5 |
| tenant_id | 05c275ecf6a64db6810a3d0c1fc18e89 | <==== Admin
+-----------------+--------------------------------------+
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# neutron net-list --shared
(NO OUTPUT)
- From admin user in other project (project1)
[root@el72-osp7-all-br1 ~(keystone_admin)]# . ~/keystonerc_admin;keystone user-role-list --user project1-admin --tenant project1
<...>
+----------------------------------+-------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+-------+----------------------------------+----------------------------------+
| f440ff150a564b1b8212ee6c94123a26 | admin | 0760bb700a6146ef8138c691782d2193 | 3c833f0deb1540ceb60e94f18eb60f3f |
+----------------------------------+-------+----------------------------------+----------------------------------+
[root@el72-osp7-all-br1 ~(keystone_admin)]# . ~/keystonerc_project1-admin ; neutron net-list
+--------------------------------------+---------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+--------------------------------------------------+
| 7d08caf9-cb9a-41b0-8fb7-438c10f78f28 | ext_net | 810c4a15-cef2-4c33-9002-1adeed31f5e5 10.0.3.0/24 | <=====
| 6ca18635-66e2-4ca4-a2f6-6c2382dc994d | net1 | c315e8e6-9e0b-43b7-a43d-b23fd6570853 10.5.5.0/24 |
+--------------------------------------+---------+--------------------------------------------------+
[root@el72-osp7-all-br1 ~(keystone_admin)]# . ~/keystonerc_admin;keystone user-role-list --user project1-user --tenant project1
<...>
+----------------------------------+----------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+----------+----------------------------------+----------------------------------+
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | 18c63d36e035435e8926e8b529df5dea | 3c833f0deb1540ceb60e94f18eb60f3f |
+----------------------------------+----------+----------------------------------+----------------------------------+
[root@el72-osp7-all-br1 ~(keystone_project1-admin)]# . ~/keystonerc_project1-user ; neutron net-list
+--------------------------------------+---------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+--------------------------------------------------+
| 7d08caf9-cb9a-41b0-8fb7-438c10f78f28 | ext_net | 810c4a15-cef2-4c33-9002-1adeed31f5e5 | <======
| 6ca18635-66e2-4ca4-a2f6-6c2382dc994d | net1 | c315e8e6-9e0b-43b7-a43d-b23fd6570853 10.5.5.0/24 |
+--------------------------------------+---------+--------------------------------------------------+
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# . ~/keystonerc_project1-user ; neutron net-list
2) After setting it to '--shared'
[root@el72-osp7-all-br1 ~(keystone_admin)]# neutron net-update ext_net --shared
Updated network: ext_net
[root@el72-osp7-all-br1 ~(keystone_admin)]# . ~/keystonerc_project1-user ; neutron net-list
+--------------------------------------+---------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+--------------------------------------------------+
| 7d08caf9-cb9a-41b0-8fb7-438c10f78f28 | ext_net | 810c4a15-cef2-4c33-9002-1adeed31f5e5 10.0.3.0/24 | <=====
| 6ca18635-66e2-4ca4-a2f6-6c2382dc994d | net1 | c315e8e6-9e0b-43b7-a43d-b23fd6570853 10.5.5.0/24 |
+--------------------------------------+---------+--------------------------------------------------+
3) When creating router by admin tenant,
[root@el72-osp7-all-br1 ~(keystone_project1-admin)]# neutron router-show e689e737-06dd-4d0b-9cdc-ab9d8feca37e
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| distributed | False |
| external_gateway_info | {"network_id": "34ab9dc8-ee4d-4b6c-8a72-c7d3d4da094a", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "7b816966-9ff7-4793-bb1c-a393c6e696c8", "ip_address": "10.0.3.202"}]} |
| ha | False |
| id | e689e737-06dd-4d0b-9cdc-ab9d8feca37e |
| name | router1 |
| routes | |
| status | ACTIVE |
| tenant_id | 05c275ecf6a64db6810a3d0c1fc18e89 |
+-----------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
- From admin user in other project (project1)
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# . ~/keystonerc_project1-admin ; neutron router-list
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
| e689e737-06dd-4d0b-9cdc-ab9d8feca37e | router1 | {"network_id": "34ab9dc8-ee4d-4b6c-8a72-c7d3d4da094a", "enable_snat": true, "external_fixed_ips": [{"subnet_id": "7b816966-9ff7-4793-bb1c-a393c6e696c8", "ip_address": "10.0.3.202"}]} | False | False |
+--------------------------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------+-------+
- From member user in other project (project1)
[root@el72-osp7-all-br1 ~(keystone_project1-user)]# . ~/keystonerc_project1-user ; neutron router-list
(NO OUTPUT)
--shared will make it available for member user for other project to use it.
We can also explain this behavior from aspect of policy as follows:
- /etc/neutron/policy.json
<...>
"get_network": "rule:admin_or_owner or rule:shared or rule:external or rule:context_is_advsvc",
"get_network:router:external": "rule:regular_user",
<...>
rule:admin_or_owner meant '"admin_or_owner": "rule:context_is_admin or tenant_id:%(tenant_id)s",'
When tenant_id is mine or rule:context_is_admin.
"context_is_admin": "role:admin",
and rule:share is
"shared": "field:networks:shared=True",
and also rule:external is
"external": "field:networks:router:external=True",
So we can see external network itself regardless of --shared property.
Diagnostic Steps
Create Non-external network
- Create neutron network as non-external network ( Without --router:external=True )
[root@osp6 ~(keystone_admin)]# ip netns
[root@osp6 ~(keystone_admin)]# neutron net-list
[root@osp6 ~(keystone_admin)]# neutron net-create ext_net --tenant-id 7793dcc0dc054f6693eb6f4173fe04df --provider:network_type flat --provider:physical_network physnet1l
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 12cf0d9d-427b-435d-977e-9d6cf12fe754 |
| name | ext_net |
| provider:network_type | flat |
| provider:physical_network | physnet1l |
| provider:segmentation_id | |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+---------------------------+--------------------------------------+
- Add subnet
[root@osp6 ~(keystone_admin)]# neutron subnet-create ext_net --allocation-pool start=10.0.3.201,end=10.0.3.210 --gateway 10.0.3.253 10.0.3.0/24 --enable-dhcp --name ext_subnet1 --tenant-id 7793dcc0dc054f6693eb6f4173fe04df
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.0.3.201", "end": "10.0.3.210"} |
| cidr | 10.0.3.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.3.253 |
| host_routes | |
| id | 5a3d8618-236a-4622-a1d5-7abe71d73e0a |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | ext_subnet1 |
| network_id | 12cf0d9d-427b-435d-977e-9d6cf12fe754 |
| tenant_id | 7793dcc0dc054f6693eb6f4173fe04df |
+-------------------+----------------------------------------------+
- Check if external network exists
[root@osp6 ~(keystone_admin)]# neutron net-external-list
[root@osp6 ~(keystone_admin)]#
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments