Regular members of projects lost capability to create ports on some external networks after update to 16.2.6

Issue

• We started to receive reports from customers about the loss of their ability to see some subnets tied with external networks.

• Example of openstack subnet list ran as admin:

openstack subnet list | grep cci
| 02a8825d-e5f7-4e91-b502-fc8361051e44 | provider_net_subnet_cci_3                                   | eb3e8289-ce41-4825-a48a-8f8e11feaec7 | 10.10.10.0/22            |
| 0bd62a3f-d665-4e93-bf80-94e493d63541 | internal_subnet_cci_6                                       | a803f0d8-ab65-41bd-b720-c7a1f4c62459 | 10.10.11.0/24           |
| 10a8b6b3-7ff5-4933-9e31-9be0f25d745e | provider_net_subnet_cci_4                                   | 68a8220a-20f4-4940-99b4-45b6f98bce6b | 10.10.12.0/22            |
| 11b95215-522d-4730-97d5-a76bdc66d6fa | provider_net_subnet_cci_2                                   | 74e8faa7-87ba-41b2-a000-438013194814 | 10.10.13.0/22            |
| 1447a1b3-c28f-4026-9edb-98af355c29c9 | provider_net_subnet_cci_10                                  | 316eeb47-1498-46b4-b39e-00ddf73bd2a5 | 10.10.14.0/22            |
| 1a14746d-8e7d-4dbe-a361-dfcc01b0bc5c | provider_net_subnet_cci_1                                   | d284bcff-d1ed-452d-b7e3-af979b9582a3 | 10.10.15.0/22            |
| 1cd47cb5-bdb7-4745-bd21-55925a3f0174 | provider_net_subnet_cci_11                                  | 9a44e25b-dafd-4781-a404-6756bb5674c8 | 10.10.14.0/22            |
| 3abbd7bc-6027-49de-ba44-96e4a6268d45 | provider_net_subnet_cci_6                                   | ee7dcdfe-2b6e-4b7e-bbe9-3dabc0972bb5 | 10.10.16.0/22            |
| 3efe14a9-3d70-47a1-a7f8-5d373539c399 | provider_net_subnet_cci_ipv6_1                              | d284bcff-d1ed-452d-b7e3-af979b9582a3 | 10.10.17.0/22        |
| 456329df-36f5-452a-bae2-404003910f09 | provider_net_subnet_cci_8                                   | 60cacaff-86a6-4f88-82a4-ed3023724df1 | 10.10.17.0/22            |
| 46c0f9b7-0028-4780-97c9-25b2e93f05d7 | provider_net_subnet_cci_9                                   | d655dcd0-b593-439c-997b-aa5bc8c03a3a | 10.10.18.0/22            |
| 47f321df-ad49-4524-9489-435892554b2c | kernel-perfqe-private-subnet-cci4                           | 52b155fe-1d0c-4d82-afc4-8e911491aa43 | 10.10.19.0/24            |
| 6159e87c-06a1-4f56-aa5a-aabad1298be5 | provider_net_subnet_cci_5                                   | 25ec4907-36fc-4035-b8d5-b797246330f2 | 10.10.20.0/22            |
| 62a381e5-9313-43fa-a515-cd0d7560907b | provider_net_subnet_cci_ipv6_3                              | eb3e8289-ce41-4825-a48a-8f8e11feaec7 | 10.10.21.0/22        |
| 63b2d4a6-6df2-417c-8ee8-d0e01bc523c8 | provider_net_subnet_cci_ipv6_2                              | 74e8faa7-87ba-41b2-a000-438013194814 | 10.10.22.0/22        |
| 6ff94455-44a4-4e7f-be82-c34d6c7a690c | mbooth-provider_net_cci_5-subnet                            | 32c7a8a2-edc5-48a8-b8d1-8396f63a85ef | 10.10.23.0/24         |
| af342799-3d03-4b51-b252-f56bed4e0997 | provider_net_subnet_cci_11                                  | b71d614c-b0b0-4f2d-b141-e78129212b98 | 10.10.24.0/22            |
| b360d82a-1375-4549-a665-1f505aae2663 | provider_net_subnet_cci_14                                  | eceac180-5a4d-4b1d-b916-1d4e8f19b873 | 10.10.25.0/22            |
| d3b1c702-bb71-4547-8cf0-2ff5f9802595 | provider_net_subnet_cci_13                                  | 0e212597-e475-4c4a-a4fa-db71f84ec04c | 10.10.26.0/22            |
| d8b5ca98-fa60-4ec7-abdb-ae6d22989536 | provider_net_cci_2_jira_subnet                              | 513e98a5-f712-4034-98b2-6a14073e267d | 10.10.27.0/24           |
| e9fa371f-2b3e-4a4f-a33b-9e39f869aea3 | provider_net_subnet_cci_12                                  | 36e46f70-99ff-48f5-aa9d-7bbd22b6218a | 10.10.28.0/22            |
| eb8db9f4-a76f-4fe2-a0bd-f932bc20dfa1 | provider_net_subnet_cci_7                                   | 5058fef2-f89f-4e70-9e01-66af2847ddc4 | 10.10.29.0/22             |

• If we run the same as regular user:

| 11b95215-522d-4730-97d5-a76bdc66d6fa | provider_net_subnet_cci_2           | 74e8faa7-87ba-41b2-a000-438013194814 | 10.10.13.0/22     |
| 1447a1b3-c28f-4026-9edb-98af355c29c9 | provider_net_subnet_cci_10          | 316eeb47-1498-46b4-b39e-00ddf73bd2a5 | 10.10.14.0/22     |
| 1a14746d-8e7d-4dbe-a361-dfcc01b0bc5c | provider_net_subnet_cci_1           | d284bcff-d1ed-452d-b7e3-af979b9582a3 | 10.10.15.0/22     |
| 3abbd7bc-6027-49de-ba44-96e4a6268d45 | provider_net_subnet_cci_6           | ee7dcdfe-2b6e-4b7e-bbe9-3dabc0972bb5 | 10.10.16.0/22     |
| 3efe14a9-3d70-47a1-a7f8-5d373539c399 | provider_net_subnet_cci_ipv6_1      | d284bcff-d1ed-452d-b7e3-af979b9582a3 | 10.10.17.0/22 |
| 46c0f9b7-0028-4780-97c9-25b2e93f05d7 | provider_net_subnet_cci_9           | d655dcd0-b593-439c-997b-aa5bc8c03a3a | 10.10.18.0/22     |
| 6159e87c-06a1-4f56-aa5a-aabad1298be5 | provider_net_subnet_cci_5           | 25ec4907-36fc-4035-b8d5-b797246330f2 | 10.10.20.0/22     |
| 63b2d4a6-6df2-417c-8ee8-d0e01bc523c8 | provider_net_subnet_cci_ipv6_2      | 74e8faa7-87ba-41b2-a000-438013194814 | 10.10.22.0/22 |
| af342799-3d03-4b51-b252-f56bed4e0997 | provider_net_subnet_cci_11          | b71d614c-b0b0-4f2d-b141-e78129212b98 | 10.10.24.0/22     |
| b360d82a-1375-4549-a665-1f505aae2663 | provider_net_subnet_cci_14          | eceac180-5a4d-4b1d-b916-1d4e8f19b873 | 10.10.25.0/22     |
| e9fa371f-2b3e-4a4f-a33b-9e39f869aea3 | provider_net_subnet_cci_12          | 36e46f70-99ff-48f5-aa9d-7bbd22b6218a | 10.10.28.0/22     |

• If we focus on just two subnets. Let's say provider_net_subnet_cci_7 and provider_net_subnet_cci_12 and check them:

openstack subnet show eb8db9f4-a76f-4fe2-a0bd-f932bc20dfa1
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                   |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 10.10.29.10-10.10.32.251                                                                                                                                  |
| cidr              | 10.10.29.0/22                                                                                                                                            |
| created_at        | 2019-08-19T07:49:01Z                                                                                                                                    |
| description       |                                                                                                                                                         |
| dns_nameservers   | 10.10.10.30                                                                                                                                |
| enable_dhcp       | True                                                                                                                                                    |
| gateway_ip        | 10.10.32.251                                                                                                                                             |
| host_routes       |                                                                                                                                                         |
| id                | eb8db9f4-a76f-4fe2-a0bd-f932bc20dfa1                                                                                                                    |
| ip_version        | 4                                                                                                                                                       |
| ipv6_address_mode | None                                                                                                                                                    |
| ipv6_ra_mode      | None                                                                                                                                                    |
| location          | cloud='', project.domain_id=, project.domain_name='Default', project.id='fe0579143bba47ae8501b5c6c7e80804', project.name='admin', region_name='', zone= |
| name              | provider_net_subnet_cci_7                                                                                                                               |
| network_id        | 5058fef2-f89f-4e70-9e01-66af2847ddc4                                                                                                                    |
| prefix_length     | None                                                                                                                                                    |
| project_id        | fe0579143bba47ae8501b5c6c7e80804                                                                                                                        |
| revision_number   | 11                                                                                                                                                      |
| segment_id        | None                                                                                                                                                    |
| service_types     |                                                                                                                                                         |
| subnetpool_id     | None                                                                                                                                                    |
| tags              |                                                                                                                                                         |
| updated_at        | 2022-10-18T10:40:32Z                                                                                                                                    |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+

openstack subnet show e9fa371f-2b3e-4a4f-a33b-9e39f869aea3
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field             | Value                                                                                                                                                   |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools  | 10.10.28.10-10.10.31.251                                                                                                                                |
| cidr              | 10.10.28.0/22                                                                                                                                           |
| created_at        | 2021-01-28T16:47:46Z                                                                                                                                    |
| description       | provider_net_subnet_cci_12                                                                                                                              |
| dns_nameservers   | 10.10.10.30                                                                                                                                |
| enable_dhcp       | True                                                                                                                                                    |
| gateway_ip        | 10.10.31.254                                                                                                                                            |
| host_routes       |                                                                                                                                                         |
| id                | e9fa371f-2b3e-4a4f-a33b-9e39f869aea3                                                                                                                    |
| ip_version        | 4                                                                                                                                                       |
| ipv6_address_mode | None                                                                                                                                                    |
| ipv6_ra_mode      | None                                                                                                                                                    |
| location          | cloud='', project.domain_id=, project.domain_name='Default', project.id='fe0579143bba47ae8501b5c6c7e80804', project.name='admin', region_name='', zone= |
| name              | provider_net_subnet_cci_12                                                                                                                              |
| network_id        | 36e46f70-99ff-48f5-aa9d-7bbd22b6218a                                                                                                                    |
| prefix_length     | None                                                                                                                                                    |
| project_id        | fe0579143bba47ae8501b5c6c7e80804                                                                                                                        |
| revision_number   | 5                                                                                                                                                       |
| segment_id        | None                                                                                                                                                    |
| service_types     |                                                                                                                                                         |
| subnetpool_id     | None                                                                                                                                                    |
| tags              |                                                                                                                                                         |
| updated_at        | 2022-10-18T10:39:38Z                                                                                                                                    |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+

• There's no obvious difference between those two and yet regular users cannot see provider_net_subnet_cci_7. The implication of the fact that they cannot see provider_net_subnet_cci_7 is that they cannot create ports on network 5058fef2-f89f-4e70-9e01-66af2847ddc4

• The related log from neutron server when attempting to create a port is:

[root@overcloud-controller-0 ~]# grep req-35594b55-cf0d-4895-bed0-66d02123990a /var/log/containers/neutron/server.log
2023-11-20 13:26:01.179 32 INFO neutron.pecan_wsgi.hooks.translation [req-35594b55-cf0d-4895-bed0-66d02123990a ad35ecf7666ba6a0b9baa5fef0421ea94258a7c3d28bf164a1f48eb914f6d213 38b5b8cd675d44d298cf6f671795b136 - 62cf1b5ec006489db99e2b0ebfb55f57 62cf1b5ec006489db99e2b0ebfb55f57] POST failed (client error): Tenant 38b5b8cd675d44d298cf6f671795b136 not allowed to create port on this network
2023-11-20 13:26:01.181 32 INFO neutron.wsgi [req-35594b55-cf0d-4895-bed0-66d02123990a ad35ecf7666ba6a0b9baa5fef0421ea94258a7c3d28bf164a1f48eb914f6d213 38b5b8cd675d44d298cf6f671795b136 - 62cf1b5ec006489db99e2b0ebfb55f57 62cf1b5ec006489db99e2b0ebfb55f57] 10.10.36.133,10.10.21.60 "POST /v2.0/ports HTTP/1.1" status: 403  len: 336 time: 0.1407633

• We did not alter any security policies during the update