None of VM reachable

Issue

• After creating a new VM in all tenant , none of the VMs are reachable through floating IP. We are unable to reach gateway from VM.

-Similar errors are seen in /var/log/containers/openvswitch/ovn-controller.log:

2022-05-12T07:09:02.322Z|00642|lflow|WARN|error parsing actions "ct_dnat_in_czone(192.168.1.218);": Syntax error at `ct_dnat_in_czone' expecting action.

• The following container images are being used:

[user@supportshell-1 ]$ cat sosreport-20220512-160746/*/sos_commands/podman/containers/podman_inspect_* |jq .[].Config.Labels.url -r | egrep 'ovn-controller|neutron'
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-neutron-server-ovn/images/16.2.1-7.1645706419
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ovn-controller/images/16.2.1-18.1645704470
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-neutron-server-ovn/images/16.2.1-7.1645706419
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ovn-controller/images/16.2.1-18.1645704470
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-neutron-server-ovn/images/16.2.1-7.1645706419
https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ovn-controller/images/16.2.1-18.1645704470

• We also see the following errors in /var/log/cotnainers/neutron/server.log:

controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     segment_index)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/managers.py", line 201, in _add_network_segment
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     context, network_id, segment, segment_index)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron/db/segments_db.py", line 57, in add_network_segment
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     segment=netseg_obj)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/registry.py", line 56, in notify
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     _get_callback_manager().notify(resource, event, trigger, **kwargs)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 108, in _wrapped
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     raise db_exc.RetryRequest(e)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     six.reraise(self.type_, self.value, self.tb)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     raise value
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/db/utils.py", line 103, in _wrapped
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     return function(*args, **kwargs)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation   File "/usr/lib/python3.6/site-packages/neutron_lib/callbacks/manager.py", line 177, in notify
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation     raise exceptions.CallbackFailure(errors=errors)
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 07:50:44.447 15 ERROR neutron.pecan_wsgi.hooks.translation neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.services.segments.db._add_segment_host_mapping_for_segment-8766057291021 failed with "'Chassis_Private' object has no attribute 'hostname'"

...


controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 08:49:05.211 16 ERROR networking_ovn.common.ovn_client [req-101129c1-ad1e-4205-afdb-767c4f6c0307 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Metadata port couldn't be found for network d60dc62b-3136-47d9-9386-77151da1bce4
controller-0/var/log/containers/neutron/server.log.7.gz:2022-05-09 08:49:25.496 22 ERROR networking_ovn.common.ovn_client [req-a3ff1406-67b3-41e7-b720-70ca0a8cab50 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Metadata port couldn't be found for network d60dc62b-3136-47d9-9386-77151da1bce4
controller-1/var/log/containers/neutron/server.log.7.gz:2022-05-09 08:50:10.154 20 ERROR networking_ovn.common.ovn_client [req-abfb732d-9271-4a9a-8ccf-9f8a20cf124f 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Metadata port couldn't be found for network d60dc62b-3136-47d9-9386-77151da1bce4


...

2022-05-11 07:19:26.868 16 INFO networking_ovn.db.revision [req-cbd0b865-e998-48b5-913e-c4fde812db6a 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Successfully bumped revision number for resource 2e8
c6e78-c792-4cef-bd10-9e82c75f724a (type: ports) to 1
2022-05-11 07:19:27.264 16 INFO networking_ovn.db.revision [req-cbd0b865-e998-48b5-913e-c4fde812db6a 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Successfully bumped revision number for resource 2e8
c6e78-c792-4cef-bd10-9e82c75f724a (type: router_ports) to 2
2022-05-11 07:19:27.528 16 ERROR ovsdbapp.backend.ovs_idl.transaction [req-cbd0b865-e998-48b5-913e-c4fde812db6a 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/commands.py", line 937, in run_idl
    resource_id=self.name, resource_type=self.resource_type)
networking_ovn.common.exceptions.RevisionConflict: OVN revision number for 2e8c6e78-c792-4cef-bd10-9e82c75f724a (type: ports) is equal or higher than the given resource. Skipping update

2022-05-11 07:19:27.529 16 INFO networking_ovn.ovsdb.impl_idl_ovn [req-cbd0b865-e998-48b5-913e-c4fde812db6a 8139e4eefe744a59bce61f2819be6122 7af45bd4b03c4c99b96aa9ec55aef3be - default default] Transaction aborted. Reason: OVN revision number for 2e8c6e78-c792-4cef-bd10-9e82c75f724a (type: ports) is equal or higher than the given resource. Skipping update