neutron-ovn-db-sync-util failing with Standard attribute ID not found for 151e9bcc-65e6-4235-8a4d-c3372118482e

Issue

• We're trying to synchronize OVN and Neutron databases using neutron-ovn-db-sync-util, exact syntax:

podman exec -it neutron_api neutron-ovn-db-sync-util --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --ovn-neutron_sync_mode repair --debug

• The above command fails with:

2023-05-29 16:12:58.236 11646 CRITICAL neutron_ovn_db_sync_util [req-c0b55238-b934-4c57-a107-1b06427c9abe - - - - -] Unhandled error: networking_ovn.common.exceptions.StandardAttributeIDNotFound: Standard attribute ID not found for 151e9bcc-65e6-4235-8a4d-c3372118482e
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/db/revision.py", line 47, in _get_standard_attr_id
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     id=resource_uuid).one()
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3282, in one
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     raise orm_exc.NoResultFound("No row was found for one()")
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util During handling of the above exception, another exception occurred:
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util Traceback (most recent call last):
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/bin/neutron-ovn-db-sync-util", line 10, in <module>
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     sys.exit(main())
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/cmd/neutron_ovn_db_sync_util.py", line 225, in main
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     synchronizer.do_sync()
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/ovn_db_sync.py", line 104, in do_sync
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     self.sync_networks_ports_and_dhcp_opts(ctx)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/ovn_db_sync.py", line 1003, in sync_networks_ports_and_dhcp_opts
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     self._create_port_in_ovn(ctx, port)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/ovn_db_sync.py", line 119, in _create_port_in_ovn
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     self._ovn_client.create_port(port)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 439, in create_port
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     db_rev.bump_revision(port, ovn_const.TYPE_PORTS)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     ectxt.value = e.inner_exc
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     self.force_reraise()
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     raise value
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/db/revision.py", line 123, in bump_revision
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     _ensure_revision_row_exist(session, resource, resource_type)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/db/revision.py", line 104, in _ensure_revision_row_exist
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     create_initial_revision(resource['id'], resource_type, session)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     ectxt.value = e.inner_exc
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     self.force_reraise()
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     six.reraise(self.type_, self.value, self.tb)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     raise value
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     return f(*args, **kwargs)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/db/revision.py", line 63, in create_initial_revision
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     session, resource_uuid, resource_type)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util   File "/usr/lib/python3.6/site-packages/networking_ovn/db/revision.py", line 51, in _get_standard_attr_id
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util     resource_uuid=resource_uuid)
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util networking_ovn.common.exceptions.StandardAttributeIDNotFound: Standard attribute ID not found for 151e9bcc-65e6-4235-8a4d-c3372118482e
2023-05-29 16:12:58.236 11646 ERROR neutron_ovn_db_sync_util

• Some of the instances failing to launch because ovn port type is set to virtual in OVN db. Based on our experience the issue is caused by Neutron/OVN databases not in sync