Unable to create instances / Networking issues with instance on Openstack

Issue

After doing a FFU 13 to 16, you are unable to create instances.
In dhcp-agent.log or l3-agent.log you see error messages like this:

2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 1c696dec-4f06-4e8c-8389-a23818454636.: multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/managers.py", line 283, in serve_client
    send(msg)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 128, in send
    s = self.dumps(obj)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 170, in dumps
    return json.dumps(obj, cls=RpcJSONEncoder).encode('utf-8')
  File "/usr/lib64/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.6/site-packages/oslo_rootwrap/jsonrpc.py", line 43, in default
    return super(RpcJSONEncoder, self).default(o)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ValueError' is not JSON serializable

---------------------------------------------------------------------------
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 209, in call_driver
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 235, in enable
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent     common_utils.wait_until_true(self._enable, timeout=300)
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 711, in wait_until_true
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent     while not predicate():
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 243, in _enable
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent     self.restart()
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 163, in restart
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent     self.disable(retain_port=True, block=True)
2022-03-24 22:00:17.348 782789 ERROR neutron.agent.dhcp.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/dhcp.py", line 268, in disable