br-tun is missing after reboot

Solution In Progress - Updated -

Issue

  • After rebooting compute node br-tun bridge is missing and the following error is seen in ovs-vsctl show:
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: system
        Port qr-a64ea052-e8
            tag: 1
            Interface qr-a64ea052-e8
                type: internal
        Port qg-de731566-5c
            tag: 2
            Interface qg-de731566-5c
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
                error: "No usable peer 'patch-int' exists in 'system' datapath."
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port qr-94e0b649-6e
            tag: 1
            Interface qr-94e0b649-6e
                type: internal
        Port ha-59828361-e7
            tag: 3
            Interface ha-59828361-e7
                type: internal
  • The following logs are seen in /var/log/containers/neutron/openvswitch-agent.log:
2020-12-08 12:25:01.172 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Switch connection timeout
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-tun
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 53, in _get_dp
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     dp = self._get_dp_by_dpid(self._cached_dpid)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 80, in _get_dp_by_dpid
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     raise RuntimeError(m)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: Switch connection timeout
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command 
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command 
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 214, in run_idl
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 107, in lookup
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 147, in _lookup
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 65, in row_by_value
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-tun
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command
[...]
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Error while processing VIF ports: TypeError: int() can't convert non-string with expl
icit base
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 53, in _get_dp
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     dp = self._get_dp_by_dpid(self._cached_dpid)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 80, in _get_dp_b
y_dpid
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Switch connection timeout
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2636, in rpc_loop
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2085, in process_networ
k_ports
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed, re_added))
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1964, in treat_devices_
added_or_updated
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1838, in treat_vif_port
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     fixed_ips, device_owner, provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1121, in port_bound
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     physical_network, segmentation_id)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 995, in provision_local_vlan
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ofports)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_tun.py", line 175, in install_flood_to_tun
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     (_dp, ofp, ofpp) = self._get_dp()
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 61, in _get_dp
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     new_dpid = int(self.get_datapath_id(), 16)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TypeError: int() can't convert non-string with explicit base
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

Environment

  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In