br-tun is missing after reboot

Solution In Progress - Updated -

Issue

  • After rebooting compute node br-tun bridge is missing and the following error is seen in ovs-vsctl show:
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: system
        Port qr-a64ea052-e8
            tag: 1
            Interface qr-a64ea052-e8
                type: internal
        Port qg-de731566-5c
            tag: 2
            Interface qg-de731566-5c
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
                error: "No usable peer 'patch-int' exists in 'system' datapath."
        Port br-int
            Interface br-int
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port qr-94e0b649-6e
            tag: 1
            Interface qr-94e0b649-6e
                type: internal
        Port ha-59828361-e7
            tag: 3
            Interface ha-59828361-e7
                type: internal
  • The following logs are seen in /var/log/containers/neutron/openvswitch-agent.log:
2020-12-08 12:25:01.172 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Switch connection timeout
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-tun
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 53, in _get_dp
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     dp = self._get_dp_by_dpid(self._cached_dpid)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 80, in _get_dp_by_dpid
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     raise RuntimeError(m)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: Switch connection timeout
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command 
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command 
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 38, in execute
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     self.run_idl(None)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 214, in run_idl
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     record = self.api.lookup(self.table, self.record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 107, in lookup
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     return self._lookup(table, record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 147, in _lookup
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 65, in row_by_value
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command     raise RowNotFound(table=table, col=column, match=match)
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Bridge with name=br-tun
2020-12-08 12:25:01.173 20795 ERROR ovsdbapp.backend.ovs_idl.command
[...]
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-561d7777-c49d-454e-8644-c6e805628000 - - - - -] Error while processing VIF ports: TypeError: int() can't convert non-string with expl
icit base
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 53, in _get_dp
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     dp = self._get_dp_by_dpid(self._cached_dpid)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ofswitch.py", line 80, in _get_dp_b
y_dpid
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: Switch connection timeout
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2636, in rpc_loop
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2085, in process_networ
k_ports
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed, re_added))
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1964, in treat_devices_
added_or_updated
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1838, in treat_vif_port
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     fixed_ips, device_owner, provisioning_needed)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1121, in port_bound
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     physical_network, segmentation_id)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 995, in provision_local_vlan
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ofports)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/br_tun.py", line 175, in install_flood_to_tun
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     (_dp, ofp, ofpp) = self._get_dp()
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_bridge.py", line 61, in _get_dp
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     new_dpid = int(self.get_datapath_id(), 16)
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TypeError: int() can't convert non-string with explicit base
2020-12-08 12:25:01.176 20795 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

Environment

  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content