ovs-vsctl errors in logs after after changing ovsdb_interface to vsctl
Issue
- To workaround our scale issue as reported in the cases with other cases, we adjusted the following settings in our OpenStack nodes.
-
Set ovsdb_timeout = 30 under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini as well as /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini.
-
Set of_inactivity_probe = 30 under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini
-
Set ovsdb_interface = vsctl under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini.
-
Then, restarted the neutron_ovs_agent neutron_metadata_agent neutron_l3_agent containers on all the controller and compute nodes.
-
Manually set inactivity_probe to 30000 on all the controller and compute nodes using the following command syntax.
ovs-vsctl set Manager $UUID inactivity_probe=30000, where $UUID is the value of the unique _uuid on every node.
- Now, we are seeing a lot of ovs-vsctl command errors in logs, like the following.
2020-06-17 16:50:13.972 402635 ERROR neutron.agent.ovsdb.impl_vsctl [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Unable to execute ['ovs-vsctl', '--timeout=30', '--oneline', '--format=json', '--', '--columns=other_config', 'list',
'Port', u'tap795fab1a-61']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "tap795fab1a-61" in table Port
: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "tap795fab1a-61" in table Port
2020-06-17 16:50:14.080 402635 ERROR neutron.agent.ovsdb.impl_vsctl [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Unable to execute ['ovs-vsctl', '--timeout=30', '--oneline', '--format=json']. Exception: Exit code: 1; Stdin: ; Stdo
ut: ; Stderr: ovs-vsctl: missing command name (use --help for help)
: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Error while processing VIF ports: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2210, in rpc_loop
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1762, in process_network_ports
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, provisioning_needed))
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1658, in treat_devices_added_or_updated
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ext_manager.handle_port(self.context, details)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l2/l2_agent_extensions_manager.py", line 42, in handle_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent extension.obj.handle_port(context, data)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 254, in handle_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._process_reset_port(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 304, in _process_reset_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.qos_driver.delete(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 96, in delete
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._handle_rule_delete(port, rule_type, ingress=True)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent handler(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 94, in delete_bandwidth_limit_ingress
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.br_int.delete_ingress_bw_limit_for_port(vif_port.port_name)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 914, in delete_ingress_bw_limit_for_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent txn.add(self.ovsdb.db_destroy('Queue', queue['_uuid']))
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.gen.next()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 102, in transaction
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent del self._nested_txns_map[cur_thread_id]
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 59, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.result = self.commit()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 56, in commit
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent res = self.run_vsctl(args)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 77, in run_vsctl
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ctxt.reraise = False
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 69, in run_vsctl
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent log_fail_as_error=False).rstrip()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent returncode=returncode)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.