ovs-vsctl errors in logs after after changing ovsdb_interface to vsctl

Solution In Progress - Updated -

Issue

  • To workaround our scale issue as reported in the cases with other cases, we adjusted the following settings in our OpenStack nodes.
  1. Set ovsdb_timeout = 30 under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini as well as /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini.

  2. Set of_inactivity_probe = 30 under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini

  3. Set ovsdb_interface = vsctl under ovs section of /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini.

  4. Then, restarted the neutron_ovs_agent neutron_metadata_agent neutron_l3_agent containers on all the controller and compute nodes.

  5. Manually set inactivity_probe to 30000 on all the controller and compute nodes using the following command syntax.
    ovs-vsctl set Manager $UUID inactivity_probe=30000, where $UUID is the value of the unique _uuid on every node.

  • Now, we are seeing a lot of ovs-vsctl command errors in logs, like the following.
2020-06-17 16:50:13.972 402635 ERROR neutron.agent.ovsdb.impl_vsctl [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Unable to execute ['ovs-vsctl', '--timeout=30', '--oneline', '--format=json', '--', '--columns=other_config', 'list',
 'Port', u'tap795fab1a-61']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "tap795fab1a-61" in table Port
: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "tap795fab1a-61" in table Port
2020-06-17 16:50:14.080 402635 ERROR neutron.agent.ovsdb.impl_vsctl [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Unable to execute ['ovs-vsctl', '--timeout=30', '--oneline', '--format=json']. Exception: Exit code: 1; Stdin: ; Stdo
ut: ; Stderr: ovs-vsctl: missing command name (use --help for help)
: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3755b0d9-735f-4ca6-8aa0-5dff5690c126 - - - - -] Error while processing VIF ports: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2210, in rpc_loop
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, provisioning_needed)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1762, in process_network_ports
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     devices_added_updated, provisioning_needed))
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1658, in treat_devices_added_or_updated
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.ext_manager.handle_port(self.context, details)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l2/l2_agent_extensions_manager.py", line 42, in handle_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     extension.obj.handle_port(context, data)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 254, in handle_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._process_reset_port(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 304, in _process_reset_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.qos_driver.delete(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 96, in delete
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self._handle_rule_delete(port, rule_type, ingress=True)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     handler(port)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 94, in delete_bandwidth_limit_ingress
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.br_int.delete_ingress_bw_limit_for_port(vif_port.port_name)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/common/ovs_lib.py", line 914, in delete_ingress_bw_limit_for_port
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     txn.add(self.ovsdb.db_destroy('Queue', queue['_uuid']))
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 102, in transaction
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     del self._nested_txns_map[cur_thread_id]
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/ovsdbapp/api.py", line 59, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.result = self.commit()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 56, in commit
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     res = self.run_vsctl(args)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 77, in run_vsctl
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ctxt.reraise = False
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 69, in run_vsctl
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     log_fail_as_error=False).rstrip()
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     returncode=returncode)
2020-06-17 16:50:14.085 402635 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: missing command name (use --help for help)

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content