neutron-openvswitch-agent fails to start with `KeyError: 'ofctl_service'` in python-ryu

Issue

neutron-openvswitch-agent start fails with:

2017-07-24 09:48:10.730 5920 ERROR oslo.messaging._drivers.impl_rabbit [req-b7d3e40a-2445-4d5d-a596-26cd4177c09b - - - - -] Failed to declare consumer for topic 'q-agent-notifier-port-update': (0, 0): (541) INTERNAL_ERROR
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-b7d3e40a-2445-4d5d-a596-26cd4177c09b - - - - -] Agent main thread died of an exception
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 42, in agent_main_wrapper
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     ovs_agent.main(bridge_classes)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2161, in main
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 294, in __init__
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.connection.consume_in_threads()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 294, in consume_in_threads
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     server.start()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 268, in wrapper
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     log_after, timeout_timer)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 188, in run_once
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     post_fn = fn()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in <lambda>
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     states[state].run_once(lambda: fn(self, *args, **kwargs),
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 420, in start
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.listener = self._create_listener()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return self.transport._listen(self._target, 1, None)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 112, in _listen
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     batch_timeout)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 482, in listen
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     conn.declare_fanout_consumer(target.topic, listener)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1152, in declare_fanout_consumer
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.declare_consumer(consumer)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1040, in declare_consumer
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     error_callback=_connect_error)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 814, in ensure
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     ret, channel = autoretry_method()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return fun(*args, **kwargs)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 508, in __call__
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return fun(*args, channel=channels[0], **kwargs), channels[0]
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 790, in execute_method
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     method()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1028, in _declare_consumer
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     consumer.declare(self)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 302, in declare
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.queue.declare()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 522, in declare
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.queue_declare(nowait, passive=False)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 548, in queue_declare
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     nowait=nowait)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/channel.py", line 1258, in queue_declare
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     (50, 11),  # Channel.queue_declare_ok
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.channel_id, allowed_methods)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     self.wait()
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return self.dispatch_method(method_sig, args, content)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     return amqp_method(self, args)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp     (class_id, method_id), ConnectionError)
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp InternalError: (0, 0): (541) INTERNAL_ERROR
2017-07-24 09:48:10.731 5920 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 
2017-07-24 09:48:10.743 5920 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 545, in close
    self.uninstantiate(app_name)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 528, in uninstantiate
    app = self.applications.pop(name)
KeyError: 'ofctl_service'