Issue

• When creating an instance, it fails and we get the following error in /var/log/neutron/server.log

 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'62c75d83-66cb-476f-b08a-7edc87fca5fb', 'name': 'network-vif-plugged', 'server_uuid': u'1f141783-6775-46f0-aff0-b8a3680d81f9'}]

/neutron/server.log

2015-09-22 10:36:44.653 17715 INFO neutron.callbacks.manager [req-59ea4621-dd62-47c0-8b2b-b85ae6d0cff5 ] Calling callback neutron.db.l3_dvrscheduler_db._notify_l3_agent_new_port
2015-09-22 10:36:44.653 17715 DEBUG neutron.db.l3_dvrscheduler_db [req-59ea4621-dd62-47c0-8b2b-b85ae6d0cff5 ] Received port after_update _notify_l3_agent_new_port /usr/lib/python2.7/site-packages/neutron/db/l3_dvrscheduler_db.py:331
2015-09-22 10:36:44.678 17715 DEBUG neutron.db.l3_dvrscheduler_db [req-59ea4621-dd62-47c0-8b2b-b85ae6d0cff5 ] DVR: dvr_update_router_addvm None  dvr_update_router_addvm /usr/lib/python2.7/site-packages/neutron/db/l3_dvrscheduler_db.py:118
2015-09-22 10:36:44.679 17715 DEBUG oslo_messaging._drivers.amqp [req-59ea4621-dd62-47c0-8b2b-b85ae6d0cff5 ] UNIQUE_ID is 7d0b3de3813e45f689863f43b19fbdd1. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2015-09-22 10:36:44.680 17715 DEBUG oslo_messaging._drivers.amqp [req-59ea4621-dd62-47c0-8b2b-b85ae6d0cff5 ] UNIQUE_ID is 4a6ccdfd26084536816ea8d0f145faef. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2015-09-22 10:36:45.306 17715 INFO neutron.wsgi [-] (17715) accepted ('127.0.0.1', 41583)
2015-09-22 10:36:45.322 17715 DEBUG neutron.policy [req-8c55ec37-0c2d-4b18-bbe7-9971e2fec971 ] Enforcing rules: ['get_port:binding:host_id'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:45.322 17715 DEBUG neutron.policy [req-8c55ec37-0c2d-4b18-bbe7-9971e2fec971 ] Enforcing rules: ['get_port:binding:profile'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:45.323 17715 DEBUG neutron.policy [req-8c55ec37-0c2d-4b18-bbe7-9971e2fec971 ] Enforcing rules: ['get_port:binding:vif_details'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:45.323 17715 DEBUG neutron.policy [req-8c55ec37-0c2d-4b18-bbe7-9971e2fec971 ] Enforcing rules: ['get_port:binding:vif_type'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:45.324 17715 INFO neutron.wsgi [req-8c55ec37-0c2d-4b18-bbe7-9971e2fec971 ] 127.0.0.1 - - [22/Sep/2015 10:36:45] "GET /v2.0/ports.json?device_id=48abbcd2-bdc6-4646-8789-2b74824fc5c1 HTTP/1.1" 200 795 0.017117
2015-09-22 10:36:45.340 17715 INFO neutron.wsgi [req-f7f9756a-e98a-4c7b-98ea-25e6d9960fa8 ] 127.0.0.1 - - [22/Sep/2015 10:36:45] "GET /v2.0/security-groups.json?id=ba46adee-99d6-410f-a37a-f5e1a91224bb HTTP/1.1" 200 2074 0.013992
2015-09-22 10:36:45.501 17715 INFO neutron.wsgi [-] (17715) accepted ('172.16.22.22', 37492)
2015-09-22 10:36:45.504 17715 DEBUG neutron.wsgi [req-a3b910fe-05c2-4a85-8586-16fda8e4d31d ] http://172.16.22.22:9696/v2.0/extensions.json returned with HTTP 200 __call__ /usr/lib/python2.7/site-packages/neutron/wsgi.py:908
2015-09-22 10:36:45.504 17715 INFO neutron.wsgi [req-a3b910fe-05c2-4a85-8586-16fda8e4d31d ] 172.16.22.22 - - [22/Sep/2015 10:36:45] "GET /v2.0/extensions.json HTTP/1.1" 200 6067 0.002669
2015-09-22 10:36:46.638 17715 DEBUG neutron.notifiers.nova [-] Sending events: [{'status': 'completed', 'tag': u'4daf688b-0d19-4cea-86b0-e89ac0517c41', 'name': 'network-vif-plugged', 'server_uuid': u'48abbcd2-bdc6-4646-8789-2b74824fc5c1'}] send_events /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:240
2015-09-22 10:36:46.639 17715 DEBUG keystoneclient.auth.identity.v3.base [-] Making authentication request to http://172.16.21.22:35357/v3/auth/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py:171
2015-09-22 10:36:46.692 17715 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'4daf688b-0d19-4cea-86b0-e89ac0517c41', 'name': 'network-vif-plugged', 'server_uuid': u'48abbcd2-bdc6-4646-8789-2b74824fc5c1'}]
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova Traceback (most recent call last):
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 243, in send_events
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     batched_events)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return_raw=True)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 161, in _create
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     _resp, body = self.api.client.post(url, body=body)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 89, in request
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return self.session.request(url, method, **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return func(*args, **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 334, in request
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     base_url = self.get_endpoint(auth, **endpoint_filter)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in get_endpoint
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return auth.get_endpoint(self, **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 219, in get_endpoint
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     service_name=service_name)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     return func(*args, **kwargs)
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova   File "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line 216, in url_for
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova     raise exceptions.EmptyCatalog(_('The service catalog is empty.'))
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova EmptyCatalog: The service catalog is empty.
2015-09-22 10:36:46.692 17715 TRACE neutron.notifiers.nova
2015-09-22 10:36:50.748 17715 INFO neutron.wsgi [-] (17715) accepted ('127.0.0.1', 41599)
2015-09-22 10:36:50.767 17715 DEBUG neutron.policy [req-6e2a0b0d-bea5-4c49-b65b-8754b993f763 ] Enforcing rules: ['get_port:binding:host_id'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:50.768 17715 DEBUG neutron.policy [req-6e2a0b0d-bea5-4c49-b65b-8754b993f763 ] Enforcing rules: ['get_port:binding:profile'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:50.768 17715 DEBUG neutron.policy [req-6e2a0b0d-bea5-4c49-b65b-8754b993f763 ] Enforcing rules: ['get_port:binding:vif_details'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:50.768 17715 DEBUG neutron.policy [req-6e2a0b0d-bea5-4c49-b65b-8754b993f763 ] Enforcing rules: ['get_port:binding:vif_type'] log_rule_list /usr/lib/python2.7/site-packages/neutron/policy.py:363
2015-09-22 10:36:50.769 17715 INFO neutron.wsgi [req-6e2a0b0d-bea5-4c49-b65b-8754b993f763 ] 127.0.0.1 - - [22/Sep/2015 10:36:50] "GET /v2.0/ports.json?device_id=48abbcd2-bdc6-4646-8789-2b74824fc5c1 HTTP/1.1" 200 795 0.020777
2015-09-22 10:36:50.786 17715 INFO neutron.wsgi [req-48d410d8-e573-4f88-b6ae-11c2ab648753 ] 127.0.0.1 - - [22/Sep/2015 10:36:50] "GET /v2.0/security-groups.json?id=ba46adee-99d6-410f-a37a-f5e1a91224bb HTTP/1.1" 200 2074 0.014805
2015-09-22 10:36:51.084 17715 INFO neutron.wsgi [-] (17715) accepted ('172.16.22.22', 37510)
2015-09-22 10:36:51.087 17715 DEBUG neutron.wsgi [req-4b6abe56-9059-4182-b235-0b55532a0582 ] http://172.16.22.22:9696/v2.0/extensions.json returned with HTTP 200 __call__ /usr/lib/python2.7/site-packages/neutron/wsgi.py:908
2015-09-22 10:36:51.087 17715 INFO neutron.wsgi [req-4b6abe56-9059-4182-b235-0b55532a0582 ] 172.16.22.22 - - [22/Sep/2015 10:36:51] "GET /v2.0/extensions.json HTTP/1.1" 200 6067 0.002895