Why ceilometer services cannot communicate with keystone and glance over https?

Issue

• Ceilometer services fail to work with keystone and glance services over https (with ssl). Problems have been detected at least in:

  • ceilometer-central-agent can not access glance over https.
  • ceilometer-alarm-evaluator can not access keystone over https.

• ceilometer-central-agent fails to get info from glance via ssh with below error message.:

2014-02-06 13:29:03.418 21957 WARNING ceilometer.central.manager [-] Continue after error from image.size: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
2014-02-06 13:29:03.418 21957 ERROR ceilometer.central.manager [-] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager Traceback (most recent call last):
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/ceilometer/central/manager.py", line 46, in poll_and_publish
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     cache,
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/ceilometer/image/glance.py", line 122, in get_samples
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     for image in self._iter_images(manager.keystone, cache):
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/ceilometer/image/glance.py", line 76, in _iter_images
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     cache['images'] = list(self._get_images(ksclient))
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/ceilometer/image/glance.py", line 52, in _get_images
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     client.images.list(filters={"is_public": False})))
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 174, in paginate
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     images = self._list(url, "images")
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/glanceclient/common/base.py", line 53, in _list
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     resp, body = self.api.json_request('GET', url)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 266, in json_request
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     resp, body_iter = self._http_request(url, method, **kwargs)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 225, in _http_request
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     conn.request(method, conn_url, **kwargs)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib64/python2.6/httplib.py", line 914, in request
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     self._send_request(method, url, body, headers)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     self.endheaders()
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     self._send_output()
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     self.send(msg)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib64/python2.6/httplib.py", line 759, in send
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     self.sock.sendall(str)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/eventlet/green/OpenSSL/SSL.py", line 99, in sendall
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     tail = self.send(data)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager   File "/usr/lib/python2.6/site-packages/eventlet/green/OpenSSL/SSL.py", line 79, in write
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager     return self.fd.write(data)
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]
2014-02-06 13:29:03.418 21957 TRACE ceilometer.central.manager

• ceilometer-agent-evaluator fails:

2014-02-04 19:09:47.498 20593 ERROR ceilometer.openstack.common.threadgroup [-] Authorization Failed: Unable to establish connection tohttps://jtechslbvip.hi.inet:35357/v2.0/tokens
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup Traceback (most recent call last):
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/threadgroup.py", line 117, in wait
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     x.wait()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/threadgroup.py", line 49, in wait
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     return self.thread.wait()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 166, in wait
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     return self._exit_event.wait()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     return self.greenlet.switch()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 192, in main
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometer/openstack/common/service.py", line 448, in run_service
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     service.start()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometer/alarm/service.py", line 182, in start
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     *[eval_interval, self._client])
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometer/alarm/service.py", line 90, in _client
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     self.api_client = ceiloclient.get_client(2, **creds)
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometerclient/client.py", line 81, in get_client
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     _ksclient = _get_ksclient(**ks_kwargs)
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/ceilometerclient/client.py", line 35, in _get_ksclient
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     insecure=kwargs.get('insecure'))
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     self.authenticate()
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 468, in authenticate
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     resp, body = self.get_raw_token_from_identity_service(**kwargs)
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/client.py", line 168, in get_raw_token_from_identity_service
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup     "%s" % e)
2014-02-04 19:09:47.498 20593 TRACE ceilometer.openstack.common.threadgroup AuthorizationFailure: Authorization Failed: Unable to establish connection tohttps://jtechslbvip.hi.inet:35357/v2.0/tokens