ceilometer-agent-compute is not connected to AMQP on initial deployment of new compute nodes in Red Hat OpenStack Platform
Issue
New compute nodes were added to the cloud (scale out). ceilometer-agent-compute is not connected to rabbitmq after deployment of these compute nodes.
The following script is being used to verify that the service properly connects to AMQP
https://github.com/openstack/osops-tools-monitoring/blob/b69dff7e6d44039c16430083bf6e2d5a539451f0/monitoring-for-openstack/oschecks/utils.py
def check_process_exists_and_amqp_connected(name):
processes = filter(lambda p: check_process_name(name, p),
psutil.process_iter())
if not processes:
critical("%s is not running" % name)
for p in processes:
try:
connections = p.get_connections(kind='inet')
except psutil.NoSuchProcess:
continue
found_amqp = (
len(list(itertools.takewhile(lambda c:
len(c.remote_address) <= 1 or
c.remote_address[1] != AMQP_PORT,
connections))) != len(connections))
if found_amqp:
ok("%s is working." % name)
critical("%s is not connected to AMQP" % name)
netstat shows that there is no ESTABLISHED connection from ceilometer to rabbitmq.
On the other hand, netstat reveals that e.g. nova-compute on the same compute node can connect to AMQP quite fine
Also, ceilometer logs look clean on this new compute node.
Environment
Red Hat Enterprise Linux OpenStack Platform 7.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.