ceilometer-agent-compute is not connected to AMQP on initial deployment of new compute nodes in Red Hat OpenStack Platform

Solution In Progress - Updated -

Issue

New compute nodes were added to the cloud (scale out). ceilometer-agent-compute is not connected to rabbitmq after deployment of these compute nodes.

The following script is being used to verify that the service properly connects to AMQP
https://github.com/openstack/osops-tools-monitoring/blob/b69dff7e6d44039c16430083bf6e2d5a539451f0/monitoring-for-openstack/oschecks/utils.py

def check_process_exists_and_amqp_connected(name):
    processes = filter(lambda p: check_process_name(name, p),
                       psutil.process_iter())
    if not processes:
        critical("%s is not running" % name)
    for p in processes:
        try:
            connections = p.get_connections(kind='inet')
        except psutil.NoSuchProcess:
            continue
        found_amqp = (
            len(list(itertools.takewhile(lambda c:
                len(c.remote_address) <= 1 or
                c.remote_address[1] != AMQP_PORT,
                connections))) != len(connections))
        if found_amqp:
            ok("%s is working." % name)
    critical("%s is not connected to AMQP" % name)

netstat shows that there is no ESTABLISHED connection from ceilometer to rabbitmq.

On the other hand, netstat reveals that e.g. nova-compute on the same compute node can connect to AMQP quite fine

Also, ceilometer logs look clean on this new compute node.

Environment

Red Hat Enterprise Linux OpenStack Platform 7.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content