Ceilometers database keeps growing even though TTL in the ceilometer.conf is defined

Solution In Progress - Updated -

Issue

We have been investigating this issue for some time and of course made sure the samples are NOT deleted, So the count is rising, even though ttl is set to 600 (10 minutes)

This shows the ttl is defined:

rs:PRIMARY> db.meter.getIndexes()
[
        {
                "v" : 1,
                "key" : {
                        "_id" : 1
                },
                "name" : "_id_",
                "ns" : "ceilometer.meter"
        },
        {
                "v" : 1,
                "key" : {
                        "resource_id" : 1,
                        "user_id" : 1,
                        "counter_name" : 1,
                        "timestamp" : 1,
                        "source" : 1
                },
                "name" : "meter_idx",
                "ns" : "ceilometer.meter",
                "background" : false
        },
        {
                "v" : 1,
                "key" : {
                        "resource_id" : 1,
                        "project_id" : 1,
                        "counter_name" : 1,
                        "timestamp" : 1,
                        "source" : 1
                },
                "name" : "meter_project_idx",
                "ns" : "ceilometer.meter",
                "background" : true
        },
        {
                "v" : 1,
                "key" : {
                        "timestamp" : -1
                },
                "name" : "timestamp_idx",
                "ns" : "ceilometer.meter"
        },
        {
                "v" : 1,
                "key" : {
                        "timestamp" : 1
                },
                "name" : "meter_ttl",
                "expireAfterSeconds" : 600,
                "ns" : "ceilometer.meter"
        }
]
rs:PRIMARY>

This shows the rising sample count:

rs:PRIMARY> db.meter.count()
15585779

few minutes later

rs:PRIMARY> db.meter.count()
15595350

Checking the ttl execution (should run every minute) shows that it does not run anymore.

rs:PRIMARY> db.serverStatus().metrics.ttl
{ "deletedDocuments" : NumberLong(26999680), "passes" : NumberLong(2470) }

few minutes later

rs:PRIMARY> db.serverStatus().metrics.ttl
{ "deletedDocuments" : NumberLong(26999680), "passes" : NumberLong(2470) }

And, of course i've ran a query, such as the following, to see that indeed there are samples that should have been deleted (the specified date is yesterday, beyond the ttl), this gives lots of results:

 db.meter.find({"timestamp" : {$lt : ISODate("2015-08-09T01:20:00Z")}})

Environment

  • Red Hat OpenStack 6.0
  • mongodb-server-2.6.5-2.el7ost

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content