Ceilometers database keeps growing even though TTL in the ceilometer.conf is defined
Issue
We have been investigating this issue for some time and of course made sure the samples are NOT deleted, So the count is rising, even though ttl is set to 600 (10 minutes)
This shows the ttl is defined:
rs:PRIMARY> db.meter.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "ceilometer.meter"
},
{
"v" : 1,
"key" : {
"resource_id" : 1,
"user_id" : 1,
"counter_name" : 1,
"timestamp" : 1,
"source" : 1
},
"name" : "meter_idx",
"ns" : "ceilometer.meter",
"background" : false
},
{
"v" : 1,
"key" : {
"resource_id" : 1,
"project_id" : 1,
"counter_name" : 1,
"timestamp" : 1,
"source" : 1
},
"name" : "meter_project_idx",
"ns" : "ceilometer.meter",
"background" : true
},
{
"v" : 1,
"key" : {
"timestamp" : -1
},
"name" : "timestamp_idx",
"ns" : "ceilometer.meter"
},
{
"v" : 1,
"key" : {
"timestamp" : 1
},
"name" : "meter_ttl",
"expireAfterSeconds" : 600,
"ns" : "ceilometer.meter"
}
]
rs:PRIMARY>
This shows the rising sample count:
rs:PRIMARY> db.meter.count()
15585779
few minutes later
rs:PRIMARY> db.meter.count()
15595350
Checking the ttl execution (should run every minute) shows that it does not run anymore.
rs:PRIMARY> db.serverStatus().metrics.ttl
{ "deletedDocuments" : NumberLong(26999680), "passes" : NumberLong(2470) }
few minutes later
rs:PRIMARY> db.serverStatus().metrics.ttl
{ "deletedDocuments" : NumberLong(26999680), "passes" : NumberLong(2470) }
And, of course i've ran a query, such as the following, to see that indeed there are samples that should have been deleted (the specified date is yesterday, beyond the ttl), this gives lots of results:
db.meter.find({"timestamp" : {$lt : ISODate("2015-08-09T01:20:00Z")}})
Environment
- Red Hat OpenStack 6.0
- mongodb-server-2.6.5-2.el7ost
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.