gnocchi's metricd log fills up quickly with repeated failure messages

Solution Verified - Updated -

Issue

OpenStack gnocchi data collection database service /var/log/gnocchi/metricd.log fills up quickly, potentially resulting in 100% file system full and causing OpenStack outages.

The following messages can be found repeating ad nauseam in /var/log/gnocchi/metricd.log:

2017-12-22 18:08:25.555 412932 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.956 413203 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.957 413079 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.960 412257 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.618 412864 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.638 412404 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:28:33.599 436628 INFO cotyledon [-] Parent process has died unexpectedly, processing(1717) [436628] exiting
2017-12-22 18:28:33.599 468665 INFO cotyledon [-] Parent process has died unexpectedly, processing(2947) [468665] exiting
2017-12-22 18:28:33.600 468665 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 468491 INFO cotyledon [-] Parent process has died unexpectedly, processing(2934) [468491] exiting
2017-12-22 18:28:33.600 436563 INFO cotyledon [-] Parent process has died unexpectedly, processing(1711) [436563] exiting
2017-12-22 18:28:33.600 468491 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 436563 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466258 INFO cotyledon [-] Parent process has died unexpectedly, processing(2843) [466258] exiting
2017-12-22 18:28:33.601 435612 INFO cotyledon [-] Parent process has died unexpectedly, processing(1673) [435612] exiting
2017-12-22 18:28:33.601 466258 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 435612 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466410 INFO cotyledon [-] Parent process has died unexpectedly, processing(2847) [466410] exiting
2017-12-22 18:28:33.602 466410 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:10:09.094 475430 INFO gnocchi.cli [-] 1 measurements bundles across 1 metrics wait to be processed.
2017-12-22 18:12:09.077 475430 INFO gnocchi.cli [-] 189 measurements bundles across 25 metrics wait to be processed.
2017-12-22 18:14:09.089 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:16:09.073 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:18:09.099 475430 INFO gnocchi.cli [-] 17 measurements bundles across 17 metrics wait to be processed.
2017-12-22 18:20:09.081 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:22:09.082 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:24:09.083 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli [-] Unexpected error during measures processing
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli Traceback (most recent call last):
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 275, in _run_job
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli     metrics = self.queue.get(block=True, timeout=10)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "<string>", line 2, in get
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 755, in _callmethod
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli     self._connect()
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "/usr/lib64/python2.7/multiprocessing/managers.py", line 742, in _connect
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli     conn = self._Client(self._token.address, authkey=self._authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "/usr/lib64/python2.7/multiprocessing/connection.py", line 179, in Client
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli     answer_challenge(c, authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli   File "/usr/lib64/python2.7/multiprocessing/connection.py", line 435, in answer_challenge
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli     message = connection.recv_bytes(256)         # reject large message
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli EOFError
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli 

Environment

Red Hat OpenStack Platform 10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content