gnocchi's metricd log fills up quickly with repeated failure messages
Issue
OpenStack gnocchi data collection database service /var/log/gnocchi/metricd.log fills up quickly, potentially resulting in 100% file system full and causing OpenStack outages.
The following messages can be found repeating ad nauseam in /var/log/gnocchi/metricd.log:
2017-12-22 18:08:25.555 412932 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.956 413203 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.957 413079 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.960 412257 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.618 412864 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.638 412404 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:28:33.599 436628 INFO cotyledon [-] Parent process has died unexpectedly, processing(1717) [436628] exiting
2017-12-22 18:28:33.599 468665 INFO cotyledon [-] Parent process has died unexpectedly, processing(2947) [468665] exiting
2017-12-22 18:28:33.600 468665 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 468491 INFO cotyledon [-] Parent process has died unexpectedly, processing(2934) [468491] exiting
2017-12-22 18:28:33.600 436563 INFO cotyledon [-] Parent process has died unexpectedly, processing(1711) [436563] exiting
2017-12-22 18:28:33.600 468491 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 436563 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466258 INFO cotyledon [-] Parent process has died unexpectedly, processing(2843) [466258] exiting
2017-12-22 18:28:33.601 435612 INFO cotyledon [-] Parent process has died unexpectedly, processing(1673) [435612] exiting
2017-12-22 18:28:33.601 466258 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 435612 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466410 INFO cotyledon [-] Parent process has died unexpectedly, processing(2847) [466410] exiting
2017-12-22 18:28:33.602 466410 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:10:09.094 475430 INFO gnocchi.cli [-] 1 measurements bundles across 1 metrics wait to be processed.
2017-12-22 18:12:09.077 475430 INFO gnocchi.cli [-] 189 measurements bundles across 25 metrics wait to be processed.
2017-12-22 18:14:09.089 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:16:09.073 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:18:09.099 475430 INFO gnocchi.cli [-] 17 measurements bundles across 17 metrics wait to be processed.
2017-12-22 18:20:09.081 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:22:09.082 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:24:09.083 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli [-] Unexpected error during measures processing
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli Traceback (most recent call last):
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 275, in _run_job
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli metrics = self.queue.get(block=True, timeout=10)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "<string>", line 2, in get
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/managers.py", line 755, in _callmethod
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli self._connect()
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/managers.py", line 742, in _connect
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli conn = self._Client(self._token.address, authkey=self._authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/connection.py", line 179, in Client
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli answer_challenge(c, authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/connection.py", line 435, in answer_challenge
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli message = connection.recv_bytes(256) # reject large message
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli EOFError
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli
Environment
Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
