gnocchi's metricd log fills up quickly with repeated failure messages
Issue
OpenStack gnocchi
data collection database service /var/log/gnocchi/metricd.log
fills up quickly, potentially resulting in 100% file system full and causing OpenStack outages.
The following messages can be found repeating ad nauseam in /var/log/gnocchi/metricd.log
:
2017-12-22 18:08:25.555 412932 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.956 413203 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.957 413079 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.960 412257 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.618 412864 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:08:25.638 412404 INFO gnocchi.storage.ceph [-] Ceph storage backend use 'cradox' python library
2017-12-22 18:28:33.599 436628 INFO cotyledon [-] Parent process has died unexpectedly, processing(1717) [436628] exiting
2017-12-22 18:28:33.599 468665 INFO cotyledon [-] Parent process has died unexpectedly, processing(2947) [468665] exiting
2017-12-22 18:28:33.600 468665 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 468491 INFO cotyledon [-] Parent process has died unexpectedly, processing(2934) [468491] exiting
2017-12-22 18:28:33.600 436563 INFO cotyledon [-] Parent process has died unexpectedly, processing(1711) [436563] exiting
2017-12-22 18:28:33.600 468491 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.600 436563 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466258 INFO cotyledon [-] Parent process has died unexpectedly, processing(2843) [466258] exiting
2017-12-22 18:28:33.601 435612 INFO cotyledon [-] Parent process has died unexpectedly, processing(1673) [435612] exiting
2017-12-22 18:28:33.601 466258 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 435612 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:28:33.601 466410 INFO cotyledon [-] Parent process has died unexpectedly, processing(2847) [466410] exiting
2017-12-22 18:28:33.602 466410 INFO gnocchi.cli [-] Waiting ongoing metric processing to finish
2017-12-22 18:10:09.094 475430 INFO gnocchi.cli [-] 1 measurements bundles across 1 metrics wait to be processed.
2017-12-22 18:12:09.077 475430 INFO gnocchi.cli [-] 189 measurements bundles across 25 metrics wait to be processed.
2017-12-22 18:14:09.089 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:16:09.073 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:18:09.099 475430 INFO gnocchi.cli [-] 17 measurements bundles across 17 metrics wait to be processed.
2017-12-22 18:20:09.081 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:22:09.082 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:24:09.083 475430 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli [-] Unexpected error during measures processing
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli Traceback (most recent call last):
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 275, in _run_job
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli metrics = self.queue.get(block=True, timeout=10)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "<string>", line 2, in get
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/managers.py", line 755, in _callmethod
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli self._connect()
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/managers.py", line 742, in _connect
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli conn = self._Client(self._token.address, authkey=self._authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/connection.py", line 179, in Client
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli answer_challenge(c, authkey)
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli File "/usr/lib64/python2.7/multiprocessing/connection.py", line 435, in answer_challenge
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli message = connection.recv_bytes(256) # reject large message
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli EOFError
2017-12-22 18:25:37.973 423885 ERROR gnocchi.cli
Environment
Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.