Exception from Cinder `Circular reference detected` when enabled JSON formatter for syslog and creating cinder volumes in Red Hat OpenStack Platform
Issue
When enabling JSON format for syslog and setting level to DEBUG
on Cinder service, one cannot create volumes because exception Circular reference detected
is thrown when log API is called.
- Exception thrown when using JSON formatter for syslog
- Circular reference detected when parsing data with oslo messaging
[root@overcloud-controller-1 ~]#
2017-07-24 19:29:38,038.038 738419 ERROR cinder.scheduler.flows.create_volume [req-f8a13919-60c1-4357-a28f-39da800ae6cf dbd711fe26684684a09803c6ebe3f83d bb9fa77129e941a8b8d9acc43bbb4238 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: Circular reference detected
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager Traceback (most recent call last):
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager result = task.execute(**arguments)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 142, in execute
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self._handle_failure(context, request_spec, e)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 103, in _handle_failure
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager {'cause': cause, 'name': self.name})
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 1446, in error
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self.logger.error(msg, *args, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 1175, in error
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self._log(ERROR, msg, args, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self.handle(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self.callHandlers(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager hdlr.handle(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager self.emit(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/handlers.py", line 843, in emit
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager msg = self.format(record) + '\000'
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager return fmt.format(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/oslo_log/formatters.py", line 161, in format
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager return jsonutils.dumps(message)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 190, in dumps
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager return json.dumps(obj, default=default, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager sort_keys=sort_keys, **kw).encode(obj)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager chunks = self.iterencode(o, _one_shot=True)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager return _iterencode(o, 0)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager ValueError: Circular reference detected
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager
For more details, see Diagnostic Steps
.
Environment
- Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.