Exception from Cinder `Circular reference detected` when enabled JSON formatter for syslog and creating cinder volumes in Red Hat OpenStack Platform

Solution Unverified - Updated -

Issue

When enabling JSON format for syslog and setting level to DEBUG on Cinder service, one cannot create volumes because exception Circular reference detected is thrown when log API is called.

  • Exception thrown when using JSON formatter for syslog
  • Circular reference detected when parsing data with oslo messaging
[root@overcloud-controller-1 ~]# 
2017-07-24 19:29:38,038.038 738419 ERROR cinder.scheduler.flows.create_volume [req-f8a13919-60c1-4357-a28f-39da800ae6cf dbd711fe26684684a09803c6ebe3f83d bb9fa77129e941a8b8d9acc43bbb4238 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: Circular reference detected
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager Traceback (most recent call last):
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     result = task.execute(**arguments)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 142, in execute
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self._handle_failure(context, request_spec, e)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib/python2.7/site-packages/cinder/scheduler/flows/create_volume.py", line 103, in _handle_failure
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     {'cause': cause, 'name': self.name})
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 1446, in error
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self.logger.error(msg, *args, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 1175, in error
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self._log(ERROR, msg, args, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self.handle(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self.callHandlers(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     hdlr.handle(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     self.emit(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/handlers.py", line 843, in emit
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     msg = self.format(record) + '\000'
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     return fmt.format(record)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib/python2.7/site-packages/oslo_log/formatters.py", line 161, in format
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     return jsonutils.dumps(message)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 190, in dumps
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     return json.dumps(obj, default=default, **kwargs)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     sort_keys=sort_keys, **kw).encode(obj)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     chunks = self.iterencode(o, _one_shot=True)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager     return _iterencode(o, 0)
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager ValueError: Circular reference detected
2017-07-24 19:29:38,602.602 738419 ERROR cinder.scheduler.manager 

For more details, see Diagnostic Steps.

Environment

  • Red Hat OpenStack Platform 10

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content