openstack commands fails with "DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') "

Solution In Progress - Updated -

Issue

  • ceilometer-expirer command fails with DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction')
  • Different openstack componant is failing with InternalError: (1205, u'Lock wait timeout exceeded; try restarting transaction')
  • Getting error below error in logs :
2017-11-17 11:06:24.201 8802 WARNING oslo_reports.guru_meditation_report [-] Guru mediation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'DELETE FROM sample WHERE sample.timestamp < %s'] [parameters: (Decimal('1510801584.364885'),)]
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     context)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     _check_mysql_exception(errinfo)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters     raise InternalError(errno, errorvalue)
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1205, u'Lock wait timeout exceeded; try restarting transaction')
2017-11-17 11:07:15.552 8802 ERROR oslo_db.sqlalchemy.exc_filters 
2017-11-17 11:07:15.557 8802 CRITICAL ceilometer [-] DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'DELETE FROM sample WHERE sample.timestamp < %s'] [parameters: (Decimal('1510801584.364885'),)]
2017-11-17 11:07:15.557 8802 ERROR ceilometer Traceback (most recent call last):
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/bin/ceilometer-expirer", line 10, in <module>
2017-11-17 11:07:15.557 8802 ERROR ceilometer     sys.exit(expirer())
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 42, in expirer
2017-11-17 11:07:15.557 8802 ERROR ceilometer     cfg.CONF.database.metering_time_to_live)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/storage/impl_sqlalchemy.py", line 384, in clear_expired_metering_data
2017-11-17 11:07:15.557 8802 ERROR ceilometer     rows = sample_q.delete()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3048, in delete
2017-11-17 11:07:15.557 8802 ERROR ceilometer     delete_op.exec_()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_
2017-11-17 11:07:15.557 8802 ERROR ceilometer     self._do_exec()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec
2017-11-17 11:07:15.557 8802 ERROR ceilometer     mapper=self.mapper)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1046, in execute
2017-11-17 11:07:15.557 8802 ERROR ceilometer     bind, close_with_result=True).execute(clause, params or {})
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-11-17 11:07:15.557 8802 ERROR ceilometer     return meth(self, multiparams, params)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2017-11-17 11:07:15.557 8802 ERROR ceilometer     return connection._execute_clauseelement(self, multiparams, params)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-11-17 11:07:15.557 8802 ERROR ceilometer     compiled_sql, distilled_params
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2017-11-17 11:07:15.557 8802 ERROR ceilometer     context)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2017-11-17 11:07:15.557 8802 ERROR ceilometer     util.raise_from_cause(newraise, exc_info)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-11-17 11:07:15.557 8802 ERROR ceilometer     reraise(type(exception), exception, tb=exc_tb)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-11-17 11:07:15.557 8802 ERROR ceilometer     context)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-11-17 11:07:15.557 8802 ERROR ceilometer     cursor.execute(statement, parameters)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2017-11-17 11:07:15.557 8802 ERROR ceilometer     result = self._query(query)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2017-11-17 11:07:15.557 8802 ERROR ceilometer     conn.query(q)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
2017-11-17 11:07:15.557 8802 ERROR ceilometer     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result
2017-11-17 11:07:15.557 8802 ERROR ceilometer     result.read()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
2017-11-17 11:07:15.557 8802 ERROR ceilometer     first_packet = self.connection._read_packet()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2017-11-17 11:07:15.557 8802 ERROR ceilometer     packet.check_error()
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2017-11-17 11:07:15.557 8802 ERROR ceilometer     err.raise_mysql_exception(self._data)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2017-11-17 11:07:15.557 8802 ERROR ceilometer     _check_mysql_exception(errinfo)
2017-11-17 11:07:15.557 8802 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2017-11-17 11:07:15.557 8802 ERROR ceilometer     raise InternalError(errno, errorvalue)
2017-11-17 11:07:15.557 8802 ERROR ceilometer DBError: (pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') [SQL: u'DELETE FROM sample WHERE sample.timestamp < %s'] [parameters: (Decimal('1510801584.364885'),)]
2017-11-17 11:07:15.557 8802 ERROR ceilometer

Environment

  • Openstack Platform 8
  • Openstack Platform 9

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content