Overcloud ComputeHCI scale-out fails at WorkflowTasks_Step2_Execution with "Got a packet bigger than 'max_allowed_packet' bytes"

Issue

Overcloud Deployment fails at WorkflowTasks_Step2_Execution with following error:

2019-04-03 16:40:15.624 1758 DEBUG mistral.services.scheduler [req-0f7abb3d-e516-4771-9a58-de54e139bc54 b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] Starting Scheduler loop [scheduler=<mistral.services.scheduler.Scheduler object at 0x7fd237afe5d0>]... _loop /usr/lib/python2.7/site-packages/mistral/services/scheduler.py:124
2019-04-03 16:40:15.715 1758 INFO workflow_trace [req-9ca67970-5443-4983-a97d-781e5475c10e b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] Action 'tripleo.ansible-playbook' (
367bf476-4f45-4dc3-a26c-1691802c6b69)(task=ceph_install) [RUNNING -> SUCCESS, result = {log_path: /tmp/ansible-mistral-actionqGtAA3/ansible.log, stderr: Using /usr/share/ceph-ansible/a...]
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters [req-9ca67970-5443-4983-a97d-781e5475c10e b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] DBAPIError exception wrapped from (pymysql.err.InternalError) (1153, u"Got a packet bigger than 'max_allowed_packet' bytes") [SQL: u'UPDATE action_executions_v2 SET updated_at=%(updated_at)s, state=%(state)s, accepted=%(accepted)
s, output=%(output)s WHERE action_executions_v2.id = %(action_executions_v2_id)s'] [parameters: {'output': '{"result": {"log_path": "/tmp/ansible-mistral-actionqGtAA3/ansible.log", "stderr": "Using /usr/share/ceph-ansible/ansible.cfg as config file\\n\\nPLA ... (15859543 characters truncated) ... statements should not include jinja2 templating delimiters\\nsuch as {{ }} or {% %}. Found: {{ groups.get(mgr_group_name, []
) | length > 0\\n}}\\n"}}', 'state': 'SUCCESS', 'accepted': 1, 'updated_at': datetime.datetime(2019, 4, 3, 9, 40, 15), 'action_executions_v2_id': u'367bf476-4f45-4dc3-a26c-1691802c6b69'}] (Background on this error at: http://sqlalche.me/e/2j85): InternalError: (1153, u"Got a packet bigger than 'max_allowed_packet' bytes")
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     context)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters     raise errorclass(errno, errval)
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1153, u"Got a packet bigger than 'max_allowed_packet' bytes")
2019-04-03 16:40:15.966 1758 ERROR oslo_db.sqlalchemy.exc_filters 
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils [req-9ca67970-5443-4983-a97d-781e5475c10e b0aa3fc44f2744c687bd42fae519a986 6a5995b0cd574b28a0187ed274fb738b - default default] DB error detected, operation will be retried: <function on_action_complete at 0x7fd237f24500>: DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils Traceback (most recent call last):
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/db/utils.py", line 50, in _with_auth_context
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     return func(*args, **kw)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/engine/action_queue.py", line 95, in decorate
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     res = func(*args, **kw)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     result = f(*args, **kwargs)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/engine/default_engine.py", line 132, in on_action_complete
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     action_handler.on_action_complete(action_ex, result)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 158, in wrapper
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     result = f(*args, **kwargs)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 54, in on_action_complete
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     task_handler.schedule_on_action_complete(action_ex)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/engine/task_handler.py", line 449, in schedule_on_action_complete
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     wf_action=isinstance(action_ex, models.WorkflowExecution)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/services/scheduler.py", line 101, in schedule_call
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     db_api.create_delayed_call(values)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/db/v2/api.py", line 358, in create_delayed_call
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     return IMPL.create_delayed_call(values)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/db/sqlalchemy/base.py", line 117, in _within_session
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     result = func(*args, **kw)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/api.py", line 996, in create_delayed_call
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     delayed_call.save(session)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     session.flush()
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2243, in flush
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self._flush(objects)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2369, in _flush
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     transaction.rollback(_capture_exception=True)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 76, in __exit__
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     compat.reraise(type_, value, traceback)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2369, in _flush
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     transaction.rollback(_capture_exception=True)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 532, in rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     util.reraise(*rollback_err)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 497, in rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     t[1].rollback()
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1632, in rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self._do_rollback()
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1670, in _do_rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self.connection._rollback_impl()
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 706, in _rollback_impl
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self._handle_dbapi_exception(e, None, None, None, None)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     util.raise_from_cause(newraise, exc_info)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 704, in _rollback_impl
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self.engine.dialect.do_rollback(self.connection)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 1775, in do_rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     dbapi_connection.rollback()
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 792, in rollback
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1092, in _execute_command
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     self._write_bytes(packet)
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1044, in _write_bytes
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils     "MySQL server has gone away (%r)" % (e,))
2019-04-03 16:40:16.010 1758 ERROR mistral.db.utils DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))") (Background on this error at: http://sqlalche.me/e/e3q8)