Upon deployment, nova nodes get scheduled but they never get powered on - Data too long for column 'instance_info' in Red Hat OpenStack Platform
Issue
Upon deployment, nova nodes get scheduled but they never get powered on - Data too long for column 'instance_info' in Red Hat OpenStack Platform
The issue's symptoms are:
- Ironic nodes introspect fine, nodes can be turned on or off using ironic node-set-power-state.
- During deployment attempts, nova nodes get scheduled but they never get powered on for the deploy:
[stack@instack ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------------+--------+------------+-------------+----------------------+
| 23c40e7a-688c-440d-8b4c-9a91fe989e22 | cmpt-0 | BUILD | spawning | NOSTATE | ctlplane=10.20.0.104 |
| 7ae97372-8c79-4905-a7cd-cc8bea993dc8 | ctrl-0 | BUILD | spawning | NOSTATE | ctlplane=10.20.0.111 |
+--------------------------------------+--------------+--------+------------+-------------+----------------------+
[stack@instack ~]$ ironic node-list
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
[....]
| 57b7a089-d3b6-44c9-8c1d-77eba0b1d011 | osp-baremetal-2 | 7ae97372-8c79-4905-a7cd-cc8bea993dc8 | power off | deploying | False |
[....]
| aa04082a-7a58-4b81-bbf9-b8cab8891b9d | osp-baremetal-11 | 23c40e7a-688c-440d-8b4c-9a91fe989e22 | power off | deploying | False |
[....]
+--------------------------------------+------------------+--------------------------------------+-------------+--------------------+-------------+
openstack overcloud deploy stays stuck at:
[....]
2017-03-16 16:20:05Z [overcloud.Compute.0.NovaCompute]: CREATE_IN_PROGRESS state changed
2017-03-16 16:20:06Z [overcloud.Controller.0.Controller]: CREATE_IN_PROGRESS state changed
The ironic-conductor logs show the following error message:
[random blob....]
bsz6syET1fOUV5rRZxsS5fOLYzEUms/FPR5M6cFL3Io85L13XGbjqV4r3B2dFefkZl+rOaZ+qKgz3JFBwX9djL3AM59czTGB1okkZjKDIujyqXeiab88r07FwsDFLhMRqnYildK/m2kOJkL66W5c6iGu3X3KswNYXF/8NcY9uH5cxeec6VTifmED7ePzqP5
ZqJWY9Gva14rsbWuJ6VLCYbjgdNpIs6/z2ebhZbeHt6sh+zmVZz86ytLt9fXU5//OPjdDxWztZWnrz68x/eXjXR7crRWW9zd3vzl/H4pDjvrfZS5fJwd/3B+OB8fav3w9HZoHpQce/9fx4PBl/vjNcHuwcr69tfP9p6+LDorW1trfceFsVab/BgOOit339Y
PBh+8/VqMRj8Uz7p0rd8O9rb6adfZ7i59mhtbX1zPf239ujxZrH2+P43jzfXN79ZXR8Ujx6ljW5987tmreLDcQqHRv37aw/XHz5+8Ojx13ki32qqrTzj6yi6idMyu726w64X9WkqdbtrDx/m6WOWqsN5dJKP8VKxdT9PeV8d0ZiENlKZUrvi09/3qfV+EAO
E+R6ok3R2zTktVh9ccw97PXMmAsDfu4+msgcA9T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPwN+m/gxomVADALAA==", "swap_mb": "0"}', 61)]
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters context)
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)
[....]
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 112, in _check_mysql_exception
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass(errno, errorvalue)
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters DataError: (1406, u"Data too long for column 'instance_info' at row 1")
2017-03-17 14:32:05.886 18764 ERROR oslo_db.sqlalchemy.exc_filters
2017-03-17 14:32:05.912 18764 ERROR oslo_db.sqlalchemy.exc_filters [req-eb000c3c-2ac0-41ec-914d-3e8aae9a2417 - - - - -] DBAPIError exception wrapped from (pymysql.err.DataError) (1406, u"Data too long for column 'last_error' at row 1") [SQL: u'UPDATE nodes SET updated_at=%s, provision_state=%s, provision_updated_at=%s, last_error=%s, instance_info=%s WHERE nodes.id = %s'] [parameters: (datetime.datetime(2017, 3, 17, 18, 32, 5, 902889), u'deploy failed', datetime.datetime(2017, 3, 17, 18, 32, 5, 902312), u'Async execution of do_node_deploy failed with error: (pymysql.err.DataError) (1406, u"Data too long for column \'instance_info\' at row 1") [SQL: u\'UPDATE nodes SET updated_at=%s, provision_state=%s, provision_updated_at=%s, instance_info=%s WHERE nodes.id = %s\'] [parameters: (datetime.datetime(2017, 3, 17, 18, 32, 5, 881427), u\'deploy failed\', datetime.datetime(2017, 3, 17, 18, 32, 5, 880807), \'{"root_gb": "40", "display_name": "cmpt-0", "image_source": "92b54d1e-99f5-4ea3-9cc9-62d496d4ff0e", "capabilities": "{\\\\"profile\\\\": \\\\"compute\\\\", \\\\"boot_option\\\\": \\\\"local\\\\"}", "memory_mb": "4096", "vcpus": "1", "local_gb": "127", "configdrive": "H4sICJIrzFgC/3RtcHNmNWNFNwDsvemSG8mWHgje29PdGd3WM9N9RzYma3V7gaybmSQCW+4s4TZzQTJB5sJKZDJJFnlhAUQAiEwgAowIAAmysk39Q+qW2Zj+ankDLW8gqWWmB5CeRDb6qX8z5xz3
[random blob....]
Environment
Red Hat OpenStack Platform 10
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
