OpenStack overcloud upgrade prepare stuck

Solution Unverified - Updated -

Issue

During an upgrade from OSP 12 to OSP 13, the step $ openstack overcloud upgrade prepare hangs for hours.

Here is the latest messages on stdout :

2020-06-04 11:19:28Z [xxxxx.NodeTimesyncUserData]: CREATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.NodeUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.DeploymentActions]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxxx.RoleUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.DeploymentActions]: UPDATE_COMPLETE  state changed
2020-06-04 11:19:30Z [xxxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:31Z [xxxxxx.NodeTimesyncUserData]: CREATE_COMPLETE  state changed
2020-06-04 11:19:31Z [xxxx.UpdateConfig]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:31Z [xxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:32Z [xxxxxUpdateConfig]: UPDATE_COMPLETE  state changed
2020-06-04 11:19:33Z [xxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed

Here some errors in Heat logs:

heat/heat-engine.log:2020-06-04 11:18:38.141 3017 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 39147, u'id_3': 34196, u'id_1': 39144, u'id_4': 34197, u'id_5': 34141}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 39147, u'id_3': 34196, u'id_1': 39144, u'id_4': 34197, u'id_5': 34141}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:18:43.480 3013 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s)'] [parameters: {u'id_2': 57825, u'id_3': 55811, u'id_1': 55808, u'id_6': 55803, u'id_4': 55881, u'id_5': 57970}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s)'] [parameters: {u'id_2': 57825, u'id_3': 55811, u'id_1': 55808, u'id_6': 55803, u'id_4': 55881, u'id_5': 57970}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:19:06.865 3016 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 52754, u'id_3': 52759, u'id_1': 52641, u'id_4': 52756, u'id_5': 52807}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 52754, u'id_3': 52759, u'id_1': 52641, u'id_4': 52756, u'id_5': 52807}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:19:23.669 3016 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s)'] [parameters: {u'id_2': 37638, u'id_3': 37607, u'id_1': 34801}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s)'] [parameters: {u'id_2': 37638, u'id_3': 37607, u'id_1': 34801}] (Background on this error at: http://sqlalche.me/e/gkpj)

Environment

  • Red Hat Openstack RH OSP 12
  • Red Hat Openstack RHOSP 13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In