OpenStack overcloud upgrade prepare stuck

Solution Unverified - Updated -

Issue

During an upgrade from OSP 12 to OSP 13, the step $ openstack overcloud upgrade prepare hangs for hours.

Here is the latest messages on stdout :

2020-06-04 11:19:28Z [xxxxx.NodeTimesyncUserData]: CREATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.NodeUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.DeploymentActions]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxxx.RoleUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:28Z [xxxxx.DeploymentActions]: UPDATE_COMPLETE  state changed
2020-06-04 11:19:30Z [xxxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:31Z [xxxxxx.NodeTimesyncUserData]: CREATE_COMPLETE  state changed
2020-06-04 11:19:31Z [xxxx.UpdateConfig]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:31Z [xxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed
2020-06-04 11:19:32Z [xxxxxUpdateConfig]: UPDATE_COMPLETE  state changed
2020-06-04 11:19:33Z [xxxxx.NodeAdminUserData]: UPDATE_IN_PROGRESS  state changed

Here some errors in Heat logs:

heat/heat-engine.log:2020-06-04 11:18:38.141 3017 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 39147, u'id_3': 34196, u'id_1': 39144, u'id_4': 34197, u'id_5': 34141}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 39147, u'id_3': 34196, u'id_1': 39144, u'id_4': 34197, u'id_5': 34141}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:18:43.480 3013 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s)'] [parameters: {u'id_2': 57825, u'id_3': 55811, u'id_1': 55808, u'id_6': 55803, u'id_4': 55881, u'id_5': 57970}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s, %(id_6)s)'] [parameters: {u'id_2': 57825, u'id_3': 55811, u'id_1': 55808, u'id_6': 55803, u'id_4': 55881, u'id_5': 57970}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:19:06.865 3016 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 52754, u'id_3': 52759, u'id_1': 52641, u'id_4': 52756, u'id_5': 52807}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s, %(id_4)s, %(id_5)s)'] [parameters: {u'id_2': 52754, u'id_3': 52759, u'id_1': 52641, u'id_4': 52756, u'id_5': 52807}] (Background on this error at: http://sqlalche.me/e/gkpj)
heat/heat-engine.log:2020-06-04 11:19:23.669 3016 ERROR heat.db.sqlalchemy.api [req-d114d045-c063-4b01-876e-ad45091635af - admin - default default] Failed to purge events: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s)'] [parameters: {u'id_2': 37638, u'id_3': 37607, u'id_1': 34801}] (Background on this error at: http://sqlalche.me/e/gkpj): DBReferenceError: (pymysql.err.IntegrityError) (1451, u'Cannot delete or update a parent row: a foreign key constraint fails (`heat`.`event`, CONSTRAINT `ev_rsrc_prop_data_ref` FOREIGN KEY (`rsrc_prop_data_id`) REFERENCES `resource_properties_data` (`id`))') [SQL: u'DELETE FROM resource_properties_data WHERE resource_properties_data.id IN (%(id_1)s, %(id_2)s, %(id_3)s)'] [parameters: {u'id_2': 37638, u'id_3': 37607, u'id_1': 34801}] (Background on this error at: http://sqlalche.me/e/gkpj)

Environment

  • Red Hat Openstack RH OSP 12
  • Red Hat Openstack RHOSP 13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content