Nova Live Migration Failed to Update Database

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux OpenStack Platform (RHEL-OSP) 6.0
  • Red Hat Enterprise Linux OpenStack Platform 7.0
  • Red Hat OpenStack Platform 8
  • Red Hat OpenStack Platform 9
  • Red Hat OpenStack Platform 10

Issue

We migrated a group of guests but one guest failed to migrate properly. The instance is running on the new compute node but the nova database information still reflects the old compute node location for the instance.

Resolution

WARNING: Database manipulation is not supported by Red Hat without a support exception or without the assistance of Red Hat technical support. The following knowledge base solution shall be used as a reference only.

First of all, verify that the instance is correctly running on the new host. Access it e.g. via SSH and perform a few sanity checks to verify that the migration went through o.k.

Next, take a database backup of the nova and nova_api databases. Connect to one of the controller nodes and run:

mysqldump nova > nova.dump
mysqldump nova_api > nova_api.dump

To correct the Nova database inconsistency the following database commands need to be run on one of the controllers:

As a sanity check, run the following command to output the current information for the instance substituting in the <UUID>:

mysql nova -e "select node,host,uuid,vm_state,task_state,power_state from instances where uuid='<UUID>';"

Get the full hostname of the destination hypervisor (the hypervisor where the instance is currently running)

[stack@undercloud-7 ~]$ source overcloudrc
[stack@undercloud-7 ~]$ nova service-list | grep compute
| 56 | nova-compute     | overcloud-compute-0.localdomain    | nova     | enabled | up    | 2016-09-12T17:49:29.000000 | -               |

Update the instance to reflect the new compute node location substituting the <NEW COMPUTE HOSTNAME> as it appears in the output of nova service-list and the <UUID> of the instance:

mysql nova -e "update instances set host='<NEW COMPUTE HOSTNAME>',node='<NEW COMPUTE HOSTNAME>' where uuid='<UUID>' and deleted = 0;"

Note: It may also be needed to update the vm_state and task_state as well as the power_state, depending on the scenario

mysql nova -e "update instances set node='<NEW COMPUTE HOSTNAME>',host='<NEW COMPUTE HOSTNAME>',vm_state='active',task_state=NULL,power_state=1 where uuid='<UUID>' and deleted = 0;"

For example:

update instances set node='overcloud-compute-0.localdomain',host='overcloud-compute-0.localdomain',vm_state='active',task_state=NULL,power_state=1  where uuid='<UUID>' and deleted = 0;

Validate with the following database statement substituting in UUID for the instance in question to confirm change:

mysql nova -e "select node,host,uuid,vm_state,task_state,power_state from instances where uuid='<UUID>';"

Validate with the nova show command on the instance to confirm everything looks correct:

nova show <UUID>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments