Live-migration of a VM fails and VM stays in 'NOSTATE' with erratic info in binding_profile.
Issue
-
We were live-migrating some VMs around to swap local hard drives on some computes.
-
It worked fine for about 8-9 VMs and then we had a failure. This VM was no different than the others but the migration failed and the VM stays in 'NOSTATE' with some strange output in its binding_profile.
-
The following error is seen in
/var/log/containers/nova/nova-compute.log
:
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [req-f0905fc1-ec77-47ec-a4ce-4324718cec3a 6828abf50d114f7cad181a7402571511 cdfe5b92c11d49fc86d242448d99f8cc - default default] [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] Post live
migration at destination belmont1-compute013.localdomain failed: NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] Traceback (most recent call last):
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6384, in _post_live_migration
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] instance, block_migration, dest)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] instance=instance, block_migration=block_migration)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] retry=self.retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] timeout=timeout, retry=retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] retry=retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] raise result
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.
-
The VM is still listed as being hosted on the source compute but is now running perfectly on the destination compute.
-
The live migration appears to have succeeded in the live migration list:
(overcloud) [stack@director ~]$ nova migration-list |grep 229f9217-67ed-4ef5-bcd8-796fa89f3901
| 99 | 5ec73d0c-c23f-4e10-b6c2-46c8c77ffe27 | - | - | source_compute.localdomain | - | - | error | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62 | 62 | 2019-08-14T17:34:45.000000 | 2019-08-14T17:34:45.000000 | live-migration |
| 96 | cc6dd42a-dfd0-434b-a06b-b23fe594fbd3 | - | - | source_compute.localdomain | - | - | error | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62 | 62 | 2019-08-14T17:34:23.000000 | 2019-08-14T17:34:24.000000 | live-migration |
| 54 | ee776084-bdd0-46ac-8adf-3ee4fc9423bd | source_compute.localdomain | destination_compute.localdomain | source_compute.localdomain | destination_compute.localdomain | - | completed | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62 | 62 | 2019-08-14T12:49:42.000000 | 2019-08-14T12:55:22.000000 | live-migration |
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.