Live-migration of a VM fails and VM stays in 'NOSTATE' with erratic info in binding_profile.

Issue

• We were live-migrating some VMs around to swap local hard drives on some computes.

• It worked fine for about 8-9 VMs and then we had a failure. This VM was no different than the others but the migration failed and the VM stays in 'NOSTATE' with some strange output in its binding_profile.

• The following error is seen in /var/log/containers/nova/nova-compute.log:

2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [req-f0905fc1-ec77-47ec-a4ce-4324718cec3a 6828abf50d114f7cad181a7402571511 cdfe5b92c11d49fc86d242448d99f8cc - default default] [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] Post live
 migration at destination belmont1-compute013.localdomain failed: NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] Traceback (most recent call last):
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6384, in _post_live_migration
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     instance, block_migration, dest)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 783, in post_live_migration_at_destination
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     instance=instance, block_migration=block_migration)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     retry=self.retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     timeout=timeout, retry=retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     retry=retry)
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901]     raise result
2019-08-14 12:55:18.671 1 ERROR nova.compute.manager [instance: 229f9217-67ed-4ef5-bcd8-796fa89f3901] NeutronAdminCredentialConfigurationInvalid_Remote: Networking client is experiencing an unauthorized exception.

• The VM is still listed as being hosted on the source compute but is now running perfectly on the destination compute.

• The live migration appears to have succeeded in the live migration list:

(overcloud) [stack@director ~]$ nova migration-list |grep 229f9217-67ed-4ef5-bcd8-796fa89f3901
| 99  | 5ec73d0c-c23f-4e10-b6c2-46c8c77ffe27 | -                                               | -                                               | source_compute.localdomain | -                               | -         | error     | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62         | 62         | 2019-08-14T17:34:45.000000 | 2019-08-14T17:34:45.000000 | live-migration |
| 96  | cc6dd42a-dfd0-434b-a06b-b23fe594fbd3 | -                                               | -                                               | source_compute.localdomain | -                               | -         | error     | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62         | 62         | 2019-08-14T17:34:23.000000 | 2019-08-14T17:34:24.000000 | live-migration |
| 54  | ee776084-bdd0-46ac-8adf-3ee4fc9423bd | source_compute.localdomain | destination_compute.localdomain | source_compute.localdomain | destination_compute.localdomain | -         | completed | 229f9217-67ed-4ef5-bcd8-796fa89f3901 | 62         | 62         | 2019-08-14T12:49:42.000000 | 2019-08-14T12:55:22.000000 | live-migration |