nova instances remain in BUILD state if qemu-img convert does not execute to completion in Red Hat OpenStack Platform
Issue
nova instances remain in BUILD state if qemu-img convert does not execute to completion in Red Hat OpenStack Platform. Attempts to reset and delete the instance fail.
When a nova instance is spawned and a new base image file is created, nova does not kill qemu-img convert if nova instance creation timeouts are hit.
Example output:
[stack@director ~]$ nova list --all-tenants --fields name,status,task_state,power_state,networks,image,host | grep -i build
(...)
| <UUID> | Test | BUILD | deleting | NOSTATE | net=10.10.10.7 | {u'id': u'<UUID>', u'links': [{u'href': u'https://<URL>:13774/images/<UUID>', u'rel': u'bookmark'}]} | compute-1.localdomain |
On the compute node, one can observe a conversion process running, even hours or days after the instance was initially started.
[heat-admin@compute-1 ~]$ ps aux | grep 'qemu-img convert'
heat-ad+ 298430 0.0 0.0 112660 976 pts/9 S+ 13:12 0:00 grep --color=auto qemu-img convert
nova 571213 0.0 0.0 589004 29556 ? S 2017 0:01 qemu-img convert -O raw /var/lib/nova/instances/_base/<UUID1>.part /var/lib/nova/instances/_base/<UUID1>.converted -f vmdk
nova 748856 0.0 0.0 589004 29576 ? S 2017 0:16 qemu-img convert -O raw /var/lib/nova/instances/_base/<UUID2>.part /var/lib/nova/instances/_base/<UUID2>.converted -f vmdk
One can also see incomplete _base files:
[heat-admin@compute-1 ~]$ sudo ls -alZ /var/lib/nova/instances/_base
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 .
drwxr-xr-x. nova nova system_u:object_r:nova_var_lib_t:s0 ..
-rw-r--r--. nova nova system_u:object_r:nova_var_lib_t:s0 <UUID1>.converted
-rw-r--r--. nova nova system_u:object_r:nova_var_lib_t:s0 <UUID1>.part
-rw-r--r--. nova nova system_u:object_r:nova_var_lib_t:s0 <UUID2>.converted
-rw-r--r--. nova nova system_u:object_r:nova_var_lib_t:s0 <UUID2>.part
Environment
Red Hat OpenStack Platform
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
