OpenStack instance won't reboot/start with error: rbd python libraries not found

Issue

When trying to start/reboot an instance it fails even though it was previously running on that compute node.

The following will be seen in the nova-compute.log file during a hard reboot of the instance:

2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server [req-id id id - default default] Exception during message handling: RuntimeError: rbd python libraries not found
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 977, in decorated_function
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3282, in reboot_instance
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     self._set_instance_obj_error_state(context, instance)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3257, in reboot_instance
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     bad_volumes_callback=bad_volumes_callback)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2702, in reboot
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     block_device_info)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2796, in _hard_reboot
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     mdevs=mdevs)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5429, in _get_guest_xml
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     context, mdevs)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5219, in _get_guest_config
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     flavor, guest.os_type)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4030, in _get_guest_storage_config
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     inst_type)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3917, in _get_guest_disk_config
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     disk = self.image_backend.by_name(instance, name, image_type)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 1171, in by_name
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     return backend(instance=instance, disk_name=name)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 823, in __init__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     rbd_user=self.rbd_user)
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 127, in __init__
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server     raise RuntimeError(_('rbd python libraries not found'))
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server RuntimeError: rbd python libraries not found
2020-08-18 14:38:00.359 1 ERROR oslo_messaging.rpc.server