Down Container After Compute Reboot

Solution In Progress - Updated -

Issue

  • It was observed that some containers don't come UP after reboot of the compute overcloud-compute-0. It was manually made UP.

  • docker logs from down container is below:

[root@overcloud-compute-0 ~]# docker logs nova_virtlogd
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Deleting /etc/nova/migration/authorized_keys
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys
INFO:__main__:Deleting /etc/nova/migration/identity
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml
INFO:__main__:Deleting /etc/sasl2/libvirt.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config
INFO:__main__:Deleting /var/lib/nova/.ssh/config
INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD='/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ [[ -c /dev/kvm ]]
++ chmod 660 /dev/kvm
++ chown root:qemu /dev/kvm
+++ lsmod
+++ grep xenfs
++ [[ -n '' ]]
++ [[ ! -d /var/log/kolla/libvirt ]]
++ mkdir -p /var/log/kolla/libvirt
++ touch /var/log/kolla/libvirt/libvirtd.log
++ chmod 644 /var/log/kolla/libvirt/libvirtd.log
+++ stat -c %a /var/log/kolla/libvirt
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/libvirt
++ chmod 644 /var/log/kolla/libvirt/libvirtd.log
+ echo 'Running command: '\''/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'\'''
+ exec /usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf
Running command: '/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'
container_linux.go:235: starting container process caused "container init exited prematurely"
  • docker ps -a | grep nova_virtlogd show the following:
57809a048155        docker_registry:5000/osp13_containers-nova-libvirt:13.0-101                 "kolla_start"            21 hours ago        Exited (255) 9 minutes ago                       nova_virtlogd
  • journalctl -u dockerd-current returns the following:
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.847+0000: 20712: info : hostname: overcloud-compute-0
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.847+0000: 20712: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.859+0000: 20712: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:40 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:40.122+0000: 20713: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:40 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:40.124+0000: 20713: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
  • VMs are failing to start with error message similar to this one seen in /var/log/containers/nova/nova-compute.log:
Mar 19 01:41:34 gbucpcpt10 nova-compute: 2020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server [req-8f7148a5-cf50-46a3-bf87-ae979710c680 ecb808d5f6d945fdb80af802c44b422c 7e513cf7de3f45e2bcfefd328f3c2176 - default default] Exception during message handling: libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2690, in start_instance#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2660, in _power_on#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     block_device_info)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2951, in power_on#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2841, in _hard_reboot#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5663, in _create_domain_and_network#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5632, in _create_domain_and_network#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5567, in _create_domain#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.
...
Mar 19 01:41:34 gbucpcpt10 nova-compute: 2020-03-19 01:41:34.083 1 ERROR nova.virt.libvirt.driver [req-8f7148a5-cf50-46a3-bf87-ae979710c680 ecb808d5f6d945fdb80af802c44b422c 7e513cf7de3f45e2bcfefd328f3c2176 - default default] [instance: 0880b8c3-4271-4364-afb5-7030e9a3a97f] Failed to start libvirt guest

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In