Down Container After Compute Reboot

Solution In Progress - Updated -

Issue

  • It was observed that some containers don't come UP after reboot of the compute overcloud-compute-0. It was manually made UP.

  • docker logs from down container is below:

[root@overcloud-compute-0 ~]# docker logs nova_virtlogd
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/libvirtd.conf to /etc/libvirt/libvirtd.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db
INFO:__main__:Deleting /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo.cnf
INFO:__main__:Deleting /etc/nova/migration/authorized_keys
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys
INFO:__main__:Deleting /etc/nova/migration/identity
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity
INFO:__main__:Deleting /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/secret.xml to /etc/nova/secret.xml
INFO:__main__:Deleting /etc/sasl2/libvirt.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config
INFO:__main__:Deleting /var/lib/nova/.ssh/config
INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config
INFO:__main__:Writing out command to execute
++ cat /run_command
+ CMD='/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'
+ ARGS=
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ [[ -c /dev/kvm ]]
++ chmod 660 /dev/kvm
++ chown root:qemu /dev/kvm
+++ lsmod
+++ grep xenfs
++ [[ -n '' ]]
++ [[ ! -d /var/log/kolla/libvirt ]]
++ mkdir -p /var/log/kolla/libvirt
++ touch /var/log/kolla/libvirt/libvirtd.log
++ chmod 644 /var/log/kolla/libvirt/libvirtd.log
+++ stat -c %a /var/log/kolla/libvirt
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/libvirt
++ chmod 644 /var/log/kolla/libvirt/libvirtd.log
+ echo 'Running command: '\''/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'\'''
+ exec /usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf
Running command: '/usr/sbin/virtlogd --config /etc/libvirt/virtlogd.conf'
container_linux.go:235: starting container process caused "container init exited prematurely"
  • docker ps -a | grep nova_virtlogd show the following:
57809a048155        docker_registry:5000/osp13_containers-nova-libvirt:13.0-101                 "kolla_start"            21 hours ago        Exited (255) 9 minutes ago                       nova_virtlogd
  • journalctl -u dockerd-current returns the following:
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.847+0000: 20712: info : hostname: overcloud-compute-0
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.847+0000: 20712: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:30 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:30.859+0000: 20712: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:40 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:40.122+0000: 20713: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
Mar 19 01:41:40 overcloud-compute-0 dockerd-current[12946]: 2020-03-18 22:41:40.124+0000: 20713: error : virNetSocketNewConnectUNIX:712 : Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory
  • VMs are failing to start with error message similar to this one seen in /var/log/containers/nova/nova-compute.log:
Mar 19 01:41:34 gbucpcpt10 nova-compute: 2020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server [req-8f7148a5-cf50-46a3-bf87-ae979710c680 ecb808d5f6d945fdb80af802c44b422c 7e513cf7de3f45e2bcfefd328f3c2176 - default default] Exception during message handling: libvirtError: Failed to connect socket to '/var/run/libvirt/virtlogd-sock': No such file or directory#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 186, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     "Error: %s", e, instance=instance)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 156, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1000, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2690, in start_instance#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2660, in _power_on#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     block_device_info)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2951, in power_on#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2841, in _hard_reboot#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5663, in _create_domain_and_network#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5632, in _create_domain_and_network#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5567, in _create_domain#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     self.force_reraise()#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit#0122020-03-19 01:41:34.370 1 ERROR oslo_messaging.rpc.
...
Mar 19 01:41:34 gbucpcpt10 nova-compute: 2020-03-19 01:41:34.083 1 ERROR nova.virt.libvirt.driver [req-8f7148a5-cf50-46a3-bf87-ae979710c680 ecb808d5f6d945fdb80af802c44b422c 7e513cf7de3f45e2bcfefd328f3c2176 - default default] [instance: 0880b8c3-4271-4364-afb5-7030e9a3a97f] Failed to start libvirt guest

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content