OpenStack Docker containers stuck in 'restarting' state

Solution Verified - Updated -

Issue

After running openstack overcloud deploy (...), most containers are in Restarting state:

19f836ead8ba sat:5000/osp-osp13_containers-neutron-sriov-agent:13.0-89 "kolla_start" 12 days ago Restarting (2) 26 minutes ago neutron_sriov_agent
b156ed0fbe14 sat:5000/osp-osp13_containers-nova-compute-hotfix:13.0-98-1703225 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_compute
44e83ede071d sat:5000/osp-osp13_containers-nova-compute-hotfix:13.0-98-1703225 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_migration_target
f851155eaafd sat:5000/osp-osp13_containers-cron:13.0-90 "kolla_start" 12 days ago Restarting (2) 26 minutes ago logrotate_crond
1e4426c5ead3 sat:5000/osp-osp13_containers-nova-libvirt:13.0-101 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_libvirt
e43197378dc1 sat:5000/osp-osp13_containers-nova-libvirt:13.0-101 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_virtlogd

In the docker logs you see the following:

Jan 20 13:26:38 compute13 journal: INFO:__main__:Deleting /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: ERROR:__main__:Unexpected error:
Jan 20 13:26:38 compute13 journal: Traceback (most recent call last):
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 411, in main
Jan 20 13:26:38 compute13 journal:    execute_config_strategy(config)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Jan 20 13:26:38 compute13 journal:    copy_config(config)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Jan 20 13:26:38 compute13 journal:    config_file.copy()
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 150, in copy
Jan 20 13:26:38 compute13 journal:    self._merge_directories(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories
Jan 20 13:26:38 compute13 journal:    self._set_properties(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties
Jan 20 13:26:38 compute13 journal:    self._set_properties_from_file(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file
Jan 20 13:26:38 compute13 journal:    shutil.copystat(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
Jan 20 13:26:38 compute13 journal:    os.utime(dst, (st.st_atime, st.st_mtime))
Jan 20 13:26:38 compute13 journal: OSError: [Errno 30] Read-only file system: '/etc/pki/ca-trust/extracted'

This issue seems to have been triggered after a change of hardware clock time.

Environment

  • Red Hat OpenStack 13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In