OpenStack Docker containers stuck in 'restarting' state

Solution Verified - Updated -

Issue

After running openstack overcloud deploy (...), most containers are in Restarting state:

19f836ead8ba sat:5000/osp-osp13_containers-neutron-sriov-agent:13.0-89 "kolla_start" 12 days ago Restarting (2) 26 minutes ago neutron_sriov_agent
b156ed0fbe14 sat:5000/osp-osp13_containers-nova-compute-hotfix:13.0-98-1703225 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_compute
44e83ede071d sat:5000/osp-osp13_containers-nova-compute-hotfix:13.0-98-1703225 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_migration_target
f851155eaafd sat:5000/osp-osp13_containers-cron:13.0-90 "kolla_start" 12 days ago Restarting (2) 26 minutes ago logrotate_crond
1e4426c5ead3 sat:5000/osp-osp13_containers-nova-libvirt:13.0-101 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_libvirt
e43197378dc1 sat:5000/osp-osp13_containers-nova-libvirt:13.0-101 "kolla_start" 12 days ago Restarting (2) 26 minutes ago nova_virtlogd

In the docker logs you see the following:

Jan 20 13:26:38 compute13 journal: INFO:__main__:Deleting /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/neutron/neutron.conf to /etc/neutron/neutron.conf
Jan 20 13:26:38 compute13 journal: ERROR:__main__:Unexpected error:
Jan 20 13:26:38 compute13 journal: Traceback (most recent call last):
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 411, in main
Jan 20 13:26:38 compute13 journal:    execute_config_strategy(config)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 377, in execute_config_strategy
Jan 20 13:26:38 compute13 journal:    copy_config(config)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 306, in copy_config
Jan 20 13:26:38 compute13 journal:    config_file.copy()
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 150, in copy
Jan 20 13:26:38 compute13 journal:    self._merge_directories(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 97, in _merge_directories
Jan 20 13:26:38 compute13 journal:    os.path.join(dest, to_copy))
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 92, in _merge_directories
Jan 20 13:26:38 compute13 journal:    self._set_properties(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 117, in _set_properties
Jan 20 13:26:38 compute13 journal:    self._set_properties_from_file(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/local/bin/kolla_set_configs", line 122, in _set_properties_from_file
Jan 20 13:26:38 compute13 journal:    shutil.copystat(source, dest)
Jan 20 13:26:38 compute13 journal:  File "/usr/lib64/python2.7/shutil.py", line 98, in copystat
Jan 20 13:26:38 compute13 journal:    os.utime(dst, (st.st_atime, st.st_mtime))
Jan 20 13:26:38 compute13 journal: OSError: [Errno 30] Read-only file system: '/etc/pki/ca-trust/extracted'

This issue seems to have been triggered after a change of hardware clock time.

Environment

  • Red Hat OpenStack 13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content