openstack overcloud container image upload not completing successfully, getting timeout error

Solution Verified - Updated -

Environment

  • Red Hat Openstack Platform 12

Issue

  • Getting an error messages while uploading overcloud container images error :

    ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

Resolution

  • The issue occurs when trying to use a proxy with docker
  • Add local registry IP xx.xx.xx.xx in NO_PROXY under /etc/systemd/system/docker.service.d/http*proxy.conf or in case of https proxy then /etc/systemd/system/docker.service.d/https*proxy.conf

  • Edit file /etc/systemd/system/docker.service.d/http*proxy.conf

    [Service]    
Environment="HTTP_PROXY=http://<proxy.example.com:80/>" "NO_PROXY=XX.XX.XX.XX,localhost,127.0.0.1"  

     $ sudo systemctl daemon*reload
 $ sudo systemctl restart docker
 $ source stackrc
 $ sudo openstack overcloud container image upload   **config*file  /home/stack/local_registry_images.yaml   **verbose

Diagnostic Steps

  • Error received while uploading container image

    (undercloud) [stack@rhosp12-undercloud ~]$ sudo openstack overcloud container image upload   --config-file  /home/stack/local_registry_images.yaml   --verbose --debug

(..)

"POST /v1.24/images/registry.access.redhat.com/rhosp12/openstack-aodh-api:12.0-20171201.1/tag?repo=XX.XX.XX.XX%3A8787%2Frhosp12%2Fopenstack-aodh-api&tag=12.0-20171201.1&force=1 HTTP/1.1" 201 0
True
Looking for auth config
No auth config in memory - loading from filesystem
Trying paths: ['/root/.docker/config.json', '/root/.dockercfg']
No config file found
Looking for auth entry for '192.168.44.1:8787'
No entry found
No auth config found
"POST /v1.24/images/192.168.44.1:8787/rhosp12/openstack-aodh-api/push?tag=12.0-20171201.1 HTTP/1.1" 200 None
UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
  result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
  return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
  return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 66, in take_action
  uploader.upload()
File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 71, in upload
  uploader.upload_image(image_name, pull_source, push_destination)
File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 132, in upload_image
  tag=tag, stream=True)]
File "/usr/lib/python2.7/site-packages/docker/client.py", line 247, in _stream_helper
  data = reader.read(1)
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 324, in read
  flush_decoder = True
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
  self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 237, in _error_catcher
  raise ReadTimeoutError(self._pool, None, 'Read timed out.')
ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
clean_up UploadImage: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
  ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
  result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
  ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
  result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
  return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 137, in run
  return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 66, in take_action
  uploader.upload()
File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 71, in upload
  uploader.upload_image(image_name, pull_source, push_destination)
File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_uploader.py", line 132, in upload_image
  tag=tag, stream=True)]
File "/usr/lib/python2.7/site-packages/docker/client.py", line 247, in _stream_helper
  data = reader.read(1)
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 324, in read
  flush_decoder = True
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
  self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 237, in _error_catcher
  raise ReadTimeoutError(self._pool, None, 'Read timed out.')
ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

END return value: 1

    (undercloud) [stack@rhosp12-undercloud ~]$ sudo docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
XX.XX.XX.XX:8787/rhosp12/openstack-aodh-api            12.0-20171201.1     e670e4d2c421        5 weeks ago         677.7 MB
registry.access.redhat.com/rhosp12/openstack-aodh-api   12.0-20171201.1     e670e4d2c421        5 weeks ago         677.7 MB
registry.access.redhat.com/rhosp12/openstack-base       12.0-20171201.1     31f8a10ce7c9        5 weeks ago         325.6 MB
registry.access.redhat.com/rhosp12/openstack-base       latest              31f8a10ce7c9        5 weeks ago         325.6 MB
(undercloud) [stack@rhosp12-undercloud ~]$

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments