unable to create snapshot of instance. It stuck at 50% for a very long time.

Solution Unverified - Updated -

Issue

  • When SSL between glanceclient/haproxy/glance is enabled, unable to create snapshot of instance. It stuck at 50% for a very long time. Following is nova image-show output:
[root@os-controller-1:~ (admin)]# nova image-show 1f2770a7-e53c-4063-ab74-5436503d122a
+-------------------------------------+--------------------------------------+
| Property                            | Value                                |
+-------------------------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size                | 85899345920                          |
| created                             | 2015-11-01T20:54:34Z                 |
| id                                  | 1f2770a7-e53c-4063-ab74-5436503d122a |
| metadata base_image_ref             | fa5a7158-0a95-4e95-aa6d-069eae2b1704 |
| metadata hypervisor_type            | qemu                                 |
| metadata image_location             | snapshot                             |
| metadata image_state                | available                            |
| metadata image_type                 | snapshot                             |
| metadata instance_type_ephemeral_gb | 0                                    |
| metadata instance_type_flavorid     | 9f58c64e-3e29-49f2-99c3-ee01d5c7af1b |
| metadata instance_type_id           | 428                                  |
| metadata instance_type_memory_mb    | 4096                                 |
| metadata instance_type_name         | test-vm                              |
| metadata instance_type_root_gb      | 80                                   |
| metadata instance_type_rxtx_factor  | 1.0                                  |
| metadata instance_type_swap         | 0                                    |
| metadata instance_type_vcpus        | 2                                    |
| metadata instance_uuid              | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| metadata network_allocated          | True                                 |
| metadata owner_id                   | 1598072861a84a559b5033370558d2d2     |
| metadata user_id                    | 6f8d994ca73a47469ac69d9d84c8f82d     |
| minDisk                             | 80                                   |
| minRam                              | 0                                    |
| name                                | test_snap                            |
| progress                            | 50                                   |
| server                              | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| status                              | SAVING                               | <===
| updated                             | 2015-11-01T20:57:37Z                 |
+-------------------------------------+--------------------------------------+
  • From compute.log:
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 374, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher     instance=instance)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
106092: TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 364, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher     *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3021, in snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3052, in _snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher     update_task_state)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1753, in snapshot
106092: TRACE oslo.messaging.rpc.dispatcher     image_file)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 126, in update
106092: TRACE oslo.messaging.rpc.dispatcher     purge_props=purge_props)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 405, in update
106092: TRACE oslo.messaging.rpc.dispatcher     _reraise_translated_image_exception(image_id)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 403, in update
106092: TRACE oslo.messaging.rpc.dispatcher     image_id, **image_meta)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 232, in call
106092: TRACE oslo.messaging.rpc.dispatcher     return getattr(client.images, method)(*args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 329, in update
106092: TRACE oslo.messaging.rpc.dispatcher     resp, body = self.client.put(url, headers=hdrs, data=image_data)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 259, in put
106092: TRACE oslo.messaging.rpc.dispatcher     return self._request('PUT', url, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 200, in _request
106092: TRACE oslo.messaging.rpc.dispatcher     **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
106092: TRACE oslo.messaging.rpc.dispatcher     resp = self.send(prep, **send_kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
106092: TRACE oslo.messaging.rpc.dispatcher     r = adapter.send(request, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 351, in send
106092: TRACE oslo.messaging.rpc.dispatcher     low_conn.send(b'\r\n')
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/httplib.py", line 805, in send
106092: TRACE oslo.messaging.rpc.dispatcher     self.sock.sendall(data)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in sendall
106092: TRACE oslo.messaging.rpc.dispatcher     tail = self.send(data)
106092: TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in write
106092: TRACE oslo.messaging.rpc.dispatcher     return self.fd.write(data)
106092: TRACE oslo.messaging.rpc.dispatcher SysCallError: (104, 'Connection reset by peer')
  • From haproxy logs:
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.165:35928 [10/Nov/2015:12:28:24.287] ceilometer_api_cluster ceilome
ter_api_cluster/<NOSRV> -1/-1/0 187 PR 2125/0/0/0/3 0/0
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.72:59614 [10/Nov/2015:12:28:24.183] neutron_api_cluster~ neutron_ap
i_cluster/os-controller-2 0/0/122 3128 -- 2125/1/0/1/0 0/0
Nov 10 12:28:45 os-controller-1.waed.net haproxy[19858]: 172.16.5.69:44501 [10/Nov/2015:12:14:05.287] glance_api_cluster~ glance_api_cluster/os-controller-2 74/1/879971 0 CD 2120/2031/2031/711/0 0/0  <<<<--------
  • haproxy configuration:
[...]
listen glance_api_cluster
  bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem  <==
  bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem  <==
  balance  source
  mode  tcp
  option  tcpka
  option  tcplog
  option  forwardfor
  timeout client  120m
  timeout server  120m
  server os-controller-1 xx.xx.xx.xx:9292  check inter 2000 rise 2 fall 5
  server os-controller-2 xx.xx.xx.xx:9292  check inter 2000 rise 2 fall 5
  # server os-controller-3 xx.xx.xx.xx:9292  check inter 2000 rise 2 fall 5

listen glance_registry_cluster
  bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem  <==
  bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem  <==
  balance  roundrobin
  http-check  expect ! rstatus ^5
  mode  http
  option  tcpka
  option  forwardfor
  option  tcplog
  option  httpchk
  timeout client  120m
  timeout server  120m
  server os-controller-1 xx.xx.xx.xx:9191  check inter 2000 rise 2 fall 5
  server os-controller-2 xx.xx.xx.xx:9191  check inter 2000 rise 2 fall 5
  server os-controller-3 xx.xx.xx.xx:9191  check inter 2000 rise 2 fall 5
[...]
  • glance-api.conf:
[keystone_authtoken]
#identity_uri=http://127.0.0.1:35357
identity_uri=https://xx.xx.xx.xx:35357
#admin_tenant_name=%SERVICE_TENANT_NAME%
admin_tenant_name=services
#admin_user=%SERVICE_USER%
admin_user=glance
#admin_password=%SERVICE_PASSWORD%
admin_password=xxxxxx
#revocation_cache_time=10
auth_port=35357
auth_host=xx.xx.xx.xx
auth_protocol=https
auth_uri=https://xx.xx.xx.xx:5000/

Environment

  • Red Hat Open Stack
  • openssl-1.0.1e-34.el7_0.7.x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content