unable to create snapshot of instance. It stuck at 50% for a very long time.
Issue
- When SSL between glanceclient/haproxy/glance is enabled, unable to create snapshot of instance. It stuck at 50% for a very long time. Following is
nova image-showoutput:
[root@os-controller-1:~ (admin)]# nova image-show 1f2770a7-e53c-4063-ab74-5436503d122a
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size | 85899345920 |
| created | 2015-11-01T20:54:34Z |
| id | 1f2770a7-e53c-4063-ab74-5436503d122a |
| metadata base_image_ref | fa5a7158-0a95-4e95-aa6d-069eae2b1704 |
| metadata hypervisor_type | qemu |
| metadata image_location | snapshot |
| metadata image_state | available |
| metadata image_type | snapshot |
| metadata instance_type_ephemeral_gb | 0 |
| metadata instance_type_flavorid | 9f58c64e-3e29-49f2-99c3-ee01d5c7af1b |
| metadata instance_type_id | 428 |
| metadata instance_type_memory_mb | 4096 |
| metadata instance_type_name | test-vm |
| metadata instance_type_root_gb | 80 |
| metadata instance_type_rxtx_factor | 1.0 |
| metadata instance_type_swap | 0 |
| metadata instance_type_vcpus | 2 |
| metadata instance_uuid | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| metadata network_allocated | True |
| metadata owner_id | 1598072861a84a559b5033370558d2d2 |
| metadata user_id | 6f8d994ca73a47469ac69d9d84c8f82d |
| minDisk | 80 |
| minRam | 0 |
| name | test_snap |
| progress | 50 |
| server | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| status | SAVING | <===
| updated | 2015-11-01T20:57:37Z |
+-------------------------------------+--------------------------------------+
- From
compute.log:
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 374, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher instance=instance)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
106092: TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 364, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3021, in snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3052, in _snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher update_task_state)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1753, in snapshot
106092: TRACE oslo.messaging.rpc.dispatcher image_file)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 126, in update
106092: TRACE oslo.messaging.rpc.dispatcher purge_props=purge_props)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 405, in update
106092: TRACE oslo.messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 403, in update
106092: TRACE oslo.messaging.rpc.dispatcher image_id, **image_meta)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 232, in call
106092: TRACE oslo.messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 329, in update
106092: TRACE oslo.messaging.rpc.dispatcher resp, body = self.client.put(url, headers=hdrs, data=image_data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 259, in put
106092: TRACE oslo.messaging.rpc.dispatcher return self._request('PUT', url, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 200, in _request
106092: TRACE oslo.messaging.rpc.dispatcher **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
106092: TRACE oslo.messaging.rpc.dispatcher resp = self.send(prep, **send_kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
106092: TRACE oslo.messaging.rpc.dispatcher r = adapter.send(request, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 351, in send
106092: TRACE oslo.messaging.rpc.dispatcher low_conn.send(b'\r\n')
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/httplib.py", line 805, in send
106092: TRACE oslo.messaging.rpc.dispatcher self.sock.sendall(data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in sendall
106092: TRACE oslo.messaging.rpc.dispatcher tail = self.send(data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in write
106092: TRACE oslo.messaging.rpc.dispatcher return self.fd.write(data)
106092: TRACE oslo.messaging.rpc.dispatcher SysCallError: (104, 'Connection reset by peer')
- From
haproxylogs:
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.165:35928 [10/Nov/2015:12:28:24.287] ceilometer_api_cluster ceilome
ter_api_cluster/<NOSRV> -1/-1/0 187 PR 2125/0/0/0/3 0/0
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.72:59614 [10/Nov/2015:12:28:24.183] neutron_api_cluster~ neutron_ap
i_cluster/os-controller-2 0/0/122 3128 -- 2125/1/0/1/0 0/0
Nov 10 12:28:45 os-controller-1.waed.net haproxy[19858]: 172.16.5.69:44501 [10/Nov/2015:12:14:05.287] glance_api_cluster~ glance_api_cluster/os-controller-2 74/1/879971 0 CD 2120/2031/2031/711/0 0/0 <<<<--------
haproxyconfiguration:
[...]
listen glance_api_cluster
bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem <==
bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem <==
balance source
mode tcp
option tcpka
option tcplog
option forwardfor
timeout client 120m
timeout server 120m
server os-controller-1 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
server os-controller-2 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
# server os-controller-3 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
listen glance_registry_cluster
bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem <==
bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem <==
balance roundrobin
http-check expect ! rstatus ^5
mode http
option tcpka
option forwardfor
option tcplog
option httpchk
timeout client 120m
timeout server 120m
server os-controller-1 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
server os-controller-2 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
server os-controller-3 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
[...]
- glance-api.conf:
[keystone_authtoken]
#identity_uri=http://127.0.0.1:35357
identity_uri=https://xx.xx.xx.xx:35357
#admin_tenant_name=%SERVICE_TENANT_NAME%
admin_tenant_name=services
#admin_user=%SERVICE_USER%
admin_user=glance
#admin_password=%SERVICE_PASSWORD%
admin_password=xxxxxx
#revocation_cache_time=10
auth_port=35357
auth_host=xx.xx.xx.xx
auth_protocol=https
auth_uri=https://xx.xx.xx.xx:5000/
Environment
- Red Hat Open Stack
- openssl-1.0.1e-34.el7_0.7.x86_64
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
