unable to create snapshot of instance. It stuck at 50% for a very long time.
Issue
- When SSL between glanceclient/haproxy/glance is enabled, unable to create snapshot of instance. It stuck at 50% for a very long time. Following is
nova image-show
output:
[root@os-controller-1:~ (admin)]# nova image-show 1f2770a7-e53c-4063-ab74-5436503d122a
+-------------------------------------+--------------------------------------+
| Property | Value |
+-------------------------------------+--------------------------------------+
| OS-EXT-IMG-SIZE:size | 85899345920 |
| created | 2015-11-01T20:54:34Z |
| id | 1f2770a7-e53c-4063-ab74-5436503d122a |
| metadata base_image_ref | fa5a7158-0a95-4e95-aa6d-069eae2b1704 |
| metadata hypervisor_type | qemu |
| metadata image_location | snapshot |
| metadata image_state | available |
| metadata image_type | snapshot |
| metadata instance_type_ephemeral_gb | 0 |
| metadata instance_type_flavorid | 9f58c64e-3e29-49f2-99c3-ee01d5c7af1b |
| metadata instance_type_id | 428 |
| metadata instance_type_memory_mb | 4096 |
| metadata instance_type_name | test-vm |
| metadata instance_type_root_gb | 80 |
| metadata instance_type_rxtx_factor | 1.0 |
| metadata instance_type_swap | 0 |
| metadata instance_type_vcpus | 2 |
| metadata instance_uuid | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| metadata network_allocated | True |
| metadata owner_id | 1598072861a84a559b5033370558d2d2 |
| metadata user_id | 6f8d994ca73a47469ac69d9d84c8f82d |
| minDisk | 80 |
| minRam | 0 |
| name | test_snap |
| progress | 50 |
| server | d91c0c90-2849-4965-abae-35e6217ee9a6 |
| status | SAVING | <===
| updated | 2015-11-01T20:57:37Z |
+-------------------------------------+--------------------------------------+
- From
compute.log
:
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 314, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 374, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher instance=instance)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
106092: TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 364, in decorated_function
106092: TRACE oslo.messaging.rpc.dispatcher *args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3021, in snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3052, in _snapshot_instance
106092: TRACE oslo.messaging.rpc.dispatcher update_task_state)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1753, in snapshot
106092: TRACE oslo.messaging.rpc.dispatcher image_file)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 126, in update
106092: TRACE oslo.messaging.rpc.dispatcher purge_props=purge_props)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 405, in update
106092: TRACE oslo.messaging.rpc.dispatcher _reraise_translated_image_exception(image_id)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 403, in update
106092: TRACE oslo.messaging.rpc.dispatcher image_id, **image_meta)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 232, in call
106092: TRACE oslo.messaging.rpc.dispatcher return getattr(client.images, method)(*args, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 329, in update
106092: TRACE oslo.messaging.rpc.dispatcher resp, body = self.client.put(url, headers=hdrs, data=image_data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 259, in put
106092: TRACE oslo.messaging.rpc.dispatcher return self._request('PUT', url, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 200, in _request
106092: TRACE oslo.messaging.rpc.dispatcher **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
106092: TRACE oslo.messaging.rpc.dispatcher resp = self.send(prep, **send_kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
106092: TRACE oslo.messaging.rpc.dispatcher r = adapter.send(request, **kwargs)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 351, in send
106092: TRACE oslo.messaging.rpc.dispatcher low_conn.send(b'\r\n')
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.7/httplib.py", line 805, in send
106092: TRACE oslo.messaging.rpc.dispatcher self.sock.sendall(data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 101, in sendall
106092: TRACE oslo.messaging.rpc.dispatcher tail = self.send(data)
106092: TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/eventlet/green/OpenSSL/SSL.py", line 81, in write
106092: TRACE oslo.messaging.rpc.dispatcher return self.fd.write(data)
106092: TRACE oslo.messaging.rpc.dispatcher SysCallError: (104, 'Connection reset by peer')
- From
haproxy
logs:
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.165:35928 [10/Nov/2015:12:28:24.287] ceilometer_api_cluster ceilome
ter_api_cluster/<NOSRV> -1/-1/0 187 PR 2125/0/0/0/3 0/0
Nov 10 12:28:24 os-controller-1.waed.net haproxy[19858]: 172.16.5.72:59614 [10/Nov/2015:12:28:24.183] neutron_api_cluster~ neutron_ap
i_cluster/os-controller-2 0/0/122 3128 -- 2125/1/0/1/0 0/0
Nov 10 12:28:45 os-controller-1.waed.net haproxy[19858]: 172.16.5.69:44501 [10/Nov/2015:12:14:05.287] glance_api_cluster~ glance_api_cluster/os-controller-2 74/1/879971 0 CD 2120/2031/2031/711/0 0/0 <<<<--------
haproxy
configuration:
[...]
listen glance_api_cluster
bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem <==
bind xx.xx.xx.xx:9292 ssl crt /etc/ssl/certs/haproxy.pem <==
balance source
mode tcp
option tcpka
option tcplog
option forwardfor
timeout client 120m
timeout server 120m
server os-controller-1 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
server os-controller-2 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
# server os-controller-3 xx.xx.xx.xx:9292 check inter 2000 rise 2 fall 5
listen glance_registry_cluster
bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem <==
bind xx.xx.xx.xx:9191 ssl crt /etc/ssl/certs/haproxy.pem <==
balance roundrobin
http-check expect ! rstatus ^5
mode http
option tcpka
option forwardfor
option tcplog
option httpchk
timeout client 120m
timeout server 120m
server os-controller-1 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
server os-controller-2 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
server os-controller-3 xx.xx.xx.xx:9191 check inter 2000 rise 2 fall 5
[...]
- glance-api.conf:
[keystone_authtoken]
#identity_uri=http://127.0.0.1:35357
identity_uri=https://xx.xx.xx.xx:35357
#admin_tenant_name=%SERVICE_TENANT_NAME%
admin_tenant_name=services
#admin_user=%SERVICE_USER%
admin_user=glance
#admin_password=%SERVICE_PASSWORD%
admin_password=xxxxxx
#revocation_cache_time=10
auth_port=35357
auth_host=xx.xx.xx.xx
auth_protocol=https
auth_uri=https://xx.xx.xx.xx:5000/
Environment
- Red Hat Open Stack
- openssl-1.0.1e-34.el7_0.7.x86_64
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.