Why swift-proxy-server crashes due to large memory consumption ?
Issue
- When uploading/downloading large image to/from
glance, processswift-proxy-serveris getting killed due to large memory consumption
Aug 19 23:05:00 overcloud-controller-1 kernel: [27063] 160 27063 11882671 11771173 23163 0 0 swift-proxy-ser
Aug 19 23:05:00 overcloud-controller-1 kernel: Out of memory: Kill process 27063 (swift-proxy-ser) score 717 or sacrifice child
Aug 19 23:05:00 overcloud-controller-1 kernel: Killed process 27063 (swift-proxy-ser) total-vm:47530684kB, anon-rss:47083632kB, file-rss:1060kB
- Right around the time glance client error out, second sub-process of swift-proxy started to grow:
[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift 13084 1.0 0.0 302536 35248 ? Ss 03:36 0:01 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 13160 35.3 20.0 13699376 13196652 ? S 03:36 0:48 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 13167 3.9 2.0 1608432 1331140 ? R 03:36 0:05 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
And after some time it settles
[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift 13084 0.7 0.0 302536 31448 ? Ss 03:36 0:04 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 26553 58.8 52.5 34980264 34553208 ? S 03:41 3:06 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift 455 0.0 0.0 302836 31556 ? S 03:43 0:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
- Why while using
swiftasglancebackend, large image download/upload is failing with below error in/var/log/glance/api.logfile ?
glance -d image-download aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa --file edge.iso --progress
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0 HTTP/1.1" 200 335
DEBUG:keystoneclient.session:RESP: [200] Vary: X-Auth-Token Content-Type: application/json Content-Length: 335 X-Openstack-Request-Id: req-8f0c1022-1eef-4bc1-8ad8-756bae12f921 Date: Sat, 20 Aug 2016 03:37:49 GMT
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://xx.xx.x.10:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://xx.xx.x.10:5000/v2.0/tokens
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token: {SHA1}2c63a209d37f798c3e98f97406f0fec4e2e67ed1"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in main
OpenStackImagesShell().main(argv)
File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in main
args.func(client, args)
File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 282, in do_image_download
body = gc.images.data(args.id)
File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 195, in data
resp, body = self.http_client.get(url)
File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 343, in request
raise exc.CommunicationError(message=message)
CommunicationError: Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file: Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file
Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file: Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file
- While trying to upload the image directly using
swiftskippingglance api, the upload fails with the following error :
# swift --debug upload test rundeck.qcow2 --segment-size 4043309056
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://11.0.2.10:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"PUT /v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 HTTP/1.1" 503 118
INFO:swiftclient:REQ: curl -i http://11.0.2.10:8080/v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 -X PUT -H "Content-Length: 4043309056" -H "Content-Type: " -H "X-Auth-Token: c6890e8f2bdf4bea982520876225ed30"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Sun, 21 Aug 2016 05:44:02 GMT', u'Content-Length': u'118', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'txccdde603043e460d8c92b-0057b93f66'}
INFO:swiftclient:RESP BODY: <html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>
Environment
- Red Hat OpenStack Platform 8.0
- Red Hat OpenStack Platform 9.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
