Why swift-proxy-server crashes due to large memory consumption ?

Issue

• When uploading/downloading large image to/from glance, process swift-proxy-server is getting killed due to large memory consumption

Aug 19 23:05:00 overcloud-controller-1 kernel: [27063]   160 27063 11882671 11771173   23163        0             0 swift-proxy-ser
Aug 19 23:05:00 overcloud-controller-1 kernel: Out of memory: Kill process 27063 (swift-proxy-ser) score 717 or sacrifice child
Aug 19 23:05:00 overcloud-controller-1 kernel: Killed process 27063 (swift-proxy-ser) total-vm:47530684kB, anon-rss:47083632kB, file-rss:1060kB

• Right around the time glance client error out, second sub-process of swift-proxy started to grow:

[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift    13084  1.0  0.0 302536 35248 ?        Ss   03:36   0:01  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13160 35.3 20.0 13699376 13196652 ?   S    03:36   0:48  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13167  3.9  2.0 1608432 1331140 ?     R    03:36   0:05  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

And after some time it settles

[root@overcloud-controller-0 swift]# ps awfux|grep swift-proxy
swift    13084  0.7  0.0 302536 31448 ?        Ss   03:36   0:04 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    26553 58.8 52.5 34980264 34553208 ?   S    03:41   3:06 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift      455  0.0  0.0 302836 31556 ?        S    03:43   0:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

• Why while using swift as glance backend, large image download/upload is failing with below error in /var/log/glance/api.log file ?

 glance -d image-download aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa --file edge.iso --progress
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0 HTTP/1.1" 200 335
DEBUG:keystoneclient.session:RESP: [200] Vary: X-Auth-Token Content-Type: application/json Content-Length: 335 X-Openstack-Request-Id: req-8f0c1022-1eef-4bc1-8ad8-756bae12f921 Date: Sat, 20 Aug 2016 03:37:49 GMT
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://xx.xx.x.10:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://xx.xx.x.10:5000/v2.0/tokens
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token: {SHA1}2c63a209d37f798c3e98f97406f0fec4e2e67ed1"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in main
    OpenStackImagesShell().main(argv)
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 282, in do_image_download
    body = gc.images.data(args.id)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 195, in data
    resp, body = self.http_client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 343, in request
    raise exc.CommunicationError(message=message)
CommunicationError: Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file: Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file
Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file: Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file

• While trying to upload the image directly using swift skipping glance api, the upload fails with the following error :

# swift --debug  upload test rundeck.qcow2 --segment-size 4043309056
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://11.0.2.10:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10


DEBUG:requests.packages.urllib3.connectionpool:"PUT /v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 HTTP/1.1" 503 118
INFO:swiftclient:REQ: curl -i http://11.0.2.10:8080/v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 -X PUT -H "Content-Length: 4043309056" -H "Content-Type: " -H "X-Auth-Token: c6890e8f2bdf4bea982520876225ed30"
INFO:swiftclient:RESP STATUS: 503 Service Unavailable
INFO:swiftclient:RESP HEADERS: {u'Date': u'Sun, 21 Aug 2016 05:44:02 GMT', u'Content-Length': u'118', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'txccdde603043e460d8c92b-0057b93f66'}
INFO:swiftclient:RESP BODY: <html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>