メモリーの大量に使用されることによる swift-proxy-server のクラッシュ

Issue

• glance にイメージをアップロードしたり、glance からイメージをダウンロードする際にメモリーが大量に使用され、swift-proxy-server プロセスが強制終了される。

Aug 19 23:05:00 overcloud-controller-1 kernel:[27063]   160 27063 11882671 11771173   23163        0             0 swift-proxy-ser
Aug 19 23:05:00 overcloud-controller-1 kernel:Out of memory:Kill process 27063 (swift-proxy-ser) score 717 or sacrifice child
Aug 19 23:05:00 overcloud-controller-1 kernel:Killed process 27063 (swift-proxy-ser) total-vm:47530684kB, anon-rss:47083632kB, file-rss:1060kB

• glance クライアントでエラーが発生する前後に、swift-proxy の 2 つ目のサブプロセスのメモリ消費量が増加します。

[root@overcloud-controller-0 swift]# ps awfux | grep swift-proxy
swift    13084  1.0  0.0 302536 35248 ?        Ss   03:36   0:01  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13160 35.3 20.0 13699376 13196652 ?   S    03:36   0:48  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    13167  3.9  2.0 1608432 1331140 ?     R    03:36   0:05  /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

一定時間経ると、メモリー消費量が減少します。

[root@overcloud-controller-0 swift]# ps awfux | grep swift-proxy
swift    13084  0.7  0.0 302536 31448 ?        Ss   03:36   0:04 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift    26553 58.8 52.5 34980264 34553208 ?   S    03:41   3:06 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
swift      455  0.0  0.0 302836 31556 ?        S    3:43   0:00 /usr/bin/python2 /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf

• glance のバックエンドに swift を使用しているときに大きなイメージをダウンロード、及びアップロードすると、以下のエラーが /var/log/glance/api.log ファイルに出力されるのはなぜですか?

 glance -d image-download aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa --file edge.iso --progress
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:5000/v2.0 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"GET /v2.0 HTTP/1.1" 200 335
DEBUG:keystoneclient.session:RESP:[200] Vary:X-Auth-Token Content-Type: application/json Content-Length:335 X-Openstack-Request-Id: req-8f0c1022-1eef-4bc1-8ad8-756bae12f921 Date:Sat, 20 Aug 2016 03:37:49 GMT
RESP BODY:{"version":{"status":"stable", "updated":"2014-04-17T00:00:00Z", "media-types":[{"base":"application/json", "type":"application/vnd.openstack.identity-v2.0+json"}], "id":"v2.0", "links":[{"href":"http://xx.xx.x.10:5000/v2.0/", "rel":"self"}, {"href":"http://docs.openstack.org/", "type":"text/html", "rel":"describedby"}]}}

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://xx.xx.x.10:5000/v2.0/tokens
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
DEBUG:keystoneclient.session:REQ: curl -g -i -X GET http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file -H "User-Agent: python-glanceclient" -H "Content-Type: application/octet-stream" -H "X-Auth-Token:{SHA1}2c63a209d37f798c3e98f97406f0fec4e2e67ed1"
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): xx.xx.x.10
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in main
    OpenStackImagesShell().main(argv)
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 282, in do_image_download
    body = gc.images.data(args.id)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 195, in data
    resp, body = self.http_client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 343, in request
    raise exc.CommunicationError(message=message)
CommunicationError:Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file:Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file
Error finding address for http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file:Unable to establish connection to http://xx.xx.x.10:9292/v2/images/aaaaaaaaa-aaaaa-aaa-aaaaa-aaaaaaaaaaaa/file

• glance api を使用せず、 swift を使用して直接イメージをアップロードしようとするとアップロードに失敗し、以下のエラーが発生します。

# swift --debug  upload test rundeck.qcow2 --segment-size 4043309056
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://11.0.2.10:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 3423
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1):xx.xx.x.10


DEBUG:requests.packages.urllib3.connectionpool:"PUT /v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 HTTP/1.1" 503 118
INFO:swiftclient:REQ: curl -i http://11.0.2.10:8080/v1/AUTH_xxxxxxxxxxxx14a3499e3490cde9d3051/test_segments/rundeck.qcow2/1471620435.000000/28124905472/4043309056/00000001 -X PUT -H "Content-Length:4043309056" -H "Content-Type:" -H "X-Auth-Token: c6890e8f2bdf4bea982520876225ed30"
INFO:swiftclient:RESP STATUS:503 Service Unavailable
INFO:swiftclient:RESP HEADERS:{u'Date': u'Sun, 21 Aug 2016 05:44:02 GMT', u'Content-Length': u'118', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'txccdde603043e460d8c92b-0057b93f66'}
INFO:swiftclient:RESP BODY:<html><h1>Service Unavailable</h1><p>The server is currently unavailable.Please try again at a later time.</p></html>