キーストアと統合した radosgw からアクセスすると libcurl エラーが発生し "server certificate verification failed" メッセージが表示される

Solution In Progress - Updated -

Environment

  • Red Hat Ceph Storage 1.2

  • Inktank Ceph Storage 1.2

Issue

  • radosgw アクセスがエラーになり "server certificate verification failed." メッセージが表示されます。

  • 以下はエラーログです。


2015-04-28 23:28:21.056244 7f14b2efc700 20 enqueued request req=0x7f1498011a30
2015-04-28 23:28:21.056268 7f14b2efc700 20 RGWWQ:
2015-04-28 23:28:21.056270 7f14b2efc700 20 req:0x7f1498011a30
2015-04-28 23:28:21.056289 7f14b2efc700 10 allocated request req=0x7f1498010650
2015-04-28 23:28:21.056335 7f14937f6700 20 dequeued request req=0x7f1498011a30
2015-04-28 23:28:21.056349 7f14937f6700 20 RGWWQ: empty
2015-04-28 23:28:21.056418 7f14937f6700 20 DOCUMENT_ROOT=/var/www
2015-04-28 23:28:21.056420 7f14937f6700 20 FCGI_ROLE=RESPONDER
2015-04-28 23:28:21.056422 7f14937f6700 20 GATEWAY_INTERFACE=CGI/1.1
2015-04-28 23:28:21.056423 7f14937f6700 20 HTTP_ACCEPT=*/*
2015-04-28 23:28:21.056424 7f14937f6700 20 HTTP_ACCEPT_ENCODING=gzip, deflate, compress
2015-04-28 23:28:21.056425 7f14937f6700 20 HTTP_AUTHORIZATION=
2015-04-28 23:28:21.056427 7f14937f6700 20 HTTP_HOST=a01:8080
2015-04-28 23:28:21.056428 7f14937f6700 20 HTTP_USER_AGENT=python-requests/2.2.1 CPython/2.7.3 Linux/3.13.0-49-generic
2015-04-28 23:28:21.056429 7f14937f6700 20 HTTP_X_AUTH_TOKEN=b61c2de41f98456cacd92c2fce4cab1f
2015-04-28 23:28:21.056430 7f14937f6700 20 PATH=/usr/local/bin:/usr/bin:/bin
2015-04-28 23:28:21.056431 7f14937f6700 20 QUERY_STRING=page=swift&params=/v1&format=json
2015-04-28 23:28:21.056432 7f14937f6700 20 REMOTE_ADDR=X.X.X.X
2015-04-28 23:28:21.056433 7f14937f6700 20 REMOTE_PORT=46023
2015-04-28 23:28:21.056434 7f14937f6700 20 REQUEST_METHOD=GET
2015-04-28 23:28:21.056435 7f14937f6700 20 REQUEST_URI=/swift/v1?format=json
2015-04-28 23:28:21.056437 7f14937f6700 20 SCRIPT_FILENAME=/var/www/s3gw.fcgi
2015-04-28 23:28:21.056438 7f14937f6700 20 SCRIPT_NAME=/swift/v1
2015-04-28 23:28:21.056439 7f14937f6700 20 SCRIPT_URI=http://a01:8080/swift/v1
2015-04-28 23:28:21.056440 7f14937f6700 20 SCRIPT_URL=/swift/v1
2015-04-28 23:28:21.056441 7f14937f6700 20 SERVER_ADDR=X.X.X.X
2015-04-28 23:28:21.056446 7f14937f6700 20 SERVER_ADMIN=
2015-04-28 23:28:21.056447 7f14937f6700 20 SERVER_NAME=
2015-04-28 23:28:21.056448 7f14937f6700 20 SERVER_PORT=8080
2015-04-28 23:28:21.056449 7f14937f6700 20 SERVER_PROTOCOL=HTTP/1.1
2015-04-28 23:28:21.056450 7f14937f6700 20 SERVER_SIGNATURE=
2015-04-28 23:28:21.056451 7f14937f6700 20 SERVER_SOFTWARE=Apache/2.2.22 (Ubuntu)
2015-04-28 23:28:21.056453 7f14937f6700  1 ====== starting new request req=0x7f1498011a30 =====
2015-04-28 23:28:21.056470 7f14937f6700  2 req 10:0.000018::GET /swift/v1::initializing
2015-04-28 23:28:21.056516 7f14937f6700 10 ver=v1 first= req=
2015-04-28 23:28:21.056519 7f14937f6700 10 s->object=<NULL> s->bucket=<NULL>
2015-04-28 23:28:21.056524 7f14937f6700  2 req 10:0.000071:swift:GET /swift/v1::getting op
2015-04-28 23:28:21.056528 7f14937f6700  2 req 10:0.000076:swift:GET /swift/v1:list_buckets:authorizing
2015-04-28 23:28:21.056537 7f14937f6700 20 token_id=b61c2de41f98456cacd92c2fce4cab1f
2015-04-28 23:28:21.056578 7f14937f6700 20 sending request to https://keystone-a01:35357/v2.0/tokens/b61c2de41f98456cacd92c2fce4cab1f
2015-04-28 23:28:21.248403 7f14937f6700  0 curl_easy_performed returned error: server certificate verification failed.CAfile:/etc/ssl/certs/ca-certificates.crt CRLfile: none
2015-04-28 23:28:21.248456 7f14937f6700 10 failed to authorize request
2015-04-28 23:28:21.248476 7f14937f6700  2 req 10:0.192024:swift:GET /swift/v1:list_buckets:http status=401
2015-04-28 23:28:21.248480 7f14937f6700  1 ====== req done req=0x7f1498011a30 http_status=401 ======
  • ceph.conf ファイルは以下のようになります。
# cat /etc/ceph/ceph.conf
[global]
fsid = cf54b912-080d-4475-a6d2-235215ee9e39
mon_initial_members = host-01, host-02, host-03
mon_host = X.X.X.X, Y.Y.Y.Y, Z.Z.Z.Z
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_journal_size = 20480
mon_osd_min_down_reporters = 5
debug rgw = 20

[osd]
osd_recovery_max_active = 1
osd_max_backfills = 1
osd_op_threads = 10
osd_mount_options_xfs = "rw,noatime,inode64,logbsize=256k,delaylog"

[client]
rbd_cache=true
rbd_cache_size=64
rbd cache writethrough until flush = true

[client.radosgw.gateway]
host = a01
keyring = /etc/ceph/keyring.radosgw.gateway
log_file = /var/log/ceph/radosgw.log
rgw_socket_path = /tmp/radosgw.sock
rgw keystone url = https://keystone-01:35357
rgw keystone admin token = 1c3h0us3dev
rgw keystone accepted roles = Member, admin, SwiftOperator
rgw keystone token cache size = 500
rgw keystone revocation interval = 600
nss db path = /var/lib/ceph/nss
  • curl を手動で使用するトークンと証明書にエラーがないことを確認します。
# curl -v -i -H "X-Auth-Token:47cac84312854bbb8c8bb5316ecb746b" https://keystone-01:35357/v2.0/tokens/47cac84312854bbb8c8bb5316ecb746b
* About to connect() to keystone-a01 port 35357 (#0)
*   Trying X.X.X.X... connected
* successfully set certificate verify locations:
......
.........
.............

Resolution

  • キーストアサーバーでは証明書の順番が重要です。

  • この問題は、証明書の順番が適切ではない場合に発生します。証明書の順番を変更して再度確認します。

  • gnutls では証明書の順番が決まっていて、順番を間違うとエラーが発生します。

Root Cause

  • 証明書の順番が適切ではありません。

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments