-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for Red Hat OpenStack Platform
10.12. Configure Object Storage features
10.12.1. Object Storage zones
10.12.2. RAID controller configuration
10.12.3. Throttle resources through rate limits
10.12.3.1. Configure rate limiting
Table 10.67. Description of configuration options for [filter-ratelimit]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
account_blacklist = c,d
|
Comma separated lists of account names that will not be allowed. Returns a 497 response. r: for containers of size x, limit requests per second to r. Will limit PUT, DELETE, and POST requests to /a/c/o. container_listing_ratelimit_x = r: for containers of size x, limit listing requests per second to r. Will limit GET requests to /a/c. |
account_ratelimit = 0
|
If set, will limit PUT and DELETE requests to /account_name/container_name. Number is in requests per second. |
account_whitelist = a,b
|
Comma separated lists of account names that will not be rate limited. |
clock_accuracy = 1000
|
Represents how accurate the proxy servers' system clocks are with each other. 1000 means that all the proxies' clock are accurate to each other within 1 millisecond. No ratelimit should be higher than the clock accuracy. |
container_listing_ratelimit_0 = 100
|
No help text available for this option. |
container_listing_ratelimit_10 = 50
|
No help text available for this option. |
container_listing_ratelimit_50 = 20
|
No help text available for this option. |
container_ratelimit_0 = 100
|
No help text available for this option. |
container_ratelimit_10 = 50
|
No help text available for this option. |
container_ratelimit_50 = 20
|
No help text available for this option. |
log_sleep_time_seconds = 0
|
To allow visibility into rate limiting set this value > 0 and all sleeps greater than the number will be logged. |
max_sleep_time_seconds = 60
|
App will immediately return a 498 response if the necessary sleep time ever exceeds the given max_sleep_time_seconds. |
rate_buffer_seconds = 5
|
Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy. |
set log_address = /dev/log
|
Location where syslog sends the logs to |
set log_facility = LOG_LOCAL0
|
Syslog log facility |
set log_headers = false
|
If True, log headers in each request |
set log_level = INFO
|
Log level |
set log_name = ratelimit
|
Label to use when logging |
use = egg:swift#ratelimit
|
Entry point of paste.deploy in the server |
with container_limit_x = r
|
No help text available for this option. |
Table 10.68. Values for Rate Limiting with Sample Configuration Settings
Container Size | Rate Limit |
0-99 | No limiting |
100 | 100 |
150 | 75 |
500 | 20 |
1000 | 20 |
10.12.4. Health check
/healthcheck
, it responds with OK
in the response body, which monitoring tools can use.
Table 10.69. Description of configuration options for [filter-healthcheck]
in account-server.conf
Configuration option = Default value | Description |
---|---|
disable_path =
|
No help text available for this option. |
use = egg:swift#healthcheck
|
Entry point of paste.deploy in the server |
10.12.5. Domain remap
Table 10.70. Description of configuration options for [filter-domain_remap]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
default_reseller_prefix =
|
No help text available for this option. |
path_root = v1
|
Root path |
reseller_prefixes = AUTH
|
Reseller prefix |
set log_address = /dev/log
|
Location where syslog sends the logs to |
set log_facility = LOG_LOCAL0
|
Syslog log facility |
set log_headers = false
|
If True, log headers in each request |
set log_level = INFO
|
Log level |
set log_name = domain_remap
|
Label to use when logging |
storage_domain = example.com
|
Domain that matches your cloud. Multiple domains can be specified using a comma-separated list. |
use = egg:swift#domain_remap
|
Entry point of paste.deploy in the server |
10.12.6. CNAME lookup
storage_domain
by looking up the given domain's CNAME record in DNS.
Table 10.71. Description of configuration options for [filter-cname_lookup]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
lookup_depth = 1
|
Because CNAMES can be recursive, specifies the number of levels through which to search. |
set log_address = /dev/log
|
Location where syslog sends the logs to |
set log_facility = LOG_LOCAL0
|
Syslog log facility |
set log_headers = false
|
If True, log headers in each request |
set log_level = INFO
|
Log level |
set log_name = cname_lookup
|
Label to use when logging |
storage_domain = example.com
|
Domain that matches your cloud. Multiple domains can be specified using a comma-separated list. |
use = egg:swift#cname_lookup
|
Entry point of paste.deploy in the server |
10.12.7. Temporary URL
temp_url_sig
- A cryptographic signature
temp_url_expires
- An expiration date, in Unix time
https://swift-cluster.example.com/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object? temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=1323479485
X-Account-Meta-Temp-URL-Key
header on your Object Storage account to an arbitrary string. This string serves as a secret key. For example, to set a key of b3968d0207b54ece87cccc06515a89d4
using the swift command-line tool:
$
swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"
- Which HTTP method to allow (typically
GET
orPUT
) - The expiry date as a Unix timestamp
- The full path to the object
- The secret key set as the
X-Account-Meta-Temp-URL-Key
/v1/AUTH_account/container/object
:
import hmac from hashlib import sha1 from time import time method = 'GET' duration_in_seconds = 60*60*24 expires = int(time() + duration_in_seconds) path = '/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object' key = 'mykey' hmac_body = '%s\n%s\n%s' % (method, expires, path) sig = hmac.new(key, hmac_body, sha1).hexdigest() s = 'https://{host}/{path}?temp_url_sig={sig}&temp_url_expires={expires}' url = s.format(host='swift-cluster.example.com', path=path, sig=sig, expires=expires)
X-Account-Meta-Temp-URL-Key
invalidates any previously generated temporary URLs within 60 seconds (the memcache time for the key). Object Storage supports up to two keys, specified by X-Account-Meta-Temp-URL-Key
and X-Account-Meta-Temp-URL-Key-2
. Signatures are checked against both keys, if present. This is to allow for key rotation without invalidating all existing temporary URLs.
$
bin/swift-temp-url GET 3600 /v1/AUTH_account/container/object mykey
/v1/AUTH_account/container/object? temp_url_sig=5c4cc8886f36a9d0919d708ade98bf0cc71c9e91& temp_url_expires=1374497657
https://swift-cluster.example.com
).
Content-Disposition
header is set on the response so that browsers interpret this as a file attachment to be saved. The file name chosen is based on the object name, but you can override this with a filename
query parameter. The following example specifies a filename of My Test File.pdf
:
https://swift-cluster.example.com/v1/AUTH_a422b2-91f3-2f46-74b7-d7c9e8958f5d30/container/object? temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=1323479485& filename=My+Test+File.pdf
Content-Disposition: inline
to be set on the response by adding the inline
parameter to the query string, as follows:
https://swift-cluster.example.com/v1/AUTH_account/container/object? temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709& temp_url_expires=1323479485&inline
/etc/swift/proxy-server.conf
to add tempurl
to the pipeline
variable defined in the [pipeline:main]
section. The tempurl
entry should appear immediately before the authentication filters in the pipeline, such as authtoken
, tempauth
or keystoneauth
. For example:
[pipeline:main]
pipeline = pipeline = healthcheck cache tempurl authtoken keystoneauth proxy-server
Table 10.72. Description of configuration options for [filter-tempurl]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
incoming_allow_headers =
|
Headers allowed as exceptions to incoming_remove_headers. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. |
incoming_remove_headers = x-timestamp
|
Headers to remove from incoming requests. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. |
methods = GET HEAD PUT POST DELETE
|
HTTP methods allowed with Temporary URLs |
outgoing_allow_headers = x-object-meta-public-*
|
Headers allowed as exceptions to outgoing_allow_headers. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. |
outgoing_remove_headers = x-object-meta-*
|
Headers to remove from outgoing responses. Simply a whitespace delimited list of header names and names can optionally end with '*' to indicate a prefix match. |
use = egg:swift#tempurl
|
Entry point of paste.deploy in the server |
10.12.8. Name Check filter
Table 10.73. Description of configuration options for [filter-name_check]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
forbidden_chars = '"`<>
|
Characters that are not allowed in a name |
forbidden_regexp = /\./|/\.\./|/\.$|/\.\.$
|
Substrings to forbid, using regular expression syntax |
maximum_length = 255
|
Maximum length of a name |
use = egg:swift#name_check
|
Entry point of paste.deploy in the server |
10.12.9. Constraints
swift-constraints
section in the swift.conf
file. Use caution when you update these values because they affect the performance in the entire cluster.
Table 10.74. Description of configuration options for [swift-constraints]
in swift.conf
Configuration option = Default value | Description |
---|---|
account_listing_limit = 10000
|
The default (and maximum) number of items returned for an account listing request. |
container_listing_limit = 10000
|
The default (and maximum) number of items returned for a container listing request. |
extra_header_count = 0
|
By default the maximum number of allowed headers depends on the number of max allowed metadata settings plus a default value of 32 for regular http headers. If for some reason this is not enough (custom middleware for example) it can be increased with the extra_header_count constraint. |
max_account_name_length = 256
|
The maximum number of bytes in the utf8 encoding of an account name. |
max_container_name_length = 256
|
The maximum number of bytes in the utf8 encoding of a container name. |
max_file_size = 5368709122
|
The largest normal object that can be saved in the cluster. This is also the limit on the size of each segment of a large object when using the large object manifest support. This value is set in bytes. Setting it to lower than 1MiB will cause some tests to fail. It is STRONGLY recommended to leave this value at the default (5 * 2**30 + 2). |
max_header_size = 8192
|
The max number of bytes in the utf8 encoding of each header. Using 8192 as default because eventlet use 8192 as maximum size of header line. You may need to increase this value when using identity v3 API tokens including more than 7 catalog entries. See also include_service_catalog in proxy-server.conf-sample (documented in overview_auth.rst). |
max_meta_count = 90
|
The max number of metadata keys that can be stored on a single account, container, or object. |
max_meta_name_length = 128
|
The max number of bytes in the utf8 encoding of the name portion of a metadata header. |
max_meta_overall_size = 4096
|
The max number of bytes in the utf8 encoding of the metadata (keys + values). |
max_meta_value_length = 256
|
The max number of bytes in the utf8 encoding of a metadata value. |
max_object_name_length = 1024
|
The max number of bytes in the utf8 encoding of an object name. |
valid_api_versions = v0,v1,v2
|
No help text available for this option. |
10.12.10. Cluster health
/etc/swift/dispersion.conf
. Example dispersion.conf
file:
[dispersion] auth_url = http://localhost:8080/auth/v1.0 auth_user = test:tester auth_key = testing
$
swift-dispersion-report
Queried 2621 containers for dispersion reporting, 19s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries 100.00% of object copies found (7857 of 7857) Sample represents 1.00% of the object partition space
$
swift-ring-builder object.builder set_weight d0 200
$
swift-ring-builder object.builder rebalance
...$
swift-dispersion-report
Queried 2621 containers for dispersion reporting, 8s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries There were 1763 partitions missing one copy. 77.56% of object copies found (6094 of 7857) Sample represents 1.00% of the object partition space
... start object replicators and monitor logs until they're caught up ... $ swift-dispersion-report Queried 2621 containers for dispersion reporting, 17s, 0 retries 100.00% of container copies found (7863 of 7863) Sample represents 1.00% of the container partition space Queried 2619 objects for dispersion reporting, 7s, 0 retries 100.00% of object copies found (7857 of 7857) Sample represents 1.00% of the object partition space
$
swift-dispersion-report -j
{"object": {"retries:": 0, "missing_two": 0, "copies_found": 7863, "missing_one": 0, "copies_expected": 7863, "pct_found": 100.0, "overlapping": 0, "missing_all": 0}, "container": {"retries:": 0, "missing_two": 0, "copies_found": 12534, "missing_one": 0, "copies_expected": 12534, "pct_found": 100.0, "overlapping": 15, "missing_all": 0}}
Table 10.75. Description of configuration options for [dispersion]
in dispersion.conf
Configuration option = Default value | Description |
---|---|
auth_key = testing
|
No help text available for this option. |
auth_url = http://localhost:8080/auth/v1.0
|
Endpoint for auth server, such as keystone |
auth_user = test:tester
|
Default user for dispersion in this context |
auth_version = 1.0
|
Indicates which version of auth |
concurrency = 25
|
Number of replication workers to spawn |
container_populate = yes
|
No help text available for this option. |
container_report = yes
|
No help text available for this option. |
dispersion_coverage = 1.0
|
No help text available for this option. |
dump_json = no
|
No help text available for this option. |
endpoint_type = publicURL
|
Indicates whether endpoint for auth is public or internal |
keystone_api_insecure = no
|
Allow accessing insecure keystone server. The keystone's certificate will not be verified. |
object_populate = yes
|
No help text available for this option. |
object_report = yes
|
No help text available for this option. |
project_domain_name = project_domain
|
No help text available for this option. |
project_name = project
|
No help text available for this option. |
retries = 5
|
No help text available for this option. |
swift_dir = /etc/swift
|
Swift configuration directory |
user_domain_name = user_domain
|
No help text available for this option. |
10.12.11. Static Large Object (SLO) support
Table 10.76. Description of configuration options for [filter-slo]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
max_get_time = 86400
|
No help text available for this option. |
max_manifest_segments = 1000
|
No help text available for this option. |
max_manifest_size = 2097152
|
No help text available for this option. |
min_segment_size = 1048576
|
No help text available for this option. |
rate_limit_after_segment = 10
|
Rate limit the download of large object segments after this segment is downloaded. |
rate_limit_segments_per_sec = 0
|
Rate limit large object downloads at this rate. contact for a normal request. You can use '* replicas' at the end to have it use the number given times the number of replicas for the ring being used for the request. paste.deploy to use for auth. To use tempauth set to: `egg:swift#tempauth` each request |
use = egg:swift#slo
|
Entry point of paste.deploy in the server |
10.12.12. Container quotas
container_quotas
middleware implements simple quotas that can be imposed on Object Storage containers by a user with the ability to set container metadata, most likely the account administrator. This can be useful for limiting the scope of containers that are delegated to non-admin users, exposed to formpost uploads, or just as a self-imposed sanity check.
- X-Container-Meta-Quota-Bytes: Maximum size of the container, in bytes.
- X-Container-Meta-Quota-Count: Maximum object count of the container.
Table 10.77. Description of configuration options for [filter-container-quotas]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
use = egg:swift#container_quotas
|
Entry point of paste.deploy in the server |
10.12.13. Account quotas
x-account-meta-quota-bytes
metadata entry must be requests (PUT, POST) if a given account quota (in bytes) is exceeded while DELETE requests are still allowed.
x-account-meta-quota-bytes
metadata entry must be set to store and enable the quota. Write requests to this metadata entry are only permitted for resellers. There is no account quota limitation on a reseller account even if x-account-meta-quota-bytes
is set.
$
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin \ --os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:10000
$
swift -A http://127.0.0.1:8080/auth/v1.0 -U test:tester -K testing stat
Account: AUTH_test Containers: 0 Objects: 0 Bytes: 0 Meta Quota-Bytes: 10000 X-Timestamp: 1374075958.37454 X-Trans-Id: tx602634cf478546a39b1be-0051e6bc7a
$
swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin --os-storage-url http://127.0.0.1:8080/v1/AUTH_test post -m quota-bytes:
10.12.14. Bulk delete
bulk-delete
to delete multiple files from an account with a single request. Responds to DELETE requests with a header 'X-Bulk-Delete: true_value'. The body of the DELETE request is a new line-separated list of files to delete. The files listed must be URL encoded and in the form:
/container_name/obj_name
HTTPOk
. If any files failed to delete, the operation returns HTTPBadGateway
. In both cases, the response body is a JSON dictionary that shows the number of files that were successfully deleted or not found. The files that failed are listed.
Table 10.78. Description of configuration options for [filter-bulk]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
delete_container_retry_count = 0
|
No help text available for this option. |
max_containers_per_extraction = 10000
|
No help text available for this option. |
max_deletes_per_request = 10000
|
No help text available for this option. |
max_failed_deletes = 1000
|
No help text available for this option. |
max_failed_extractions = 1000
|
No help text available for this option. |
use = egg:swift#bulk
|
Entry point of paste.deploy in the server |
yield_frequency = 10
|
No help text available for this option. |
10.12.15. Drive audit
swift-drive-audit
configuration items reference a script that can be run by using cron to watch for bad drives. If errors are detected, it unmounts the bad drive, so that OpenStack Object Storage can work around it. It takes the following options:
Table 10.79. Description of configuration options for [drive-audit]
in drive-audit.conf
Configuration option = Default value | Description |
---|---|
device_dir = /srv/node
|
Directory devices are mounted under |
error_limit = 1
|
Number of errors to find before a device is unmounted |
log_address = /dev/log
|
Location where syslog sends the logs to |
log_facility = LOG_LOCAL0
|
Syslog log facility |
log_file_pattern = /var/log/kern.*[!.][!g][!z]
|
Location of the log file with globbing pattern to check against device errors locate device blocks with errors in the log file |
log_level = INFO
|
Logging level |
log_max_line_length = 0
|
Caps the length of log lines to the value given; no limit if set to 0, the default. |
log_to_console = False
|
No help text available for this option. |
minutes = 60
|
Number of minutes to look back in `/var/log/kern.log` |
recon_cache_path = /var/cache/swift
|
Directory where stats for a few items will be stored |
regex_pattern_1 = \berror\b.*\b(dm-[0-9]{1,2}\d?)\b
|
No help text available for this option. |
unmount_failed_device = True
|
No help text available for this option. |
10.12.16. Form post
10.12.16. Form post
<![CDATA[ <form action="<swift-url>" method="POST" enctype="multipart/form-data"> <input type="hidden" name="redirect" value="<redirect-url>" /> <input type="hidden" name="max_file_size" value="<bytes>" /> <input type="hidden" name="max_file_count" value="<count>" /> <input type="hidden" name="expires" value="<unix-timestamp>" /> <input type="hidden" name="signature" value="<hmac>" /> <input type="hidden" name="x_delete_at" value="<unix-timestamp>"/> <input type="hidden" name="x_delete_after" value="<seconds>"/> <input type="file" name="file1" /><br /> <input type="submit" /> </form>]]>
action="<swift-url>"
The URL to the Object Storage destination, such as https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix.The name of each uploaded file is appended to the specifiedswift-url
. So, you can upload directly to the root of container with a URL like https://swift-cluster.example.com/v1/AUTH_account/container/.Optionally, you can include an object prefix to separate different users' uploads, such as https://swift-cluster.example.com/v1/AUTH_account/container/object_prefix.method="POST"
The formmethod
must be POST.enctype="multipart/form-data
Theenctype
must be set tomultipart/form-data
.name="redirect"
The URL to which to redirect the browser after the upload completes. The URL has status and message query parameters added to it that indicate the HTTP status code for the upload and, optionally, additional error information. The 2nn status code indicates success. If an error occurs, the URL might include error information, such as"max_file_size exceeded"
.name="max_file_size"
Required. The maximum number of bytes that can be uploaded in a single file upload.name="max_file_count"
Required. The maximum number of files that can be uploaded with the form.name="expires"
The expiration date and time for the form in UNIX Epoch time stamp format. After this date and time, the form is no longer valid.For example,1440619048
is equivalent toMon, Wed, 26 Aug 2015 19:57:28 GMT
.name="signature"
The HMAC-SHA1 signature of the form. This sample Python code shows how to compute the signature:import hmac from hashlib import sha1 from time import time path = '/v1/account/container/object_prefix' redirect = 'https://myserver.com/some-page' max_file_size = 104857600 max_file_count = 10 expires = int(time() + 600) key = 'mykey' hmac_body = '%s\n%s\n%s\n%s\n%s' % (path, redirect, max_file_size, max_file_count, expires) signature = hmac.new(key, hmac_body, sha1).hexdigest()
The key is the value of theX-Account-Meta-Temp-URL-Key
header on the account.Use the full path from the/v1/
value and onward.During testing, you can use the swift-form-signature command-line tool to compute theexpires
andsignature
values.name="x_delete_at"
The date and time in UNIX Epoch time stamp format when the object will be removed.For example,1440619048
is equivalent toMon, Wed, 26 Aug 2015 19:57:28 GMT
.This attribute enables you to specify theX-Delete- At
header value in the form POST.name="x_delete_after"
The number of seconds after which the object is removed. Internally, the Object Storage system stores this value in theX-Delete-At
metadata item. This attribute enables you to specify theX-Delete-After
header value in the form POST.type="file" name="filexx"
Optional. One or more files to upload. Must appear after the other attributes to be processed correctly. If attributes come after thefile
attribute, they are not sent with the sub- request because on the server side, all attributes in the file cannot be parsed unless the whole file is read into memory and the server does not have enough memory to service these requests. So, attributes that follow thefile
attribute are ignored.
Table 10.80. Description of configuration options for [filter-formpost]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
use = egg:swift#formpost
|
Entry point of paste.deploy in the server |
10.12.17. Static web sites
Table 10.81. Description of configuration options for [filter-staticweb]
in proxy-server.conf
Configuration option = Default value | Description |
---|---|
use = egg:swift#staticweb
|
Entry point of paste.deploy in the server |
10.12.18. Cross-origin resource sharing
cors_allow_origin
option in the proxy-server.conf
file to set a list of hosts that are included with any CORS request by default.
10.12.19. Endpoint listing middleware
/endpoints/{account}/{container}/{object} /endpoints/{account}/{container} /endpoints/{account}
list_endpoints_path
configuration option in the proxy_server.conf
file to customize the /endpoints/
path.
http://{server}:{port}/{dev}/{part}/{acc}/{cont}/{obj} http://{server}:{port}/{dev}/{part}/{acc}/{cont} http://{server}:{port}/{dev}/{part}/{acc}
http://10.1.1.1:6000/sda1/2/a/c2/o1 http://10.1.1.1:6000/sda1/2/a/c2 http://10.1.1.1:6000/sda1/2/a