Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 6. Data processing service

The Data processing service (sahara) provides a scalable data-processing stack and associated management interfaces.
The following tables provide a comprehensive list of the Data processing service configuration options.

Table 6.1.  Description of AMQP configuration options

Configuration option = Default valueDescription
[DEFAULT]
control_exchange = openstack (String) The default exchange under which topics are scoped. May be overridden by an exchange name specified in the transport_url option.
transport_url = None (String) A URL representing the messaging driver to use and its full configuration. If not set, we fall back to the rpc_backend option and driver specific configuration.

Table 6.2.  Description of API configuration options

Configuration option = Default valueDescription
[oslo_middleware]
max_request_body_size = 114688 (Integer) The maximum body size for each request, in bytes.
secure_proxy_ssl_header = X-Forwarded-Proto (String) DEPRECATED: The HTTP Header that will be used to determine what the original request protocol scheme was, even if it was hidden by an SSL termination proxy.
[retries]
retries_number = 5 (Integer) Number of times to retry the request to client before failing
retry_after = 10 (Integer) Time between the retries to client (in seconds).
[service_auth]
admin_password = password (String) The service admin password
admin_project_domain = admin (String) The admin project domain name
admin_tenant_name = admin (String) The service admin tenant name
admin_user = admin (String) The service admin user name
admin_user_domain = admin (String) The admin user domain name
auth_url = http://127.0.0.1:5000/v2.0 (String) Authentication endpoint
auth_version = 2 (String) The auth version used to authenticate
endpoint_type = public (String) The endpoint_type to be used
region = RegionOne (String) The deployment region
service_name = lbaas (String) The name of the service

Table 6.3.  Description of authorization token configuration options

Configuration option = Default valueDescription
[keystone_authtoken]
admin_password = None (String) Service user password.
admin_tenant_name = admin (String) Service tenant name.
admin_token = None (String) This option is deprecated and may be removed in a future release. Single shared secret with the Keystone configuration used for bootstrapping a Keystone installation, or otherwise bypassing the normal authentication process. This option should not be used, use `admin_user` and `admin_password` instead.
admin_user = None (String) Service username.
auth_admin_prefix = (String) Prefix to prepend at the beginning of the path. Deprecated, use identity_uri.
auth_host = 127.0.0.1 (String) Host providing the admin Identity API endpoint. Deprecated, use identity_uri.
auth_port = 35357 (Integer) Port of the admin Identity API endpoint. Deprecated, use identity_uri.
auth_protocol = https (String) Protocol of the admin Identity API endpoint. Deprecated, use identity_uri.
auth_section = None (Unknown) Config Section from which to load plugin specific options
auth_type = None (Unknown) Authentication type to load
auth_uri = None (String) Complete public Identity API endpoint.
auth_version = None (String) API version of the admin Identity API endpoint.
cache = None (String) Env key for the swift cache.
cafile = None (String) A PEM encoded Certificate Authority to use when verifying HTTPs connections. Defaults to system CAs.
certfile = None (String) Required if identity server requires client certificate
check_revocations_for_cached = False (Boolean) If true, the revocation list will be checked for cached tokens. This requires that PKI tokens are configured on the identity server.
delay_auth_decision = False (Boolean) Do not handle authorization requests within the middleware, but delegate the authorization decision to downstream WSGI components.
enforce_token_bind = permissive (String) Used to control the use and type of token binding. Can be set to: "disabled" to not check token binding. "permissive" (default) to validate binding information if the bind type is of a form known to the server and ignore it if not. "strict" like "permissive" but if the bind type is unknown the token will be rejected. "required" any form of token binding is needed to be allowed. Finally the name of a binding method that must be present in tokens.
hash_algorithms = md5 (List) Hash algorithms to use for hashing PKI tokens. This may be a single algorithm or multiple. The algorithms are those supported by Python standard hashlib.new(). The hashes will be tried in the order given, so put the preferred one first for performance. The result of the first hash will be stored in the cache. This will typically be set to multiple values only while migrating from a less secure algorithm to a more secure one. Once all the old tokens are expired this option should be set to a single value for better performance.
http_connect_timeout = None (Integer) Request timeout value for communicating with Identity API server.
http_request_max_retries = 3 (Integer) How many times are we trying to reconnect when communicating with Identity API Server.
identity_uri = None (String) Complete admin Identity API endpoint. This should specify the unversioned root endpoint e.g. https://localhost:35357/
include_service_catalog = True (Boolean) (Optional) Indicate whether to set the X-Service-Catalog header. If False, middleware will not ask for service catalog on token validation and will not set the X-Service-Catalog header.
insecure = False (Boolean) Verify HTTPS connections.
keyfile = None (String) Required if identity server requires client certificate
memcache_pool_conn_get_timeout = 10 (Integer) (Optional) Number of seconds that an operation will wait to get a memcached client connection from the pool.
memcache_pool_dead_retry = 300 (Integer) (Optional) Number of seconds memcached server is considered dead before it is tried again.
memcache_pool_maxsize = 10 (Integer) (Optional) Maximum total number of open connections to every memcached server.
memcache_pool_socket_timeout = 3 (Integer) (Optional) Socket timeout in seconds for communicating with a memcached server.
memcache_pool_unused_timeout = 60 (Integer) (Optional) Number of seconds a connection to memcached is held unused in the pool before it is closed.
memcache_secret_key = None (String) (Optional, mandatory if memcache_security_strategy is defined) This string is used for key derivation.
memcache_security_strategy = None (String) (Optional) If defined, indicate whether token data should be authenticated or authenticated and encrypted. If MAC, token data is authenticated (with HMAC) in the cache. If ENCRYPT, token data is encrypted and authenticated in the cache. If the value is not one of these options or empty, auth_token will raise an exception on initialization.
memcache_use_advanced_pool = False (Boolean) (Optional) Use the advanced (eventlet safe) memcached client pool. The advanced pool will only work under python 2.x.
region_name = None (String) The region in which the identity server can be found.
revocation_cache_time = 10 (Integer) Determines the frequency at which the list of revoked tokens is retrieved from the Identity service (in seconds). A high number of revocation events combined with a low cache duration may significantly reduce performance.
signing_dir = None (String) Directory used to cache files related to PKI tokens.
token_cache_time = 300 (Integer) In order to prevent excessive effort spent validating tokens, the middleware caches previously-seen tokens for a configurable duration (in seconds). Set to -1 to disable caching completely.

Table 6.4.  Description of clients configuration options

Configuration option = Default valueDescription
[cinder]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to cinder.
api_version = 2 (Integer) Version of the Cinder API to use.
ca_file = None (String) Location of ca certificates file to use for cinder client requests.
endpoint_type = internalURL (String) Endpoint type for cinder client requests
[heat]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to heat.
ca_file = None (String) Location of ca certificates file to use for heat client requests.
endpoint_type = internalURL (String) Endpoint type for heat client requests
[keystone]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to keystone.
ca_file = None (String) Location of ca certificates file to use for keystone client requests.
endpoint_type = internalURL (String) Endpoint type for keystone client requests
[manila]
api_insecure = True (Boolean) Allow to perform insecure SSL requests to manila.
api_version = 1 (Integer) Version of the manila API to use.
ca_file = None (String) Location of ca certificates file to use for manila client requests.
[neutron]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to neutron.
ca_file = None (String) Location of ca certificates file to use for neutron client requests.
endpoint_type = internalURL (String) Endpoint type for neutron client requests
[nova]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to nova.
ca_file = None (String) Location of ca certificates file to use for nova client requests.
endpoint_type = internalURL (String) Endpoint type for nova client requests
[swift]
api_insecure = False (Boolean) Allow to perform insecure SSL requests to swift.
ca_file = None (String) Location of ca certificates file to use for swift client requests.
endpoint_type = internalURL (String) Endpoint type for swift client requests

Table 6.5.  Description of common configuration options

Configuration option = Default valueDescription
[DEFAULT]
admin_project_domain_name = default (String) The name of the domain for the service project(ex. tenant).
admin_user_domain_name = default (String) The name of the domain to which the admin user belongs.
api_workers = 1 (Integer) Number of workers for Sahara API service (0 means all-in-one-thread configuration).
cleanup_time_for_incomplete_clusters = 0 (Integer) Maximal time (in hours) for clusters allowed to be in states other than "Active", "Deleting" or "Error". If a cluster is not in "Active", "Deleting" or "Error" state and last update of it was longer than "cleanup_time_for_incomplete_clusters" hours ago then it will be deleted automatically. (0 value means that automatic clean up is disabled).
cluster_remote_threshold = 70 (Integer) The same as global_remote_threshold, but for a single cluster.
compute_topology_file = etc/sahara/compute.topology (String) File with nova compute topology. It should contain mapping between nova computes and racks.
coordinator_heartbeat_interval = 1 (Integer) Interval size between heartbeat execution in seconds. Heartbeats are executed to make sure that connection to the coordination server is active.
default_ntp_server = pool.ntp.org (String) Default ntp server for time sync
disable_event_log = False (Boolean) Disables event log feature.
enable_data_locality = False (Boolean) Enables data locality for hadoop cluster. Also enables data locality for Swift used by hadoop. If enabled, 'compute_topology' and 'swift_topology' configuration parameters should point to OpenStack and Swift topology correspondingly.
enable_hypervisor_awareness = True (Boolean) Enables four-level topology for data locality. Works only if corresponding plugin supports such mode.
executor_thread_pool_size = 64 (Integer) Size of executor thread pool.
global_remote_threshold = 100 (Integer) Maximum number of remote operations that will be running at the same time. Note that each remote operation requires its own process to run.
hash_ring_replicas_count = 40 (Integer) Number of points that belongs to each member on a hash ring. The larger number leads to a better distribution.
heat_enable_wait_condition = True (Boolean) Enable wait condition feature to reduce polling during cluster creation
heat_stack_tags = data-processing-cluster (List) List of tags to be used during operating with stack.
infrastructure_engine = heat (String) DEPRECATED: An engine which will be used to provision infrastructure for Hadoop cluster.
job_binary_max_KB = 5120 (Integer) Maximum length of job binary data in kilobytes that may be stored or retrieved in a single operation.
job_canceling_timeout = 300 (Integer) Timeout for canceling job execution (in seconds). Sahara will try to cancel job execution during this time.
job_workflow_postfix = (String) Postfix for storing jobs in hdfs. Will be added to '/user/<hdfs user>/' path.
memcached_servers = None (List) Memcached servers or None for in process cache.
min_transient_cluster_active_time = 30 (Integer) Minimal "lifetime" in seconds for a transient cluster. Cluster is guaranteed to be "alive" within this time period.
node_domain = novalocal (String) The suffix of the node's FQDN. In nova-network that is the dhcp_domain config parameter.
os_region_name = None (String) Region name used to get services endpoints.
periodic_coordinator_backend_url = None (String) The backend URL to use for distributed periodic tasks coordination.
periodic_enable = True (Boolean) Enable periodic tasks.
periodic_fuzzy_delay = 60 (Integer) Range in seconds to randomly delay when starting the periodic task scheduler to reduce stampeding. (Disable by setting to 0).
periodic_interval_max = 60 (Integer) Max interval size between periodic tasks execution in seconds.
periodic_workers_number = 1 (Integer) Number of threads to run periodic tasks.
plugins = vanilla, spark, cdh, ambari (List) List of plugins to be loaded. Sahara preserves the order of the list when returning it.
proxy_command = (String) Proxy command used to connect to instances. If set, this command should open a netcat socket, that Sahara will use for SSH and HTTP connections. Use {host} and {port} to describe the destination. Other available keywords: {tenant_id}, {network_id}, {router_id}.
remote = ssh (String) A method for Sahara to execute commands on VMs.
rootwrap_command = sudo sahara-rootwrap /etc/sahara/rootwrap.conf (String) Rootwrap command to leverage. Use in conjunction with use_rootwrap=True
swift_topology_file = etc/sahara/swift.topology (String) File with Swift topology.It should contain mapping between Swift nodes and racks.
use_barbican_key_manager = False (Boolean) Enable the usage of the OpenStack Key Management service provided by barbican.
use_floating_ips = True (Boolean) If set to True, Sahara will use floating IPs to communicate with instances. To make sure that all instances have floating IPs assigned in Nova Network set "auto_assign_floating_ip=True" in nova.conf. If Neutron is used for networking, make sure that all Node Groups have "floating_ip_pool" parameter defined.
use_identity_api_v3 = True (Boolean) Enables Sahara to use Keystone API v3. If that flag is disabled, per-job clusters will not be terminated automatically.
use_namespaces = False (Boolean) Use network namespaces for communication (only valid to use in conjunction with use_neutron=True).
use_neutron = False (Boolean) Use Neutron Networking (False indicates the use of Nova networking).
use_rootwrap = False (Boolean) Use rootwrap facility to allow non-root users to run the sahara-all server instance and access private network IPs (only valid to use in conjunction with use_namespaces=True)
[castellan]
barbican_api_endpoint = None (String) The endpoint to use for connecting to the barbican api controller. By default, castellan will use the URL from the service catalog.
barbican_api_version = v1 (String) Version of the barbican API, for example: "v1"
[certificates]
barbican_auth = barbican_acl_auth (String) Name of the Barbican authentication method to use
cert_manager_type = barbican (String) Certificate Manager plugin. Defaults to barbican.
[cluster_verifications]
verification_enable = True (Boolean) Option to enable verifications for all clusters
verification_periodic_interval = 600 (Integer) Interval between two consecutive periodic tasks forverifications, in seconds.
[conductor]
use_local = True (Boolean) Perform sahara-conductor operations locally.
[keystone_authtoken]
memcached_servers = None (List) Optionally specify a list of memcached server(s) to use for caching. If left undefined, tokens will instead be cached in-process.

Table 6.6.  Description of CORS configuration options

Configuration option = Default valueDescription
[cors]
allow_credentials = True (Boolean) Indicate that the actual request can include user credentials
allow_headers = Content-Type, Cache-Control, Content-Language, Expires, Last-Modified, Pragma (List) Indicate which header field names may be used during the actual request.
allow_methods = GET, POST, PUT, DELETE, OPTIONS (List) Indicate which methods can be used during the actual request.
allowed_origin = None (List) Indicate whether this resource may be shared with the domain received in the requests "origin" header.
expose_headers = Content-Type, Cache-Control, Content-Language, Expires, Last-Modified, Pragma (List) Indicate which headers are safe to expose to the API. Defaults to HTTP Simple Headers.
max_age = 3600 (Integer) Maximum cache age of CORS preflight requests.
[cors.subdomain]
allow_credentials = True (Boolean) Indicate that the actual request can include user credentials
allow_headers = Content-Type, Cache-Control, Content-Language, Expires, Last-Modified, Pragma (List) Indicate which header field names may be used during the actual request.
allow_methods = GET, POST, PUT, DELETE, OPTIONS (List) Indicate which methods can be used during the actual request.
allowed_origin = None (List) Indicate whether this resource may be shared with the domain received in the requests "origin" header.
expose_headers = Content-Type, Cache-Control, Content-Language, Expires, Last-Modified, Pragma (List) Indicate which headers are safe to expose to the API. Defaults to HTTP Simple Headers.
max_age = 3600 (Integer) Maximum cache age of CORS preflight requests.

Table 6.7.  Description of database configuration options

Configuration option = Default valueDescription
[DEFAULT]
db_driver = sahara.db (String) Driver to use for database access.
[database]
backend = sqlalchemy (String) The back end to use for the database.
connection = None (String) The SQLAlchemy connection string to use to connect to the database.
connection_debug = 0 (Integer) Verbosity of SQL debugging information: 0=None, 100=Everything.
connection_trace = False (Boolean) Add Python stack traces to SQL as comment strings.
db_inc_retry_interval = True (Boolean) If True, increases the interval between retries of a database operation up to db_max_retry_interval.
db_max_retries = 20 (Integer) Maximum retries in case of connection error or deadlock error before error is raised. Set to -1 to specify an infinite retry count.
db_max_retry_interval = 10 (Integer) If db_inc_retry_interval is set, the maximum seconds between retries of a database operation.
db_retry_interval = 1 (Integer) Seconds between retries of a database transaction.
idle_timeout = 3600 (Integer) Timeout before idle SQL connections are reaped.
max_overflow = 50 (Integer) If set, use this value for max_overflow with SQLAlchemy.
max_pool_size = None (Integer) Maximum number of SQL connections to keep open in a pool.
max_retries = 10 (Integer) Maximum number of database connection retries during startup. Set to -1 to specify an infinite retry count.
min_pool_size = 1 (Integer) Minimum number of SQL connections to keep open in a pool.
mysql_sql_mode = TRADITIONAL (String) The SQL mode to be used for MySQL sessions. This option, including the default, overrides any server-set SQL mode. To use whatever SQL mode is set by the server configuration, set this to no value. Example: mysql_sql_mode=
pool_timeout = None (Integer) If set, use this value for pool_timeout with SQLAlchemy.
retry_interval = 10 (Integer) Interval between retries of opening a SQL connection.
slave_connection = None (String) The SQLAlchemy connection string to use to connect to the slave database.
sqlite_db = oslo.sqlite (String) The file name to use with SQLite.
sqlite_synchronous = True (Boolean) If True, SQLite uses synchronous mode.
use_db_reconnect = False (Boolean) Enable the experimental use of database reconnect on connection lost.

Table 6.8.  Description of domain configuration options

Configuration option = Default valueDescription
[DEFAULT]
proxy_user_domain_name = None (String) The domain Sahara will use to create new proxy users for Swift object access.
proxy_user_role_names = Member (List) A list of the role names that the proxy user should assume through trust for Swift object access.
use_domain_for_proxy_users = False (Boolean) Enables Sahara to use a domain for creating temporary proxy users to access Swift. If this is enabled a domain must be created for Sahara to use.

Table 6.9.  Description of logging configuration options

Configuration option = Default valueDescription
[DEFAULT]
debug = False (Boolean) If set to true, the logging level will be set to DEBUG instead of the default INFO level.
default_log_levels = amqplib=WARN, qpid.messaging=INFO, stevedore=INFO, eventlet.wsgi.server=WARN, sqlalchemy=WARN, boto=WARN, suds=INFO, keystone=INFO, paramiko=WARN, requests=WARN, iso8601=WARN, oslo_messaging=INFO, neutronclient=INFO (List) List of package logging levels in logger=LEVEL pairs. This option is ignored if log_config_append is set.
fatal_deprecations = False (Boolean) Enables or disables fatal status of deprecations.
instance_format = "[instance: %(uuid)s] " (String) The format for an instance that is passed with the log message.
instance_uuid_format = "[instance: %(uuid)s] " (String) The format for an instance UUID that is passed with the log message.
log_config_append = None (String) The name of a logging configuration file. This file is appended to any existing logging configuration files. For details about logging configuration files, see the Python logging module documentation. Note that when logging configuration files are used then all logging configuration is set in the configuration file and other logging configuration options are ignored (for example, logging_context_format_string).
log_date_format = %Y-%m-%d %H:%M:%S (String) Defines the format string for %%(asctime)s in log records. Default: %(default)s . This option is ignored if log_config_append is set.
log_dir = None (String) (Optional) The base directory used for relative log_file paths. This option is ignored if log_config_append is set.
log_file = None (String) (Optional) Name of log file to send logging output to. If no default is set, logging will go to stderr as defined by use_stderr. This option is ignored if log_config_append is set.
logging_context_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s (String) Format string to use for log messages with context.
logging_debug_format_suffix = %(funcName)s %(pathname)s:%(lineno)d (String) Additional data to append to log message when logging level for the message is DEBUG.
logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s (String) Format string to use for log messages when context is undefined.
logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s (String) Prefix each line of exception output with this format.
logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s (String) Defines the format string for %(user_identity)s that is used in logging_context_format_string.
publish_errors = False (Boolean) Enables or disables publication of error events.
syslog_log_facility = LOG_USER (String) Syslog facility to receive log lines. This option is ignored if log_config_append is set.
use_stderr = True (Boolean) Log output to standard error. This option is ignored if log_config_append is set.
use_syslog = False (Boolean) Use syslog for logging. Existing syslog format is DEPRECATED and will be changed later to honor RFC5424. This option is ignored if log_config_append is set.
verbose = True (Boolean) DEPRECATED: If set to false, the logging level will be set to WARNING instead of the default INFO level.
watch_log_file = False (Boolean) Uses logging handler designed to watch file system. When log file is moved or removed this handler will open a new log file with specified path instantaneously. It makes sense only if log_file option is specified and Linux platform is used. This option is ignored if log_config_append is set.

Table 6.10.  Description of Auth options for Swift access for VM configuration options

Configuration option = Default valueDescription
[object_store_access]
public_identity_ca_file = None (String) Location of ca certificate file to use for identity client requests via public endpoint
public_object_store_ca_file = None (String) Location of ca certificate file to use for object-store client requests via public endpoint

Table 6.11.  Description of policy configuration options

Configuration option = Default valueDescription
[oslo_policy]
policy_default_rule = default (String) Default rule. Enforced when a requested rule is not found.
policy_dirs = ['policy.d'] (Multi-valued) Directories where policy configuration files are stored. They can be relative to any directory in the search path defined by the config_dir option, or absolute paths. The file defined by policy_file must exist for these directories to be searched. Missing or empty directories are ignored.
policy_file = policy.json (String) The JSON file that defines policies.

Table 6.12. Description of Qpid configuration options

Configuration option = Default value Description
[oslo_messaging_qpid]
amqp_auto_delete = False (BoolOpt) Auto-delete queues in AMQP.
amqp_durable_queues = False (BoolOpt) Use durable queues in AMQP.
qpid_heartbeat = 60 (IntOpt) Seconds between connection keepalive heartbeats.
qpid_hostname = localhost (StrOpt) Qpid broker hostname.
qpid_hosts = $qpid_hostname:$qpid_port (ListOpt) Qpid HA cluster host:port pairs.
qpid_password = (StrOpt) Password for Qpid connection.
qpid_port = 5672 (IntOpt) Qpid broker port.
qpid_protocol = tcp (StrOpt) Transport to use, either 'tcp' or 'ssl'.
qpid_receiver_capacity = 1 (IntOpt) The number of prefetched messages held by receiver.
qpid_sasl_mechanisms = (StrOpt) Space separated list of SASL mechanisms to use for auth.
qpid_tcp_nodelay = True (BoolOpt) Whether to disable the Nagle algorithm.
qpid_topology_version = 1 (IntOpt) The qpid topology version to use. Version 1 is what was originally used by impl_qpid. Version 2 includes some backwards-incompatible changes that allow broker federation to work. Users should update to version 2 when they are able to take everything down, as it requires a clean break.
qpid_username = (StrOpt) Username for Qpid connection.
send_single_reply = False (BoolOpt) Send a single AMQP reply to call message. The current behavior since oslo-incubator is to send two AMQP replies - first one with the payload, a second one to ensure the other has finished to send the payload. We are going to remove it in the N release, but we must keep backward compatible at the same time. This option provides such compatibility - it defaults to False in Liberty and can be turned on for early adopters with new installations or for testing. This option will be removed in the Mitaka release.

Table 6.13.  Description of RabbitMQ configuration options

Configuration option = Default valueDescription
[oslo_messaging_rabbit]
amqp_auto_delete = False (Boolean) Auto-delete queues in AMQP.
amqp_durable_queues = False (Boolean) Use durable queues in AMQP.
channel_max = None (Integer) Maximum number of channels to allow
default_notification_exchange = ${control_exchange}_notification (String) Exchange name for for sending notifications
default_notification_retry_attempts = -1 (Integer) Reconnecting retry count in case of connectivity problem during sending notification, -1 means infinite retry.
default_rpc_exchange = ${control_exchange}_rpc (String) Exchange name for sending RPC messages
default_rpc_retry_attempts = -1 (Integer) Reconnecting retry count in case of connectivity problem during sending RPC message, -1 means infinite retry. If actual retry attempts in not 0 the rpc request could be processed more then one time
fake_rabbit = False (Boolean) Deprecated, use rpc_backend=kombu+memory or rpc_backend=fake
frame_max = None (Integer) The maximum byte size for an AMQP frame
heartbeat_interval = 1 (Integer) How often to send heartbeats for consumer's connections
heartbeat_rate = 2 (Integer) How often times during the heartbeat_timeout_threshold we check the heartbeat.
heartbeat_timeout_threshold = 60 (Integer) Number of seconds after which the Rabbit broker is considered down if heartbeat's keep-alive fails (0 disable the heartbeat). EXPERIMENTAL
host_connection_reconnect_delay = 0.25 (Floating point) Set delay for reconnection to some host which has connection error
kombu_compression = None (String) EXPERIMENTAL: Possible values are: gzip, bz2. If not set compression will not be used. This option may notbe available in future versions.
kombu_failover_strategy = round-robin (String) Determines how the next RabbitMQ node is chosen in case the one we are currently connected to becomes unavailable. Takes effect only if more than one RabbitMQ node is provided in config.
kombu_missing_consumer_retry_timeout = 60 (Integer) How long to wait a missing client beforce abandoning to send it its replies. This value should not be longer than rpc_response_timeout.
kombu_reconnect_delay = 1.0 (Floating point) How long to wait before reconnecting in response to an AMQP consumer cancel notification.
kombu_ssl_ca_certs = (String) SSL certification authority file (valid only if SSL enabled).
kombu_ssl_certfile = (String) SSL cert file (valid only if SSL enabled).
kombu_ssl_keyfile = (String) SSL key file (valid only if SSL enabled).
kombu_ssl_version = (String) SSL version to use (valid only if SSL enabled). Valid values are TLSv1 and SSLv23. SSLv2, SSLv3, TLSv1_1, and TLSv1_2 may be available on some distributions.
notification_listener_prefetch_count = 100 (Integer) Max number of not acknowledged message which RabbitMQ can send to notification listener.
notification_persistence = False (Boolean) Persist notification messages.
notification_retry_delay = 0.25 (Floating point) Reconnecting retry delay in case of connectivity problem during sending notification message
pool_max_overflow = 0 (Integer) Maximum number of connections to create above `pool_max_size`.
pool_max_size = 10 (Integer) Maximum number of connections to keep queued.
pool_recycle = 600 (Integer) Lifetime of a connection (since creation) in seconds or None for no recycling. Expired connections are closed on acquire.
pool_stale = 60 (Integer) Threshold at which inactive (since release) connections are considered stale in seconds or None for no staleness. Stale connections are closed on acquire.
pool_timeout = 30 (Integer) Default number of seconds to wait for a connections to available
rabbit_ha_queues = False (Boolean) Try to use HA queues in RabbitMQ (x-ha-policy: all). If you change this option, you must wipe the RabbitMQ database. In RabbitMQ 3.0, queue mirroring is no longer controlled by the x-ha-policy argument when declaring a queue. If you just want to make sure that all queues (except those with auto-generated names) are mirrored across all nodes, run: "rabbitmqctl set_policy HA '^(?!amq\.).*' '{"ha-mode": "all"}' "
rabbit_host = localhost (String) The RabbitMQ broker address where a single node is used.
rabbit_hosts = $rabbit_host:$rabbit_port (List) RabbitMQ HA cluster host:port pairs.
rabbit_interval_max = 30 (Integer) Maximum interval of RabbitMQ connection retries. Default is 30 seconds.
rabbit_login_method = AMQPLAIN (String) The RabbitMQ login method.
rabbit_max_retries = 0 (Integer) Maximum number of RabbitMQ connection retries. Default is 0 (infinite retry count).
rabbit_password = guest (String) The RabbitMQ password.
rabbit_port = 5672 (Port number) The RabbitMQ broker port where a single node is used.
rabbit_qos_prefetch_count = 0 (Integer) Specifies the number of messages to prefetch. Setting to zero allows unlimited messages.
rabbit_retry_backoff = 2 (Integer) How long to backoff for between retries when connecting to RabbitMQ.
rabbit_retry_interval = 1 (Integer) How frequently to retry connecting with RabbitMQ.
rabbit_transient_queues_ttl = 1800 (Integer) Positive integer representing duration in seconds for queue TTL (x-expires). Queues which are unused for the duration of the TTL are automatically deleted. The parameter affects only reply and fanout queues.
rabbit_use_ssl = False (Boolean) Connect over SSL for RabbitMQ.
rabbit_userid = guest (String) The RabbitMQ userid.
rabbit_virtual_host = / (String) The RabbitMQ virtual host.
rpc_listener_prefetch_count = 100 (Integer) Max number of not acknowledged message which RabbitMQ can send to rpc listener.
rpc_queue_expiration = 60 (Integer) Time to live for rpc queues without consumers in seconds.
rpc_reply_exchange = ${control_exchange}_rpc_reply (String) Exchange name for receiving RPC replies
rpc_reply_listener_prefetch_count = 100 (Integer) Max number of not acknowledged message which RabbitMQ can send to rpc reply listener.
rpc_reply_retry_attempts = -1 (Integer) Reconnecting retry count in case of connectivity problem during sending reply. -1 means infinite retry during rpc_timeout
rpc_reply_retry_delay = 0.25 (Floating point) Reconnecting retry delay in case of connectivity problem during sending reply.
rpc_retry_delay = 0.25 (Floating point) Reconnecting retry delay in case of connectivity problem during sending RPC message
socket_timeout = 0.25 (Floating point) Set socket timeout in seconds for connection's socket
ssl = None (Boolean) Enable SSL
ssl_options = None (Dict) Arguments passed to ssl.wrap_socket
tcp_user_timeout = 0.25 (Floating point) Set TCP_USER_TIMEOUT in seconds for connection's socket

Table 6.14.  Description of Redis configuration options

Configuration option = Default valueDescription
[matchmaker_redis]
check_timeout = 20000 (Integer) Time in ms to wait before the transaction is killed.
host = 127.0.0.1 (String) Host to locate redis.
password = (String) Password for Redis server (optional).
port = 6379 (Port number) Use this port to connect to redis host.
sentinel_group_name = oslo-messaging-zeromq (String) Redis replica set name.
sentinel_hosts = (List) List of Redis Sentinel hosts (fault tolerance mode) e.g. [host:port, host1:port ... ]
socket_timeout = 1000 (Integer) Timeout in ms on blocking socket operations
wait_timeout = 500 (Integer) Time in ms to wait between connection attempts.

Table 6.15.  Description of RPC configuration options

Configuration option = Default valueDescription
[DEFAULT]
rpc_backend = rabbit (String) The messaging driver to use, defaults to rabbit. Other drivers include amqp and zmq.
rpc_cast_timeout = -1 (Integer) Seconds to wait before a cast expires (TTL). The default value of -1 specifies an infinite linger period. The value of 0 specifies no linger period. Pending messages shall be discarded immediately when the socket is closed. Only supported by impl_zmq.
rpc_conn_pool_size = 30 (Integer) Size of RPC connection pool.
rpc_poll_timeout = 1 (Integer) The default number of seconds that poll should wait. Poll raises timeout exception when timeout expired.
rpc_response_timeout = 60 (Integer) Seconds to wait for a response from a call.
[oslo_concurrency]
disable_process_locking = False (Boolean) Enables or disables inter-process locks.
lock_path = None (String) Directory to use for lock files. For security, the specified directory should only be writable by the user running the processes that need locking. Defaults to environment variable OSLO_LOCK_PATH. If external locks are used, a lock path must be set.
[oslo_messaging]
event_stream_topic = neutron_lbaas_event (String) topic name for receiving events from a queue
[oslo_messaging_amqp]
allow_insecure_clients = False (Boolean) Accept clients using either SSL or plain TCP
broadcast_prefix = broadcast (String) address prefix used when broadcasting to all servers
container_name = None (String) Name for the AMQP container
group_request_prefix = unicast (String) address prefix when sending to any server in group
idle_timeout = 0 (Integer) Timeout for inactive connections (in seconds)
password = (String) Password for message broker authentication
sasl_config_dir = (String) Path to directory that contains the SASL configuration
sasl_config_name = (String) Name of configuration file (without .conf suffix)
sasl_mechanisms = (String) Space separated list of acceptable SASL mechanisms
server_request_prefix = exclusive (String) address prefix used when sending to a specific server
ssl_ca_file = (String) CA certificate PEM file to verify server certificate
ssl_cert_file = (String) Identifying certificate PEM file to present to clients
ssl_key_file = (String) Private key PEM file used to sign cert_file certificate
ssl_key_password = None (String) Password for decrypting ssl_key_file (if encrypted)
trace = False (Boolean) Debug: dump AMQP frames to stdout
username = (String) User name for message broker authentication
[oslo_messaging_notifications]
driver = [] (Multi-valued) The Drivers(s) to handle sending notifications. Possible values are messaging, messagingv2, routing, log, test, noop
enable = False (Boolean) Enables sending notifications to Ceilometer
level = INFO (String) Notification level for outgoing notifications
publisher_id = None (String) Notification publisher_id for outgoing notifications
topics = notifications (List) AMQP topic used for OpenStack notifications.
transport_url = None (String) A URL representing the messaging driver to use for notifications. If not set, we fall back to the same configuration used for RPC.

Table 6.16.  Description of timeouts configuration options

Configuration option = Default valueDescription
[timeouts]
delete_instances_timeout = 10800 (Integer) Wait for instances to be deleted, in seconds
detach_volume_timeout = 300 (Integer) Timeout for detaching volumes from instance, in seconds
ips_assign_timeout = 10800 (Integer) Assign IPs timeout, in seconds
wait_until_accessible = 10800 (Integer) Wait for instance accessibility, in seconds

6.1. New, updated, and deprecated options in Mitaka for Data Processing service

Table 6.17.  New options

Configuration option = Default valueDescription
[DEFAULT] coordinator_heartbeat_interval = 1 (IntOpt) Interval size between heartbeat execution in seconds. Heartbeats are executed to make sure that connection to the coordination server is active.
[DEFAULT] hash_ring_replicas_count = 40 (IntOpt) Number of points that belongs to each member on a hash ring. The larger number leads to a better distribution.
[DEFAULT] periodic_coordinator_backend_url = None (StrOpt) The backend URL to use for distributed periodic tasks coordination.
[DEFAULT] periodic_workers_number = 1 (IntOpt) Number of threads to run periodic tasks.
[DEFAULT] use_barbican_key_manager = False (BoolOpt) Enable the usage of the OpenStack Key Management service provided by barbican.
[castellan] barbican_api_endpoint = None (StrOpt) The endpoint to use for connecting to the barbican api controller. By default, castellan will use the URL from the service catalog.
[castellan] barbican_api_version = v1 (StrOpt) Version of the barbican API, for example: "v1"
[cluster_verifications] verification_enable = True (BoolOpt) Option to enable verifications for all clusters
[cluster_verifications] verification_periodic_interval = 600 (IntOpt) Interval between two consecutive periodic tasks forverifications, in seconds.
[oslo_messaging_notifications] enable = False (BoolOpt) Enables sending notifications to Ceilometer
[oslo_messaging_notifications] level = INFO (StrOpt) Notification level for outgoing notifications
[oslo_messaging_notifications] publisher_id = None (StrOpt) Notification publisher_id for outgoing notifications

Table 6.18.  New default values

OptionPrevious default valueNew default value
[DEFAULT] api_workers 0 1
[DEFAULT] plugins hdp, cdh cdh

Table 6.19.  Deprecated options

Configuration option = Default valueDescription
[DEFAULT] enable_notifications [oslo_messaging_notifications] enable
[DEFAULT] notification_level [oslo_messaging_notifications] level
[DEFAULT] notification_publisher_id [oslo_messaging_notifications] publisher_id
[DEFAULT] use_syslog None