Chapter 18. Securing endpoints in Red Hat OpenStack Platform

The process of engaging with an OpenStack cloud begins by querying an API endpoint. While there are different challenges for public and private endpoints, these are high value assets that can pose a significant risk if compromised.

This chapter recommends security enhancements for both public and private-facing API endpoints.

18.1. Internal API communications

OpenStack provides both public-facing, internal admin, and private API endpoints. By default, OpenStack components use the publicly defined endpoints. The recommendation is to configure these components to use the API endpoint within the proper security domain. The internal admin endpoint allows further elevated access to keystone, so it might be desirable to further isolate this.

Services select their respective API endpoints based on the OpenStack service catalog. These services might not obey the listed public or internal API endpoint values. This can lead to internal management traffic being routed to external API endpoints.

18.2. Configure internal URLs in the Identity service catalog

The Identity service catalog should be aware of your internal URLs. While this feature is not used by default, it may be available through configuration. In addition, it should be forward-compatible with expectant changes once this behavior becomes the default.

Consider isolating the configured endpoints from a network level, given that they have different levels of access. The Admin endpoint is intended for access by cloud administrators, as it provides elevated access to keystone operations not available on the internal or public endpoints. The internal endpoints are intended for uses internal to the cloud (for example, by OpenStack services), and usually would not be accessible outside of the deployment network. The public endpoints should be TLS-enabled, and the only API endpoints accessible outside of the deployment for cloud users to operate on.

Registration of an internal URL for an endpoint is automated by director. For more information, see https://github.com/openstack/tripleo-heat-templates/blob/a7857d6dfcc875eb2bc611dd9334104c18fe8ac6/network/endpoints/build_endpoint_map.py.

18.3. Configure applications for internal URLs

You can force some services to use specific API endpoints. As a result, it is recommended that any OpenStack service that contacts the API of another service must be explicitly configured to access the proper internal API endpoint.

Each project might present an inconsistent way of defining target API endpoints. Future releases of OpenStack seek to resolve these inconsistencies through consistent use of the Identity service catalog.

18.4. Paste and middleware

Most API endpoints and other HTTP services in OpenStack use the Python Paste Deploy library. From a security perspective, this library enables manipulation of the request filter pipeline through the application’s configuration. Each element in this chain is referred to as middleware. Changing the order of filters in the pipeline or adding additional middleware might have unpredictable security impact.

Commonly, implementers add middleware to extend OpenStack’s base functionality. Consider giving careful consideration to the potential exposure introduced by the addition of non-standard software components to the HTTP request pipeline.

18.5. API endpoint process isolation and policy

You can isolate API endpoint processes for increased security. Consider deploying API endpoints within the public security domain on separate hosts for increased isolation.

18.5.1. Configuring a policy to restrict metadef APIs

In Red Hat OpenStack Platform (RHOSP), users can define key value pairs and tag metadata with metadata definition (metadef) APIs. Currently, there is no limit on the number of metadef namespaces, objects, properties, resources, or tags that users can create.

Metadef APIs can leak information to unauthorized users. A malicious user can exploit the lack of restrictions and fill the Image service (glance) database with unlimited resources, which can create a Denial of Service (DoS) style attack.

Image service policies control metadef APIs. However, the default policy setting for metadef APIs allows all users to create or read the metadef information. Because metadef resources are not isolated to the owner, metadef resources with potentially sensitive names, such as internal infrastructure details or customer names, can expose that information to malicious users.

To make the Image service (glance) more secure, restrict metadef modification APIs to admin-only access by default in your Red Hat OpenStack Platform (RHOSP) deployments.

Procedure

  1. As a cloud administrator, create a separate heat template environment file, such as lock-down-glance-metadef-api.yaml, to contain policy overrides for the Image service metadef API: 

    ...
parameter_defaults:
  GlanceApiPolicies: {
        glance-metadef_default: { key: 'metadef_default', value: '' },
        glance-metadef_admin: { key: 'metadef_admin', value: 'role:admin' },
        glance-get_metadef_namespace: { key: 'get_metadef_namespace', value: 'rule:metadef_default' },
        glance-get_metadef_namespaces: { key: 'get_metadef_namespaces', value: 'rule:metadef_default' },
        glance-modify_metadef_namespace: { key: 'modify_metadef_namespace', value: 'rule:metadef_admin' },
        glance-add_metadef_namespace: { key: 'add_metadef_namespace', value: 'rule:metadef_admin' },
        glance-delete_metadef_namespace: { key: 'delete_metadef_namespace', value: 'rule:metadef_admin' },
        glance-get_metadef_object: { key: 'get_metadef_object', value: 'rule:metadef_default' },
        glance-get_metadef_objects: { key: 'get_metadef_objects', value: 'rule:metadef_default' },
        glance-modify_metadef_object: { key: 'modify_metadef_object', value: 'rule:metadef_admin' },
        glance-add_metadef_object: { key: 'add_metadef_object', value: 'rule:metadef_admin' },
        glance-delete_metadef_object: { key: 'delete_metadef_object', value: 'rule:metadef_admin' },
        glance-list_metadef_resource_types: { key: 'list_metadef_resource_types', value: 'rule:metadef_default' },
        glance-get_metadef_resource_type: { key: 'get_metadef_resource_type', value: 'rule:metadef_default' },
        glance-add_metadef_resource_type_association: { key: 'add_metadef_resource_type_association', value: 'rule:metadef_admin' },
        glance-remove_metadef_resource_type_association: { key: 'remove_metadef_resource_type_association', value: 'rule:metadef_admin' },
        glance-get_metadef_property: { key: 'get_metadef_property', value: 'rule:metadef_default' },
        glance-get_metadef_properties: { key: 'get_metadef_properties', value: 'rule:metadef_default' },
        glance-modify_metadef_property: { key: 'modify_metadef_property', value: 'rule:metadef_admin' },
        glance-add_metadef_property: { key: 'add_metadef_property', value: 'rule:metadef_admin' },
        glance-remove_metadef_property: { key: 'remove_metadef_property', value: 'rule:metadef_admin' },
        glance-get_metadef_tag: { key: 'get_metadef_tag', value: 'rule:metadef_default' },
        glance-get_metadef_tags: { key: 'get_metadef_tags', value: 'rule:metadef_default' },
        glance-modify_metadef_tag: { key: 'modify_metadef_tag', value: 'rule:metadef_admin' },
        glance-add_metadef_tag: { key: 'add_metadef_tag', value: 'rule:metadef_admin' },
        glance-add_metadef_tags: { key: 'add_metadef_tags', value: 'rule:metadef_admin' },
        glance-delete_metadef_tag: { key: 'delete_metadef_tag', value: 'rule:metadef_admin' },
        glance-delete_metadef_tags: { key: 'delete_metadef_tags', value: 'rule:metadef_admin' }
  }

…

  2. Include the environment file that contains the policy overrides in the deployment command with the -e option when you deploy the overcloud: 

    $ openstack overcloud deploy -e lock-down-glance-metadef-api.yaml

18.5.2. Enabling metadef APIs

If you previously restricted metadata definition (metadef) APIs or want to relax the new defaults, you can override metadef modification policies to allow users to update their respective resources.

Important

Cloud administrators with users who depend on write access to the metadef APIs can make those APIs accessible to all users. In this type of configuration, however, there is the potential to unintentionally leak sensitive resource names, such as customer names and internal projects. Administrators must audit their systems to identify previously created resources that might be vulnerable even if only read access is enabled for all users.

Procedure

  1. As a cloud administrator, log in to the undercloud and create a file for policy overrides. For example: 

    $ cat open-up-glance-api-metadef.yaml

  2. Configure the policy override file to allow metadef API read-write access to all users: 

    GlanceApiPolicies: {
    glance-metadef_default: { key: 'metadef_default', value: '' },
    glance-get_metadef_namespace: { key: 'get_metadef_namespace', value: 'rule:metadef_default' },
    glance-get_metadef_namespaces: { key: 'get_metadef_namespaces', value: 'rule:metadef_default' },
    glance-modify_metadef_namespace: { key: 'modify_metadef_namespace', value: 'rule:metadef_default' },
    glance-add_metadef_namespace: { key: 'add_metadef_namespace', value: 'rule:metadef_default' },
    glance-delete_metadef_namespace: { key: 'delete_metadef_namespace', value: 'rule:metadef_default' },
    glance-get_metadef_object: { key: 'get_metadef_object', value: 'rule:metadef_default' },
    glance-get_metadef_objects: { key: 'get_metadef_objects', value: 'rule:metadef_default' },
    glance-modify_metadef_object: { key: 'modify_metadef_object', value: 'rule:metadef_default' },
    glance-add_metadef_object: { key: 'add_metadef_object', value: 'rule:metadef_default' },
    glance-delete_metadef_object: { key: 'delete_metadef_object', value: 'rule:metadef_default' },
    glance-list_metadef_resource_types: { key: 'list_metadef_resource_types', value: 'rule:metadef_default' },
    glance-get_metadef_resource_type: { key: 'get_metadef_resource_type', value: 'rule:metadef_default' },
    glance-add_metadef_resource_type_association: { key: 'add_metadef_resource_type_association', value: 'rule:metadef_default' },
    glance-remove_metadef_resource_type_association: { key: 'remove_metadef_resource_type_association', value: 'rule:metadef_default' },
    glance-get_metadef_property: { key: 'get_metadef_property', value: 'rule:metadef_default' },
    glance-get_metadef_properties: { key: 'get_metadef_properties', value: 'rule:metadef_default' },
    glance-modify_metadef_property: { key: 'modify_metadef_property', value: 'rule:metadef_default' },
    glance-add_metadef_property: { key: 'add_metadef_property', value: 'rule:metadef_default' },
    glance-remove_metadef_property: { key: 'remove_metadef_property', value: 'rule:metadef_default' },
    glance-get_metadef_tag: { key: 'get_metadef_tag', value: 'rule:metadef_default' },
    glance-get_metadef_tags: { key: 'get_metadef_tags', value: 'rule:metadef_default' },
    glance-modify_metadef_tag: { key: 'modify_metadef_tag', value: 'rule:metadef_default' },
    glance-add_metadef_tag: { key: 'add_metadef_tag', value: 'rule:metadef_default' },
    glance-add_metadef_tags: { key: 'add_metadef_tags', value: 'rule:metadef_default' },
    glance-delete_metadef_tag: { key: 'delete_metadef_tag', value: 'rule:metadef_default' },
    glance-delete_metadef_tags: { key: 'delete_metadef_tags', value: 'rule:metadef_default' }
  }
    Note

    You must configure all metadef policies to use rule:metadeta_default.

  3. Include the new policy file in the deployment command with the -e option when you deploy the overcloud: 

    $ openstack overcloud deploy -e open-up-glance-api-metadef.yaml

18.6. Changing the SSL/TLS cipher and rules for HAProxy

If you enabled SSL/TLS in the overcloud, consider hardening the SSL/TLS ciphers and rules that are used with the HAProxy configuration. By hardening the SSL/TLS ciphers, you help avoid SSL/TLS vulnerabilities, such as the POODLE vulnerability.

  1. Create a heat template environment file called tls-ciphers.yaml: 

    touch ~/templates/tls-ciphers.yaml

  2. Use the ExtraConfig hook in the environment file to apply values to the tripleo::haproxy::ssl_cipher_suite and tripleo::haproxy::ssl_options hieradata: 

    parameter_defaults:
  ExtraConfig:
    tripleo::haproxy::ssl_cipher_suite: 'DHE-RSA-AES128-CCM:DHE-RSA-AES256-CCM:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-CCM:ECDHE-ECDSA-AES256-CCM:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305'

    tripleo::haproxy::ssl_options: 'no-sslv3 no-tls-tickets'
    Note

    The cipher collection is one continuous line.

  3. Include the tls-ciphers.yaml environment file with the overcloud deploy command when deploying the overcloud: 

    openstack overcloud deploy --templates \
...
-e /home/stack/templates/tls-ciphers.yaml
...

18.7. Network policy

API endpoints will typically span multiple security zones, so you must pay particular attention to the separation of the API processes. For example, at the network design level, you can consider restricting access to specified systems only. See the guidance on security zones for more information.

With careful modeling, you can use network ACLs and IDS technologies to enforce explicit point-to-point communication between network services. As a critical cross domain service, this type of explicit enforcement works well for OpenStack’s message queue service.

To enforce policies, you can configure services, host-based firewalls (such as iptables), local policy (SELinux), and optionally global network policy.

18.8. Mandatory access controls

You should isolate API endpoint processes from each other and other processes on a machine. The configuration for those processes should be restricted to those processes by Discretionary Access Controls (DAC) and Mandatory Access Controls (MAC). The goal of these enhanced access controls is to aid in the containment of API endpoint security breaches.

18.9. API endpoint rate-limiting

Rate Limiting is a means to control the frequency of events received by a network based application. When robust rate limiting is not present, it can result in an application being susceptible to various denial of service attacks. This is especially true for APIs, which by their nature are designed to accept a high frequency of similar request types and operations.

It is recommended that all endpoints (especially public) are give an extra layer of protection, for example, using physical network design, a rate-limiting proxy, or web application firewall.

It is key that the operator carefully plans and considers the individual performance needs of users and services within their OpenStack cloud when configuring and implementing any rate limiting functionality.

NOTE For Red Hat OpenStack Platform deployments, all services are placed behind load balancing proxies.