Chapter 5. Deploying storage at the edge
You can leverage Red Hat OpenStack Platform director to extend distributed compute node deployments to include distributed image management and persistent storage at the edge with the benefits of using Red Hat OpenStack Platform and Ceph Storage.
5.1. Limitations of distributed compute node (DCN) architecture
The following features are not currently supported for DCN architectures:
- Fast forward updates (FFU) on a distributed compute node architecture from Red Hat OpenStack Platform 13 to 16.
- Non-hyperconverged storage nodes at edge sites.
- Copying a volume snapshot between edge sites. You can work around this by creating an image from the volume and using glance to copy the image. Once the image is copied, you can create a volume from it.
- Migrating or retyping a volume between sites.
- Ceph Rados Gateway (RGW) at the edge.
- CephFS at the edge.
- Block storage (cinder) backup for edge volumes.
- Instance high availability (HA) at the edge sites.
- Live migration between edge sites or from the central location to edge sites. You can still live migrate instances within a site boundary.
- RBD mirroring between sites.
The ML2/OVN mechanism driver is available on DCN as a Technology Preview, and therefore using these solutions together is not fully supported by Red Hat. This feature should only be used with DCN for testing, and should not be deployed in a production environment. For more information about Technology Preview features, see Scope of Coverage Details.
The ML2/OVN mechanism driver is fully supported outside of DCN environments.
5.1.1. Requirements of storage edge architecture
- A copy of each image must exist in the Image service at the central location.
- Prior to creating an instance at an edge site, you must have a local copy of the image at that edge site.
centralrcauthentication file to schedule workloads at edge sites as well as at the central location. Authentication files that are automatically generated for edge sites are not needed.
- Images uploaded to an edge site must be copied to the central location before they can be copied to other edge sites.
- Use the Image service RBD driver for all edge sites. Mixed architecture is not supported.
- Multistack must be used with a single stack at each site.
- RBD must be the storage driver for the Image, Compute and Block Storage services.
For each site, you must assign the same value to the
Red Hat recommends leaving the
<role>HostnameFormatparameter as the default value:
%stackname%-<role>-%index%. If you do not include the
%stackname%prefix, your overcloud will use the same hostnames for distributed compute nodes in different stacks. Ensure your distributed compute nodes use the
%stackname%prefix to distinguish nodes from different edge sites. For example, if you deploy two edge sites named
dcn0and dcn1`, the stack name prefix helps you distinguish between
dcn1-distributedcompute-0when you run
openstack server liston the undercloud.
5.2. Deploying the central site
To deploy the Image service with multiple stores and Ceph Storage as the back end, complete the following steps:
- Hardware for a Ceph cluster at the hub and in each availability zone, or in each geographic location where storage services are required.
- You must deploy edge sites in a hyper converged architecture.
- Hardware for three Image Service servers at the hub and in each availability zone, or in each geographic location where storage services are required.
The following is an example deployment of two or more stacks:
One stack at the central, or hub location, called
One stack at an edge site called
Additional stacks deployed similarly to
dcn0, such as
dcn2, and so on.
The following procedure outlines the steps for the initial deployment of the central location.
The following steps detail the deployment commands and environment files associated with an example DCN deployment that uses the Image service with multiple stores. These steps do not include unrelated, but necessary, aspects of configuration, such as networking.
In the home directory, create directories for each stack that you plan to deploy.
mkdir /home/stack/central mkdir /home/stack/dcn0 mkdir /home/stack/dcn1
Set the name of the Ceph cluster, as well as configuration parameters relative to the available hardware. For more information, see Configuring Ceph with Custom Config Settings:
cat > /home/stack/central/ceph.yaml << EOF parameter_defaults: CephClusterName: central CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb CephPoolDefaultSize: 3 CephPoolDefaultPgNum: 128 EOF
Generate roles for the central location using roles appropriate for your environment:
openstack overcloud roles generate Compute Controller CephStorage \ -o ~/central/central_roles.yaml cat > /home/stack/central/role-counts.yaml << EOF parameter_defaults: ControllerCount: 3 ComputeCount: 2 CephStorage: 3 EOF
Generate an environment file
openstack tripleo container image prepare \ -e containers.yaml \ --output-env-file ~/central/central-images-env.yaml
Configure the naming conventions for your site in the
site-name.yamlenvironment file. The Nova availability zone and the Cinder storage availability zone must match:
cat > /home/stack/central/site-name.yaml << EOF parameter_defaults: NovaComputeAvailabilityZone: central ControllerExtraConfig: nova::availability_zone::default_schedule_zone: central NovaCrossAZAttach: false CinderStorageAvailabilityZone: central GlanceBackendID: central EOF
glance.yamltemplate with contents similar to the following:
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' GlanceBackendID: central CephClusterName: central
After you prepare all of the other templates, deploy the
openstack overcloud deploy \ --stack central \ --templates /usr/share/openstack-tripleo-heat-templates/ \ -r ~/central/central_roles.yaml \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \ -e ~/central/central-images-env.yaml \ -e ~/central/role-counts.yaml \ -e ~/central/site-name.yaml \ -e ~/central/ceph.yaml \ -e ~/central/glance.yaml
You must include heat templates for the configuration of networking in your
openstack overcloud deploy command. Designing for edge architecture requires spine and leaf networking. See Spine Leaf Networking for more details.
ceph-ansible.yaml file is configured with the following parameters:
- NovaEnableRbdBackend: true
- GlanceBackend: rbd
When you use these settings together, the glance.conf parameter
image_import_plugins is configured by heat to have a value
image_conversion, automating the conversion of QCOW2 images with commands such as
glance image-create-via-import --disk-format qcow2…
This is optimal for the Ceph RBD. If you want to disable image conversion, you can do so with the
parameter_defaults: GlanceImageImportPlugin: 
5.3. Deploying edge sites with storage
After you deploy the central site, build out the edge sites and ensure that each edge location connects primarily to its own storage backend, as well as to the storage back end at the central location.
A spine and leaf networking configuration should be included with this configuration, with the addition of the
storage_mgmt networks that ceph needs. For more information see Spine leaf networking.
You must have connectivity between the storage network at the central location and the storage network at each edge site so that you can move glance images between sites.
Ensure that the central location can communicate with the
osds at each of the edge sites. However, you should terminate the storage management network at site location boundaries, because the storage management network is used for OSD rebalancing.
Export stack information from the
centralstack. You must deploy the
centralstack before running this command:
openstack overcloud export \ --config-download-dir /var/lib/mistral/central/ \ --stack central \ --output-file ~/dcn-common/central-export.yamlNote
config-download-dirvalue defaults to
central_ceph_external.yamlfile. This environment file connects DCN sites to the central hub Ceph cluster, so the information is specific to the Ceph cluster deployed in the previous steps.
sudo -E openstack overcloud export ceph \ --stack central \ --config-download-dir /var/lib/mistral \ --output-file ~/dcn-common/central_ceph_external.yaml
~/dcn0/glance.yamlfile for glance configuration overrides:
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'dcn0 rbd glance store' GlanceBackendID: dcn0 GlanceMultistoreConfig: central: GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' CephClientUserName: 'openstack' CephClusterName: central
ceph.yamlfile with configuration parameters relative to the available hardware.
cat > /home/stack/dcn0/ceph.yaml << EOF parameter_defaults: CephClusterName: dcn0 CephAnsibleDisksConfig: osd_scenario: lvm osd_objectstore: bluestore devices: - /dev/sda - /dev/sdb CephPoolDefaultSize: 3 CephPoolDefaultPgNum: 128 EOF
For more information, see Mapping the Ceph Storage node disk layout.
Implement system tuning by using a file that contains the following parameters tuned to the requirements of you environment:
cat > /home/stack/dcn0/tuning.yaml << EOF parameter_defaults: CephAnsibleExtraConfig: is_hci: true CephConfigOverrides: osd_recovery_op_priority: 3 osd_recovery_max_active: 3 osd_max_backfills: 1 ## Set relative to your hardware: # DistributedComputeHCIParameters: # NovaReservedHostMemory: 181000 # DistributedComputeHCIExtraConfig: # nova::cpu_allocation_ratio: 8.2 EOF
Configure the naming conventions for your site in the
site-name.yamlenvironment file. The Nova availability zone and the Cinder storage availability zone must match. The
CinderVolumeClusterparameter is included when deploying an edge site with storage. This parameter is used when cinder-volume is deployed as active/active, which is required at edge sites. As a best practice, set the Cinder cluster name to match the availability zone:
cat > /home/stack/central/site-name.yaml << EOF parameter_defaults: ... NovaComputeAvailabilityZone: dcn0 NovaCrossAZAttach: false CinderStorageAvailabilityZone: dcn0 CinderVolumeCluster: dcn0
roles.yamlfile to be used for the dcn0 deployment, for example:
openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/roles_data.yaml
Set the number systems in each role by creating the
~/dcn0/roles-counts.yamlfile with the desired values for each role.
When using hyperconverged infrastructure (HCI), you must allocate three nodes to the DistributedComputeHCICount role to satisfy requirements for Ceph Mon and
parameter_defaults: ControllerCount: 0 ComputeCount: 0 DistributedComputeHCICount: 3 DistributedComputeHCIScaleOutCount: 1 # Optional DistributedComputeScaleOutCount: 1 # Optional
Retrieve the container images for the edge site:
openstack tripleo container image prepare \ --environment-directory dcn0 \ -r ~/dcn0/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ ... -e /home/stack/dcn-common/central-export.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ --output-env-file ~/dcn0/dcn0-images-env.yamlNote
You must include all environment files to be used for the deployment in the
openstack tripleo container image preparecommand.
Deploy the edge site:
openstack overcloud deploy \ --stack dcn0 \ --templates /usr/share/openstack-tripleo-heat-templates/ \ -r ~/dcn0/roles_data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/dcn-hci.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \ -e ~/dnc0/dcn0-images-env.yaml \ .... -e ~/dcn-common/central-export.yaml \ -e ~/dcn-common/central_ceph_external.yaml \ -e ~/dcn0/dcn_ceph_keys.yaml \ -e ~/dcn0/role-counts.yaml \ -e ~/dcn0/ceph.yaml \ -e ~/dcn0/site-name.yaml \ -e ~/dcn0/tuning.yaml \ -e ~/dcn0/glance.yamlNote
You must include heat templates for the configuration of networking in your
openstack overcloud deploycommand. Designing for edge architecture requires spine and leaf networking. See Spine Leaf Networking for more details.
5.4. Creating additional distributed compute node sites
A new distributed compute node (DCN) site has its own directory of YAML files on the undercloud. For more information, see Section 3.10, “Managing separate heat stacks”. This procedure contains example commands.
As the stack user on the undercloud, create a new directory for
$ cd ~ $ mkdir dcn1
Copy the existing
dcn0templates to the new directory and replace the
$ cp dcn0/ceph.yaml dcn1/ceph.yaml $ sed s/dcn0/dcn1/g -i dcn1/ceph.yaml $ cp dcn0/overrides.yaml dcn1/overrides.yaml $ sed s/dcn0/dcn1/g -i dcn1/overrides.yaml $ sed s/"0-ceph-%index%"/"1-ceph-%index%"/g -i dcn1/overrides.yaml $ cp dcn0/deploy.sh dcn1/deploy.sh $ sed s/dcn0/dcn1/g -i dcn1/deploy.sh
Review the files in the
dcn1directory to confirm that they suit your requirements.
Verify that your nodes are available and in
$ openstack baremetal node list
When your nodes are available, run the
$ bash dcn1/deploy.sh
5.5. Updating the central location
After you configure and deploy all of the edge sites using the sample procedure, update the configuration at the central location so that the central Image service can push images to the edge sites.
~/central/glance_update.yamlfile similar to the following. This example includes a configuration for two edge sites, dcn0 and dcn1:
parameter_defaults: GlanceEnabledImportMethods: web-download,copy-image GlanceBackend: rbd GlanceStoreDescription: 'central rbd glance store' CephClusterName: central GlanceBackendID: central GlanceMultistoreConfig: dcn0: GlanceBackend: rbd GlanceStoreDescription: 'dcn0 rbd glance store' CephClientUserName: 'openstack' CephClusterName: dcn0 GlanceBackendID: dcn0 dcn1: GlanceBackend: rbd GlanceStoreDescription: 'dcn1 rbd glance store' CephClientUserName: 'openstack' CephClusterName: dcn1 GlanceBackendID: dcn1
dcn_ceph.yamlfile. In the following example, this file configures the glance service at the central site as a client of the Ceph clusters of the edge sites,
sudo -E openstack overcloud export ceph \ --stack dcn0,dcn1 \ --config-download-dir /var/lib/mistral \ --output-file ~/central/dcn_ceph.yaml
Redeploy the central site using the original templates and include the newly created
openstack overcloud deploy \ --stack central \ --templates /usr/share/openstack-tripleo-heat-templates/ \ -r ~/central/central_roles.yaml \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/nova-az-config.yaml \ -e ~/central/central-images-env.yaml \ -e ~/central/role-counts.yaml \ -e ~/central/site-name.yaml -e ~/central/ceph.yaml \ -e ~/central/ceph_keys.yaml \ -e ~/central/glance.yaml \ -e ~/central/dcn_ceph_external.yaml
5.6. Deploying Red Hat Ceph Storage Dashboard on DCN
To deploy the Red Hat Ceph Storage Dashboard to the central location, see Adding the Red Hat Ceph Storage Dashboard to an overcloud deployment. These steps should be completed prior to deploying the central location.
To deploy Red Hat Ceph Storage Dashboard to edge locations, complete the same steps that you completed for central, however you must complete the following following:
Ensure that the
ManageNetworksparameter has a value of
falsein your templates for deploying the edge site. When you set
false, Edge sites will use the existing networks that were already created in the central stack:
parameter_defaults: ManageNetworks: false
- You must deploy your own solution for load balancing in order to create a high availability virtual IP. Edge sites do not deploy haproxy, nor pacemaker. When you deploy Red Hat Ceph Storage Dashboard to edge locations, the deployment is exposed on the storage network. The dashboard is installed on each of the three DistributedComputeHCI nodes with distinct IP addresses without a load balancing solution.
5.6.1. Creating a composable network for a Virtual IP
You can create an additional network to host virtual IP where the Ceph dashboard can be exposed. You must not be reusing network resources for multiple stacks. For more information on reusing network resources, see Reusing network resources in multiple stacks.
To create this additional network resource, use the provided
network_data_dashboard.yaml heat template. The name of the created network is
Log in to Red Hat OpenStack Platform Director as
DistributedComputeHCIDashboardrole and any other roles appropriate for your environment:
openstack overcloud roles generate DistributedComputeHCIDashboard -o ~/dnc0/roles.yaml
network_data_dashboard.yamlin the overcloud deploy command:
$ openstack overcloud deploy --templates \ -r ~/<dcn>/<dcn_site_roles>.yaml \ -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml \ -e <overcloud_environment_files> \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-dashboard.yaml
The deployment provides the three ip addresses where the dashboard is enabled on the storage network.
To confirm the dashboard is operational at the central location and that the data it displays from the Ceph cluster is correct, see Accessing Ceph Dashboard.
You can confirm that the dashboard is operating at an edge location through similar steps, however, there are exceptions as there is no load balancer at edge locations.
Retrieve dashboard admin login credentials specific to the selected stack from
Within the inventory specific to the selected stack,
/var/lib/mistral/<stackname>/ceph-ansible/inventory.yml, locate the DistributedComputeHCI role hosts list and save all three of the
storage_ipvalues. In the example below the first two dashboard IPs are 172.16.11.84 and 172.16.11.87:
DistributedComputeHCI: hosts: dcn1-distributed-compute-hci-0: ansible_host: 192.168.24.16 ... storage_hostname: dcn1-distributed-compute-hci-0.storage.localdomain storage_ip: 172.16.11.84 ... dcn1-distributed-compute-hci-1: ansible_host: 192.168.24.22 ... storage_hostname: dcn1-distributed-compute-hci-1.storage.localdomain storage_ip: 172.16.11.87
- You can check that the Ceph Dashboard is active at one of these IP addresses if they are accessible to you. These IP addresses are on the storage network and are not routed. If these IP addresses are not available, you must configure a load balancer for the three IP addresses that you get from the inventory to obtain a virtual IP address for verification.