1. Create the osd-compute flavor:

   [stack@director ~]$ openstack flavor create --id auto --ram 2048 --disk 40 --vcpus 2 osd-compute
   [stack@director ~]$ openstack flavor set --property "capabilities:boot_option"="local" --property "capabilities:profile"="osd-compute" osd-compute

2. Create and populate a host definition file for the Ironic service to manage the nodes.

   1. Create the instackenv.json host definition file:

      [stack@director ~]$ touch ~/instackenv.json

   2. Add a definition block for each node between the nodes stanza square brackets ({"nodes": []}), using this template:

      {
        "pm_password": "IPMI_USER_PASSWORD",
        "name": "NODE_NAME",
        "pm_user": "IPMI_USER_NAME",
        "pm_addr": "IPMI_IP_ADDR",
        "pm_type": "pxe_ipmitool",
        "mac": [
                "NIC_MAC_ADDR"
               ],
        "arch": "x86_64",
        "capabilities": "node:_NODE_ROLE-INSTANCE_NUM_,boot_option:local"
      },

      Replace…​

      • IPMI_USER_PASSWORD with the IPMI password.
      • NODE_NAME with a descriptive name of the node. This is an optional parameter.
      • IPMI_USER_NAME with the IPMI user name that has access to power the node on or off.
      • IPMI_IP_ADDR with the IPMI IP address.
      • NIC_MAC_ADDR with the network card MAC address handling the PXE boot.

      • NODE_ROLE-INSTANCE_NUM with the node’s role, along with a node number. This solution uses two roles: control and osd-compute.

        Example

        {
          "nodes": [
             {
                 "pm_password": "AbC1234",
                 "name": "m630_slot1",
                 "pm_user": "ipmiadmin",
                 "pm_addr": "10.19.143.61",
                 "pm_type": "pxe_ipmitool",
                 "mac": [
                     "c8:1f:66:65:33:41"
                 ],
                 "arch": "x86_64",
                  "capabilities": "node:control-0,boot_option:local"
             },
             {
                 "pm_password": "AbC1234",
                 "name": "m630_slot2",
                 "pm_user": "ipmiadmin",
                 "pm_addr": "10.19.143.62",
                 "pm_type": "pxe_ipmitool",
                 "mac": [
                     "c8:1f:66:65:33:42"
                 ],
                 "arch": "x86_64",
                  "capabilities": "node:osd-compute-0,boot_option:local"
             },
             ... Continue adding node definition blocks for each node in the initial deployment here.
          ]
        }

        Note

        The osd-compute role is a custom role that is created in a later step. To predictably control node placement, add these nodes in order. For example:

        [stack@director ~]$ grep capabilities ~/instackenv.json
        	 "capabilities": "node:control-0,boot_option:local"
        	 "capabilities": "node:control-1,boot_option:local"
        	 "capabilities": "node:control-2,boot_option:local"
        	 "capabilities": "node:osd-compute-0,boot_option:local"
        	 "capabilities": "node:osd-compute-1,boot_option:local"
        	 "capabilities": "node:osd-compute-2,boot_option:local"

3. Import the nodes into the Ironic database:

   [stack@director ~]$ openstack baremetal import ~/instackenv.json

   1. Verify that the openstack baremetal import command populated the Ironic database with all the nodes:

      [stack@director ~]$ openstack baremetal node list

4. Assign the bare metal boot kernel and RAMdisk images to all the nodes:

   [stack@director ~]$ openstack baremetal configure boot

5. To start the nodes, collect their hardware data and store the information in the Ironic database, execute the following:

   [stack@director ~]$ openstack baremetal introspection bulk start

   Note

   Bulk introspection can take a long time to complete based on the number of nodes imported. Setting the inspection_runbench value to false in ~/undercloud.conf file will speed up the bulk introspection process, but it will not collect the sysbench and fio benchmark data will not be collected, which can be useful data for the RHOSP-d.

   1. Verify that the introspection process completes without errors for all the nodes:

      [stack@director ~]$ openstack baremetal introspection bulk status

1. Configure the root disk device to use the smallest root device:

   [stack@director ~]$ openstack baremetal configure boot --root-device=smallest

1. Configure the root disk device to use the disk’s by-path name:

   [stack@director ~]$ openstack baremetal configure boot --root-device=disk/by-path/pci-0000:00:1f.1-scsi-0:0:0:0

1. Verify the correct root disk device was set:

   openstack baremetal introspection data save NODE_NAME_or_UUID | jq .

   Replace…​

   NODE_NAME_or_UUID with the host name or UUID of the node.

Procedure

1. Set the node’s state to manage:

   openstack baremetal node manage $NODE_NAME

   Example

   [stack@director ~]$ openstack baremetal node manage osdcompute-0

2. Set the node’s state to provide:

   openstack baremetal node provide NODE_NAME

   Example

   [stack@director ~]$ openstack baremetal node provide osdcompute-0

3. Check the node status:

   openstack node list

Procedure

1. As the stack user, on the undercloud node, using the openstack overcloud container image prepare command to include the additional services.

   1. Include the following environment file using the -e option:

      • Ceph Storage Cluster : /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml

   2. Include the following --set options for Red Hat Ceph Storage:

      --set ceph_namespace

      Defines the namespace for the Red Hat Ceph Storage container image.

      --set ceph_image

      Defines the name of the Red Hat Ceph Storage container image. Use image name: rhceph-3-rhel7.

      --set ceph_tag

      Defines the tag to use for the Red Hat Ceph Storage container image. When --tag-from-label is specified, the versioned tag is discovered starting from this tag.

2. Run the image prepare command:

   Example

   [stack@director ~]$ openstack overcloud container image prepare \
     ...
     -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
     --set ceph_namespace=registry.access.redhat.com/rhceph \
     --set ceph_image=rhceph-3-rhel7 \
     --tag-from-label {version}-{release} \
     ...

   Note

   These options are passed in addition to any other options (…​) that need to be passed to the openstack overcloud container image prepare command.

Procedure

1. To pull the images directly from registry.access.redhat.com in the overcloud deployment, an environment file is required to specify the image parameters. The following command automatically creates this environment file:

   (undercloud) [stack@director ~]$ openstack overcloud container image prepare \
     --namespace=registry.access.redhat.com/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   Note

   Use the -e option to include any environment files for optional services.

2. This creates an overcloud_images.yaml environment file, which contains image locations, on the undercloud. Include this file with all future upgrade and deployment operations.

Procedure

1. Create a template to pull the images to the local registry:

   (undercloud) [stack@director ~]$ openstack overcloud container image prepare \
     --namespace=registry.access.redhat.com/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-images-file /home/stack/local_registry_images.yaml

   • Use the -e option to include any environment files for optional services.

     Note

     This version of the openstack overcloud container image prepare command targets the registry on the registry.access.redhat.com to generate an image list. It uses different values than the openstack overcloud container image prepare command used in a later step.

2. This creates a file called local_registry_images.yaml with the container image information. Pull the images using the local_registry_images.yaml file:

   (undercloud) [stack@director ~]$ sudo openstack overcloud container image upload \
     --config-file  /home/stack/local_registry_images.yaml \
     --verbose

   Note

   The container images consume approximately 10 GB of disk space.

3. Find the namespace of the local images. The namespace uses the following pattern:

   <REGISTRY_IP_ADDRESS>:8787/rhosp13

   Use the IP address of the undercloud, which you previously set with the local_ip parameter in the undercloud.conf file. Alternatively, you can also obtain the full namespace with the following command:

   (undercloud) [stack@director ~]$ docker images | grep -v redhat.com | grep -o '^.*rhosp13' | sort -u

4. Create a template for using the images in our local registry on the undercloud. For example:

   (undercloud) [stack@director ~]$ openstack overcloud container image prepare \
     --namespace=192.168.24.1:8787/rhosp13 \
     --prefix=openstack- \
     --tag-from-label {version}-{release} \
     --output-env-file=/home/stack/templates/overcloud_images.yaml

   • Use the -e option to include any environment files for optional services.
   • If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location: --set ceph_namespace, --set ceph_image, --set ceph_tag.
   Note

   This version of the openstack overcloud container image prepare command targets a Red Hat Satellite server. It uses different values than the openstack overcloud container image prepare command used in a previous step.

5. This creates an overcloud_images.yaml environment file, which contains image locations on the undercloud. Include this file with all future upgrade and deployment operations.

1. Retrieve hardware introspection data.
2. Calculate optimal CPU and memory constraints for Compute on hyper-converged nodes based on that data.
3. Autogenerate the necessary parameters to configure those constraints.

1. Creating a Directory for Custom Templates
2. Configuring the Overcloud Networks
3. Creating the Controller and ComputeHCI Roles
4. Configuring Red Hat Ceph Storage for the overcloud
5. Configuring the Overcloud Node Profile Layouts

1. Create new directories for the custom templates:

   [stack@director ~]$ mkdir -p ~/templates/nic-configs

1. Choose the Compute NIC configuration template applicable to the environment:

   • /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-vlans/compute.yaml
   • /usr/share/openstack-tripleo-heat-templates/network/config/single-nic-linux-bridge-vlans/compute.yaml
   • /usr/share/openstack-tripleo-heat-templates/network/config/multiple-nics/compute.yaml

   • /usr/share/openstack-tripleo-heat-templates/network/config/bond-with-vlans/compute.yaml

     Note

     See the README.md in each template’s respective directory for details about the NIC configuration.

2. Create a new directory within the ~/templates/ directory:

   [stack@director ~]$ touch ~/templates/nic-configs

3. Copy the chosen template to the ~/templates/nic-configs/ directory and rename it to compute-hci.yaml:

   Example

   [stack@director ~]$ cp /usr/share/openstack-tripleo-heat-templates/network/config/bond-with-vlans/compute.yaml ~/templates/nic-configs/compute-hci.yaml

4. Add following definition, if it does note already exist, in the parameters: section of the ~/templates/nic-configs/compute-hci.yaml file:

   StorageMgmtNetworkVlanID:
       default: 40
       description: Vlan ID for the storage mgmt network traffic.
       type: number

5. Map StorageMgmtNetworkVlanID to a specific NIC on each node. For example, if you chose to trunk VLANs to a single NIC (single-nic-vlans/compute.yaml), then add the following entry to the network_config: section of ~/templates/nic-configs/compute-hci.yaml:

   type: vlan
   device: em2
   mtu: 9000
   use_dhcp: false
   vlan_id: {get_param: StorageMgmtNetworkVlanID}
   addresses:
     -
       ip_netmask: {get_param: StorageMgmtIpSubnet}

   Important

   Red Hat recommends setting the mtu to 9000, when mapping a NIC to StorageMgmtNetworkVlanID. This MTU setting provides measurable performance improvement to the performance of Red Hat Ceph Storage. For more details, see Configuring Jumbo Frames in the Red Hat OpenStack Platform Advanced Overcloud Customization guide.

6. Create a new file in the custom templates directory:

   [stack@director ~]$ touch ~/templates/network.yaml

7. Open and edit the network.yaml file.

   1. Add the resource_registry section:

      resource_registry:

   2. Add the following two lines under the resource_registry: section:

      OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller-nics.yaml
      OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-nics.yaml

      These two lines point the RHOSP services to the network configurations of the Controller/Monitor and Compute/OSD nodes respectively.

   3. Add the parameter_defaults section:

      parameter_defaults:

   4. Add the following default parameters for the Neutron bridge mappings for the tenant network:

      NeutronBridgeMappings: 'datacentre:br-ex,tenant:br-tenant'
      NeutronNetworkType: 'vxlan'
      NeutronTunnelType: 'vxlan'
      NeutronExternalNetworkBridge: "''"

      This defines the bridge mappings assigned to the logical networks and enables the tenants to use vxlan.

   5. The two TripleO Heat templates referenced in step 2b requires parameters to define each network. Under the parameter_defaults section add the following lines:

      # Internal API used for private OpenStack Traffic
      InternalApiNetCidr: IP_ADDR_CIDR
      InternalApiAllocationPools: [{'start': 'IP_ADDR_START', 'end': 'IP_ADDR_END'}]
      InternalApiNetworkVlanID: VLAN_ID
      
      # Tenant Network Traffic - will be used for VXLAN over VLAN
      TenantNetCidr: IP_ADDR_CIDR
      TenantAllocationPools: [{'start': 'IP_ADDR_START', 'end': 'IP_ADDR_END'}]
      TenantNetworkVlanID: VLAN_ID
      
      # Public Storage Access - Nova/Glance <--> Ceph
      StorageNetCidr: IP_ADDR_CIDR
      StorageAllocationPools: [{'start': 'IP_ADDR_START', 'end': 'IP_ADDR_END'}]
      StorageNetworkVlanID: VLAN_ID
      
      # Private Storage Access - Ceph cluster/replication
      StorageMgmtNetCidr: IP_ADDR_CIDR
      StorageMgmtAllocationPools: [{'start': 'IP_ADDR_START', 'end': 'IP_ADDR_END'}]
      StorageMgmtNetworkVlanID: VLAN_ID
      
      # External Networking Access - Public API Access
      ExternalNetCidr: IP_ADDR_CIDR
      
      # Leave room for floating IPs in the External allocation pool (if required)
      ExternalAllocationPools: [{'start': 'IP_ADDR_START', 'end': 'IP_ADDR_END'}]
      
      # Set to the router gateway on the external network
      ExternalInterfaceDefaultRoute: IP_ADDRESS
      
      # Gateway router for the provisioning network (or undercloud IP)
      ControlPlaneDefaultRoute: IP_ADDRESS
      
      # The IP address of the EC2 metadata server, this is typically the IP of the undercloud
      EC2MetadataIp: IP_ADDRESS
      
      # Define the DNS servers (maximum 2) for the Overcloud nodes
      DnsServers: ["DNS_SERVER_IP","DNS_SERVER_IP"]

      Replace…​

      • IP_ADDR_CIDR with the appropriate IP address and net mask (CIDR).
      • IP_ADDR_START with the appropriate starting IP address.
      • IP_ADDR_END with the appropriate ending IP address.
      • IP_ADDRESS with the appropriate IP address.
      • VLAN_ID with the appropriate VLAN identification number for the corresponding network.

      • DNS_SERVER_IP with the appropriate IP address for defining two DNS servers, separated by a comma (,).

        See the appendix for an example network.yaml file.

1. Generate a custom roles_data_custom.yaml file that includes the Controller and the ComputeHCI:

   [stack@director ~]$ openstack overcloud roles generate -o ~/custom-templates/roles_data_custom.yaml Controller ComputeHCI

1. Open for editing the ~/templates/ceph.yaml file.

   1. To use the BlueStore object store backend, update the following lines under the CephAnsibleExtraConfig section:

      Example

      CephAnsibleExtraConfig:
        osd_scenario: lvm
        osd_objectstore: bluestore

   2. Update the following options under the parameter_defaults section:

      Example

      parameter_defaults:
          CephPoolDefaultSize: 3
          CephPoolDefaultPgNum: NUM
          CephAnsibleDisksConfig:
            osd_scenario: lvm
            osd_objectstore: bluestore
            devices:
              - /dev/sda
              - /dev/sdb
              - /dev/sdc
              - /dev/sdd
              - /dev/nvme0n1
              - /dev/sde
              - /dev/sdf
              - /dev/sdg
              - /dev/nvme1n1
          CephAnsibleExtraConfig:
            osd_scenario: lvm
            osd_objectstore: bluestore
            ceph_osd_docker_cpu_limit: 2
            is_hci: true
      
          CephConfigOverrides:
            osd_recovery_op_priority: 3
            osd_recovery_max_active: 3
            osd_max_backfills: 1

      Replace…​

      NUM with the calculated values from the Ceph PG calculator.

      For this example, the following Compute/OSD node disk configuration is being used:

      • OSD : 12 x 1TB SAS disks presented as /dev/[sda, sdb, …​, sdg] block devices
      • OSD WAL and DB devices : 2 x 400GB NVMe SSD disks presented as /dev/[nvme0n1, nvme1n1] block devices

1. Create the layout.yaml file in the custom templates directory:

   [stack@director ~]$ touch ~/templates/layout.yaml

2. Open the layout.yaml file for editing.

   1. Add the resource registry section by adding the following line:

      resource_registry:

   2. Add the following lines under the resource_registry section for configuring the Controller and ComputeHCI roles to use a pool of IP addresses:

        OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
        OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml
        OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
        OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml
      
        OS::TripleO::ComputeHCI::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
        OS::TripleO::ComputeHCI::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml
        OS::TripleO::ComputeHCI::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
        OS::TripleO::ComputeHCI::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml

   3. Add a new section for the parameter defaults called parameter_defaults and include the following parameters underneath this section:

      parameter_defaults:
        NtpServer: NTP_IP_ADDR
        ControllerHostnameFormat: 'controller-%index%'
        ComputeHCIHostnameFormat: 'compute-hci-%index%'
        ControllerCount: 3
        ComputeHCICount: 3
        OvercloudComputeFlavor: compute
        OvercloudComputeHCIFlavor: osd-compute

      Replace…​

      NTP_IP_ADDR with the IP address of the NTP source. Time synchronization is very important!

      Example

      parameter_defaults:
        NtpServer: 10.5.26.10
        ControllerHostnameFormat: 'controller-%index%'
        ComputeHCIHostnameFormat: 'compute-hci-%index%'
        ControllerCount: 3
        ComputeHCICount: 3
        OvercloudComputeFlavor: compute
        OvercloudComputeHCIFlavor: osd-compute

      The value of 3 for the ControllerCount and ComputeHCICount parameters means three Controller/Monitor nodes and three Compute/OSD nodes will be deployed.

   4. Under the parameter_defaults section, add a two scheduler hints, one called ControllerSchedulerHints and the other called ComputeHCISchedulerHints. Under each scheduler hint, add the node name format for predictable node placement, as follows:

        ControllerSchedulerHints:
          'capabilities:node': 'control-%index%'
        ComputeHCISchedulerHints:
          'capabilities:node': 'osd-compute-%index%'

   5. Under the parameter_defaults section, add the required IP addresses for each node profile, for example:

      Example

        ControllerIPs:
          internal_api:
            - 192.168.2.200
            - 192.168.2.201
            - 192.168.2.202
          tenant:
            - 192.168.3.200
            - 192.168.3.201
            - 192.168.3.202
          storage:
            - 172.16.1.200
            - 172.16.1.201
            - 172.16.1.202
          storage_mgmt:
            - 172.16.2.200
            - 172.16.2.201
            - 172.16.2.202
      
        ComputeHCIIPs:
          internal_api:
            - 192.168.2.203
            - 192.168.2.204
            - 192.168.2.205
          tenant:
            - 192.168.3.203
            - 192.168.3.204
            - 192.168.3.205
          storage:
            - 172.16.1.203
            - 172.16.1.204
            - 172.16.1.205
          storage_mgmt:
            - 172.16.2.203
            - 172.16.2.204
            - 172.16.2.205

      From this example, node control-0 would have the following IP addresses: 192.168.2.200, 192.168.3.200, 172.16.1.200, and 172.16.2.200.

Procedure

1. Run the following command to verify all nodes are powered off, and available:

   [stack@director ~]$ openstack baremetal node list

Procedure

1. Generate the fencing Heat environment file:

   [stack@director ~]$ openstack overcloud generate fencing --ipmi-lanplus instackenv.json --output fencing.yaml

2. Include the fencing.yaml file with the openstack overcloud deploy command.

Procedure

1. Run the following command:

   [stack@director ~]$ time openstack overcloud deploy \
   --templates /usr/share/openstack-tripleo-heat-templates \
   --stack overcloud \
   -p /usr/share/openstack-tripleo-heat-templates/plan-samples/plan-environment-derived-params.yaml
   -r /home/stack/templates/roles_data_custom.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
   -e /home/stack/templates/overcloud_images.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
   -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
   -e ~/templates/network.yaml \
   -e ~/templates/ceph.yaml \
   -e ~/templates/layout.yaml
   -e /home/stack/fencing.yaml

   Command Details

   • The time command is used to tell you how long the deployment takes.
   • The openstack overcloud deploy command does the actual deployment.
   • Replace $NTP_IP_ADDR with the IP address of the NTP source. Time synchronization is very important!
   • The --templates argument uses the default directory (/usr/share/openstack-tripleo-heat-templates/) containing the TripleO Heat templates to deploy.
   • The -p argument points to the plan environment file for HCI deployments. See the Reserving CPU and memory resources for hyperconverged nodes section for more details.
   • The -r argument points to the roles file and overrides the default role_data.yaml file.
   • The -e argument points to an explicit template file to use during the deployment.
   • The puppet-pacemaker.yaml file configures the controller node services in a highly available pacemaker cluster.
   • The storage-environment.yaml file configures Ceph as a storage backend, whose parameter_defaults are passed by the custom template, ceph.yaml.
   • The network-isolation.yaml file configures network isolation for different services, whose parameters are passed by the custom template, network.yaml. This file will be created automatically when the deployment starts.
   • The network.yaml file is explained in Configuring the overcloud networks section for more details.
   • The ceph.yaml file is explained in Setting the Red Hat Ceph Storage parameters section for more details.
   • The compute.yaml file is explained in Changing Nova reserved memory and CPU allocations section for more details.
   • The layout.yaml file is explained in Configuring the overcloud node profile layouts section for more details.

   • The fencing.yaml file is explained in Configuring the controller for Pacemaker fencing section for more details.

     Important

     The order of the arguments matters. The custom template files will override the default template files.

     Note

     Optionally, add the --rhel-reg, --reg-method, --reg-org options, if you want to use the Red Hat OpenStack Platform director (RHOSP-d) node as a software repository for package installations.

2. Wait for the overcloud deployment to finish.

Procedure

1. Watch the deployment process and look for failures:

   [stack@director ~]$ heat resource-list -n5 overcloud | egrep -i 'fail|progress'

   Example output from a successful overcloud deployment:

   2016-12-20 23:25:04Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully
   
    Stack overcloud CREATE_COMPLETE
   
   Started Mistral Workflow. Execution ID: aeca4d71-56b4-4c72-a980-022623487c05
   /home/stack/.ssh/known_hosts updated.
   Original contents retained as /home/stack/.ssh/known_hosts.old
   Overcloud Endpoint: http://10.19.139.46:5000/v2.0
   Overcloud Deployed

2. After the deployment finishes, view the IP addresses for the overcloud nodes:

   [stack@director ~]$ openstack server list