Chapter 8. Director operator deployment scenario: Overcloud with external Ceph Storage

Procedure

1. The default QCOW2 image that you have downloaded from access.redhat.com does not use biosdev predictable network interface names. Modify the image with virt-customize to use biosdev predictable network interface names:

   $ sudo virt-customize -a <local path to image> --run-command 'sed -i -e "s/^\(kernelopts=.*\)net.ifnames=0 \(.*\)/\1\2/" /boot/grub2/grubenv'
   $ sudo virt-customize -a <local path to image> --run-command 'sed -i -e "s/^\(GRUB_CMDLINE_LINUX=.*\)net.ifnames=0 \(.*\)/\1\2/" /etc/default/grub' --truncate /etc/machine-id

2. Upload the image to OpenShift Virtualization with virtctl:

   $ virtctl image-upload dv openstack-base-img -n openstack --size=50Gi --image-path=<local path to image> --storage-class <storage class> --insecure

   Choose a suitable size for your environment, the minimum size is 50GB. For the --storage-class option, choose a storage class from your cluster. View a list of storage classes with the following command:

   $ oc get storageclass

3. When you create the OpenStackControlPlane resource and individual OpenStackVmSet resources, set the baseImageVolumeName parameter to the data volume name:

   ...
   spec:
     ...
     baseImageVolumeName: openstack-base-img
   ...

Procedure

1. Create the Secret resource:

   $ oc create secret generic git-secret -n openstack --from-file=git_ssh_identity=<path_to_private_SSH_key> --from-literal=git_url=<git_server_URL>

   The git-secret Secret resource contains two key-value pairs:

   git_ssh_identity

   The private key to access the Git repository. The --from-file option stores the content of the private SSH key file.

   git_url

   The SSH URL of the git repository to store the configuration. The --from-literal option stores the URL that you enter for this key.

Verification

1. View the Secret resource:

   $ oc get secret/git-secret -n openstack

Procedure

1. Convert your chosen password to a base64 value:

   $ echo -n "p@ssw0rd!" | base64
   cEBzc3cwcmQh

   Note

   The -n option removes the trailing newline from the echo output.

2. Create a file named openstack-userpassword.yaml on your workstation. Include the following resource specification for the Secret in the file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: userpassword
     namespace: openstack
   data:
     NodeRootPassword: "cEBzc3cwcmQh"

   Set the NodeRootPassword parameter to your base64 encoded password.

3. Create the userpassword Secret:

   $ oc create -f openstack-userpassword.yaml -n openstack

Procedure

1. Create a file named osnetconfig.yaml on your workstation. Include the resource specification for the control plane network, which is named ctlplane. For example, the specification for a control plane that uses a Linux bridge connected to the enp6s0 Ethernet device on each worker node is as follows:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig
   spec:
     attachConfigurations:
       br-osp:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp6s0
               description: Linux bridge with enp6s0 as a port
               name: br-osp
               state: up
               type: linux-bridge
               mtu: 1500
     # optional DnsServers list
     dnsServers:
     - 192.168.25.1
     # optional DnsSearchDomains list
     dnsSearchDomains:
     - osptest.test.metalkube.org
     - some.other.domain
     # DomainName of the OSP environment
     domainName: osptest.test.metalkube.org
     networks:
     - name: Control
       nameLower: ctlplane
       subnets:
       - name: ctlplane
         ipv4:
           allocationEnd: 172.22.0.250
           allocationStart: 172.22.0.100
           cidr: 172.22.0.0/24
           gateway: 172.22.0.1
         attachConfiguration: br-osp
     # optional: configure static mapping for the networks per nodes. If there is none, a random gets created
     reservations:
       controller-0:
         ipReservations:
           ctlplane: 172.22.0.120
       compute-0:
         ipReservations:
           ctlplane: 172.22.0.140

   Set the following values in the networks specification:

   name

   Set to the name of the control plane network, which is Control.

   nameLower

   Set to the lower name of the control plane network, which is ctlplane.

   subnets

   Set the subnet specifications.

   subnets.name

   Set the name of the control plane subnet, which is ctlplane.

   subnets.attachConfiguration

   Set the reference to which of the attach configuration should be used.

   subnets.ipv4

   Details of the ipv4 subnet with allocationStart, allocationEnd, cidr, gateway and optional list of routes (with destination and nexthop)

   For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstacknetconfig CRD:

   $ oc describe crd openstacknetconfig

   Save the file when you have finished configuring the network specification.

2. Create the control plane network:

   $ oc create -f osnetconfig.yaml -n openstack

Verification

1. View the resource for the control plane network:

   $ oc get openstacknetconfig/openstacknetconfig

Procedure

1. Create a file for your network configuration. Include the resource specification for the VLAN network. For example, the specification for internal API, storage, storage mgmt, tenant, and external network that manages VLAN-tagged traffic over Linux bridges br-ex and br-osp connected to the enp6s0 and enp7s0 Ethernet device on each worker node is as follows:

   kind: OpenStackNetConfig
   metadata:
     name: openstacknetconfig
   spec:
     attachConfigurations:
       br-osp:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp7s0
               description: Linux bridge with enp7s0 as a port
               name: br-osp
               state: up
               type: linux-bridge
               mtu: 1500
       br-ex:
         nodeNetworkConfigurationPolicy:
           nodeSelector:
             node-role.kubernetes.io/worker: ""
           desiredState:
             interfaces:
             - bridge:
                 options:
                   stp:
                     enabled: false
                 port:
                 - name: enp6s0
               description: Linux bridge with enp6s0 as a port
               name: br-ex
               state: up
               type: linux-bridge
               mtu: 1500
     # optional DnsServers list
     dnsServers:
     - 172.22.0.1
     # optional DnsSearchDomains list
     dnsSearchDomains:
     - osptest.test.metalkube.org
     - some.other.domain
     # DomainName of the OSP environment
     domainName: osptest.test.metalkube.org
     networks:
     - name: Control
       nameLower: ctlplane
       subnets:
       - name: ctlplane
         ipv4:
           allocationEnd: 172.22.0.250
           allocationStart: 172.22.0.10
           cidr: 172.22.0.0/24
           gateway: 172.22.0.1
         attachConfiguration: br-osp
     - name: InternalApi
       nameLower: internal_api
       mtu: 1350
       subnets:
       - name: internal_api
         attachConfiguration: br-osp
         vlan: 20
         ipv4:
           allocationEnd: 172.17.0.250
           allocationStart: 172.17.0.10
           cidr: 172.17.0.0/24
     - name: External
       nameLower: external
       subnets:
       - name: external
         ipv4:
           allocationEnd: 10.0.0.250
           allocationStart: 10.0.0.10
           cidr: 10.0.0.0/24
           gateway: 10.0.0.1
         attachConfiguration: br-ex
     - name: Storage
       nameLower: storage
       mtu: 1500
       subnets:
       - name: storage
         ipv4:
           allocationEnd: 172.18.0.250
           allocationStart: 172.18.0.10
           cidr: 172.18.0.0/24
         vlan: 30
         attachConfiguration: br-osp
     - name: StorageMgmt
       nameLower: storage_mgmt
       mtu: 1500
       subnets:
       - name: storage_mgmt
         ipv4:
           allocationEnd: 172.19.0.250
           allocationStart: 172.19.0.10
           cidr: 172.19.0.0/24
         vlan: 40
         attachConfiguration: br-osp
     - name: Tenant
       nameLower: tenant
       vip: False
       mtu: 1500
       subnets:
       - name: tenant
         ipv4:
           allocationEnd: 172.20.0.250
           allocationStart: 172.20.0.10
           cidr: 172.20.0.0/24
         vlan: 50
         attachConfiguration: br-osp

   When you use VLAN for network isolation with linux-bridge the following happens:

   • The director Operator creates a Node Network Configuration Policy for the bridge interface specified in the resource, which uses nmstate to configure the bridge on worker nodes.
   • The director Operator creates a Network Attach Definition for each network, which defines the Multus CNI plugin configuration. When you specify the VLAN ID on the Network Attach Definition, the Multus CNI plugin enables vlan-filtering on the bridge.
   • The director Operator attaches a dedicated interface for each network on a virtual machine. This means that the network template for the OpenStackVMSet is a multi-NIC network template.

   Set the following values in the resource specification:

   metadata.name

   Set to the name of the OpenStackNetConfig.

   spec

   Set the network configuration for attaching the networks and the network specifics. For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstacknetconfig CRD:

   $ oc describe crd openstacknetconfig

   Save the file when you have finished configuring the network specification.

2. Create the network configuration:

   $ oc apply -f openstacknetconfig.yaml -n openstack

Verification

1. View the OpenStackNetConfig API and created child resources:

   $ oc get openstacknetconfig/openstacknetconfig -n openstack
   $ oc get openstacknetattachment -n openstack
   $ oc get openstacknet -n openstack

   If you see errors, check the underlying network-attach-definition and node network configuration policies:

   $ oc get network-attachment-definitions -n openstack
   $ oc get nncp

Procedure

1. Create a file named openstack-controller.yaml on your workstation. Include the resource specification for the Controller nodes. For example, the specification for a control plane that consists of 3 Controller nodes is as follows:

   apiVersion: osp-director.openstack.org/v1beta2
   kind: OpenStackControlPlane
   metadata:
     name: overcloud
     namespace: openstack
   spec:
     gitSecret: git-secret
     openStackClientNetworks:
           - ctlplane
           - internal_api
           - external
     openStackClientStorageClass: host-nfs-storageclass
     passwordSecret: userpassword
     virtualMachineRoles:
       Controller:
         roleName: Controller
         roleCount: 3
         networks:
           - ctlplane
           - internal_api
           - external
           - tenant
           - storage
           - storage_mgmt
         cores: 12
         memory: 64
         rootDisk:
           diskSize: 100
           baseImageVolumeName: openstack-base-img
           # storageClass must support RWX to be able to live migrate VMs
           storageClass: host-nfs-storageclass
           storageAccessMode:  ReadWriteMany
           # When using OpenShift Virtualization with OpenShift Container Platform Container Storage,
           # specify RBD block mode persistent volume claims (PVCs) when creating virtual machine disks.
           # With virtual machine disks, RBD block mode volumes are more efficient and provide better
           # performance than Ceph FS or RBD filesystem-mode PVCs.
           # To specify RBD block mode PVCs, use the 'ocs-storagecluster-ceph-rbd' storage class and
           # VolumeMode: Block.
           storageVolumeMode: Filesystem
         # optional configure additional discs to be attached to the VMs,
         # need to be configured manually inside the VMs where to be used.
         additionalDisks:
           - name: datadisk
             diskSize: 500
             storageClass: host-nfs-storageclass
             storageAccessMode:  ReadWriteMany
             storageVolumeMode: Filesystem
     openStackRelease: "16.2"

   Set the following values in the resource specification:

   metadata.name

   Set to the name of the overcloud control plane, which is overcloud.

   metadata.namespace

   Set to the director Operator namespace, which is openstack.

   spec

   Set the configuration for the control plane. For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstackcontrolplane CRD:

   $ oc describe crd openstackcontrolplane

   Save the file when you have finished configuring the control plane specification.

2. Create the control plane:

   $ oc create -f openstack-controller.yaml -n openstack

   Wait until OCP creates the resources related to OpenStackControlPlane resource.

   As a part of the OpenStackControlPlane resource, the director Operator also creates an OpenStackClient pod that you can access through a remote shell and run RHOSP commands.

Verification

1. View the resource for the control plane:

   $ oc get openstackcontrolplane/overcloud -n openstack

2. View the OpenStackVMSet resources to verify the creation of the control plane virtual machine set:

   $ oc get openstackvmsets -n openstack

3. View the virtual machine resources to verify the creation of the control plane virtual machines in OpenShift Virtualization:

   $ oc get virtualmachines

4. Test access to the openstackclient remote shell:

   $ oc rsh -n openstack openstackclient

Procedure

1. Create a directory for your custom templates:

   $ mkdir custom_templates

2. Create a directory for your custom environment files:

   $ mkdir custom_environment_files

Procedure

1. Navigate to the location of your custom templates:

   $ cd ~/custom_templates

2. Archive the templates into a tarball:

   $ tar -cvzf custom-config.tar.gz *.yaml

3. Create the tripleo-tarball-config ConfigMap and use the tarball as data:

   $ oc create configmap tripleo-tarball-config --from-file=custom-config.tar.gz -n openstack

Verification

1. View the ConfigMap:

   $ oc get configmap/tripleo-tarball-config -n openstack

Procedure

1. Create the heat-env-config ConfigMap and use the directory that contains the environment files as data:

   $ oc create configmap -n openstack heat-env-config --from-file=~/custom_environment_files/ --dry-run=client -o yaml | oc apply -f -

Verification

1. View the ConfigMap:

   $ oc get configmap/heat-env-config -n openstack

Procedure

1. Create a file named openstack-compute.yaml on your workstation. Include the resource specification for the Compute nodes. For example, the specification for 1 Compute node is as follows:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackBaremetalSet
   metadata:
     name: compute
     namespace: openstack
   spec:
     count: 1
     baseImageUrl: http://host/images/rhel-image-8.4.x86_64.qcow2
     deploymentSSHSecret: osp-controlplane-ssh-keys
     # If you manually created an OpenStackProvisionServer, you can use it here,
     # otherwise the director Operator will create one for you (with `baseImageUrl` as the image that it server)
     # to use with this OpenStackBaremetalSet
     # provisionServerName: openstack-provision-server
     ctlplaneInterface: enp2s0
     networks:
       - ctlplane
       - internal_api
       - tenant
       - storage
     roleName: Compute
     passwordSecret: userpassword

   Set the following values in the resource specification:

   metadata.name

   Set to the name of the Compute node bare metal set, which is overcloud.

   metadata.namespace

   Set to the director Operator namespace, which is openstack.

   spec

   Set the configuration for the Compute nodes. For descriptions of the values you can use in this section, view the specification schema in the custom resource definition for the openstackbaremetalset CRD:

   $ oc describe crd openstackbaremetalset

   Save the file when you have finished configuring the Compute node specification.

2. Create the Compute nodes:

   $ oc create -f openstack-compute.yaml -n openstack

Verification

1. View the resource for the Compute nodes:

   $ oc get openstackbaremetalset/compute -n openstack

2. View the bare metal machines that OpenShift manages to verify the creation of the Compute nodes:

   $ oc get baremetalhosts -n openshift-machine-api

Procedure

1. Create a file named openstack-config-generator.yaml on your workstation. Include the resource specification to generate the Ansible playbooks. For example, the specification to generate the playbooks is as follows:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackConfigGenerator
   metadata:
     name: default
     namespace: openstack
   spec:
     enableFencing: true
     gitSecret: git-secret
     imageURL: registry.redhat.io/rhosp-rhel8/openstack-tripleoclient:16.2
     heatEnvConfigMap: heat-env-config
     # List of heat environment files to include from tripleo-heat-templates/environments
     heatEnvs:
     - ssl/tls-endpoints-public-dns.yaml
     - ssl/enable-tls.yaml
     tarballConfigMap: tripleo-tarball-config

   Set the following values in the resource specification:

   metadata.name

   Set to the name of the Compute node bare metal set, by default default.

   metadata.namespace

   Set to the director Operator namespace, by default openstack.

   spec.enableFencing

   Enable the automatic creation of required heat environment files to enable fencing.

   Note

   Production OSP environments must have fencing enabled. Virtual machines running pacemaker require the fence-agents-kubevirt package.

   spec.gitSecret

   Set to the ConfigMap that contains the Git authentication credentials, by default git-secret.

   spec.heatEnvs

   A list of default tripleo environment files used to generate the playbooks.

   spec.heatEnvConfigMap

   Set to the ConfigMap that contains your custom environment files, by default heat-env-config.

   spec.tarballConfigMap

   Set to the ConfigMap that contains the tarball with your custom heat templates, by default tripleo-tarball-config.

   For more descriptions of the values you can use in the spec section, view the specification schema in the custom resource definition for the openstackconfiggenerator CRD:

   $ oc describe crd openstackconfiggenerator

   Save the file when you have finished configuring the Ansible config generator specification.

2. Create the Ansible config generator:

   $ oc create -f openstack-config-generator.yaml -n openstack

Verification

1. View the resource for the config generator:

   $ oc get openstackconfiggenerator/default -n openstack

Procedure

1. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

2. Change to the cloud-admin home directory:

   $ cd /home/cloud-admin

3. Create a playbook that uses the redhat_subscription modules to register your nodes. For example, the following playbook registers Controller nodes:

   ---
   - name: Register Controller nodes
     hosts: Controller
     become: yes
     vars:
       repos:
         - rhel-8-for-x86_64-baseos-eus-rpms
         - rhel-8-for-x86_64-appstream-eus-rpms
         - rhel-8-for-x86_64-highavailability-eus-rpms
         - ansible-2.9-for-rhel-8-x86_64-rpms
         - openstack-16.2-for-rhel-8-x86_64-rpms
         - fast-datapath-for-rhel-8-x86_64-rpms
     tasks:
       - name: Register system
         redhat_subscription:
           username: myusername
           password: p@55w0rd!
           org_id: 1234567
           release: 8.4
           pool_ids: 1a85f9223e3d5e43013e3d6e8ff506fd
       - name: Disable all repos
         command: "subscription-manager repos --disable *"
       - name: Enable Controller node repos
         command: "subscription-manager repos --enable {{ item }}"
         with_items: "{{ repos }}"

   This play contains the following three tasks:

   • Register the node.
   • Disable any auto-enabled repositories.
   • Enable only the repositories relevant to the Controller node. The repositories are listed with the repos variable.

4. Register the overcloud nodes to required repositories:

   ansible-playbook -i /home/cloud-admin/ctlplane-ansible-inventory ./rhsm.yaml

Procedure

1. Create a file named openstack-deployment.yaml on your workstation. Include the resource specification to the Ansible playbooks. For example:

   apiVersion: osp-director.openstack.org/v1beta1
   kind: OpenStackDeploy
   metadata:
     name: default
   spec:
     configVersion: n5fch96h548h75hf4hbdhb8hfdh676h57bh96h5c5h59hf4h88h…
     configGenerator: default

   Set the following values in the resource specification:

   metadata.name

   Set the name of the Compute node baremetal set, by default default.

   metadata.namespace

   Set to the diretor Operator namespace, by default openstack.

   spec.configVersion

   The config version/git hash of the playbooks to deploy.

   spec.configGenerator

   The name of the configGenerator.

   For more descriptions of the values you can use inthe spec section, view the specification schema in the custom resource definition of the openstackdeploy CRD:

   $ oc describe crd openstackdeploy

   Save the file when you have finished configuring the OpenStackDeploy specification.

2. Create the OpenStackDeploy resource:

   $ oc create -f openstack-deployment.yaml -n openstack

   As the deployment runs it creates a Kubernetes job to execute the Ansible playbooks. You can tail the logs of the job to watch the Ansible playbooks running:

   $ oc logs -f jobs/deploy-openstack-default

   Additionally, you can manually access the executed Ansible playbooks by logging into the openstackclient pod. In the /home/cloud-admin/work/directory you can find the ansible playbooks and the ansible.log file for the current deployment.