Procedure

1. Create the following pools in your Ceph cluster relevant to your environment:

   • volumes: Storage for OpenStack Block Storage (cinder)
   • images: Storage for OpenStack Image Storage (glance)
   • vms: Storage for instances
   • backups: Storage for OpenStack Block Storage Backup (cinder-backup)

   • metrics: Storage for OpenStack Telemetry Metrics (gnocchi)

     Use the following commands as a guide:

     [root@ceph ~]# ceph osd pool create volumes <_pgnum_>
     [root@ceph ~]# ceph osd pool create images <_pgnum_>
     [root@ceph ~]# ceph osd pool create vms <_pgnum_>
     [root@ceph ~]# ceph osd pool create backups <_pgnum_>
     [root@ceph ~]# ceph osd pool create metrics <_pgnum_>

     If your overcloud deploys the Shared File Systems (manila) backed by CephFS, also create CephFS data and metadata pools:

     [root@ceph ~]# ceph osd pool create manila_data PGNUM
     [root@ceph ~]# ceph osd pool create manila_metadata PGNUM

   Replace <_pgnum_> with the number of placement groups. Approximately 100 placement groups per OSD is the best practice. For example, the total number of OSDs multiplied by 100, divided by the number of replicas, osd pool default size. You can also use the Ceph Placement Groups (PGs) per Pool Calculator to determine a suitable value.

2. Create a client.openstack user in your Ceph cluster with the following capabilities:

   • cap_mgr: “allow *”
   • cap_mon: profile rbd

   • cap_osd: profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics

     Use the following command as a guide:

     [root@ceph ~]# ceph auth add client.openstack mgr 'allow *' mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics'

3. Note the Ceph client key created for the client.openstack user:

   [root@ceph ~]# ceph auth list
   ...
   [client.openstack]
   	key = AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==
   	caps mgr = "allow *"
   	caps mon = "profile rbd"
   	caps osd = "profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images, profile rbd pool=backups, profile rbd pool=metrics"
   ...

   The key value in the example, AQC+vYNXgDAgAhAAc8UoYt+OTz5uhV7ItLdwUw==, is your Ceph client key.

4. If your overcloud deploys the Shared File Systems service backed by CephFS, create the client.manila user in your Ceph cluster with the following capabilities:

   • cap_mds: allow *
   • cap_mgr: allow *
   • cap_mon: allow r, allow command "auth del", allow command "auth caps", allow command "auth get", allow command "auth get-or-create"

   • cap_osd: allow rw Use the following command as a guide:

     [root@ceph ~]# ceph auth add client.manila mon 'allow r, allow command "auth del", allow command "auth caps", allow command "auth get", allow command "auth get-or-create"' osd 'allow rw' mds 'allow *' mgr 'allow *'

5. Note the manila client name and the key value to use in overcloud deployment templates:

   [root@ceph ~]# ceph auth get-key client.manila
        AQDQ991cAAAAABAA0aXFrTnjH9aO39P0iVvYyg==

6. Note the file system ID of your Ceph Storage cluster. This value is specified with the fsid setting in the configuration file of your cluster in the [global] section:

   [global]
   fsid = 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
   ...

Procedure

1. Log in to the director host as the stack user.

2. Enter the following command to initialize your director configuration:

   $ source ~/stackrc

Procedure

1. Create an inventory file. Use the example node definition template, instackenv.json as a reference:

   {
       "nodes":[
           {
               "mac":[
                   "bb:bb:bb:bb:bb:bb"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.205"
           },
           {
               "mac":[
                   "cc:cc:cc:cc:cc:cc"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.206"
           },
           {
               "mac":[
                   "dd:dd:dd:dd:dd:dd"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.207"
           },
           {
               "mac":[
                   "ee:ee:ee:ee:ee:ee"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.208"
           }
           {
               "mac":[
                   "ff:ff:ff:ff:ff:ff"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.209"
           }
           {
               "mac":[
                   "gg:gg:gg:gg:gg:gg"
               ],
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"pxe_ipmitool",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.0.2.210"
           }
       ]
   }

2. Save the file to the home directory of the stack user: /home/stack/instackenv.json.

3. Initialize the stack user, then import the instackenv.json inventory file into director:

   $ source ~/stackrc
   $ openstack overcloud node import ~/instackenv.json

   The openstack overcloud node import command imports the inventory file and registers each node with the director.

4. Assign the kernel and ramdisk images to each node:

   $ openstack overcloud node configure <node>

   Result

   The nodes are registered and configured in director.

Procedure

1. Trigger hardware introspection to retrieve the hardware attributes of each node:

   $ openstack overcloud node introspect --all-manageable --provide

   • The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state.

   • The --provide option resets all nodes to an active state after introspection.

     Important

     Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes.

2. Retrieve a list of your nodes to identify their UUIDs:

   $ openstack baremetal node list

3. Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile.

   As an alternative to manual tagging, you can configure the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data.

   For example, to tag three nodes to use the control profile and another three nodes to use the compute profile, create the following profile options:

   $ ironic node-update 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 6faba1a9-e2d8-4b7c-95a2-c7fbdc12129a add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 5e3b2f50-fcd9-4404-b0a2-59d79924b38e add properties/capabilities='profile:control,boot_option:local'
   $ ironic node-update 484587b2-b3b3-40d5-925b-a26a2fa3036f add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d010460b-38f2-4800-9cc4-d69f0d067efe add properties/capabilities='profile:compute,boot_option:local'
   $ ironic node-update d930e613-3e14-44b9-8240-4f3559801ea6 add properties/capabilities='profile:compute,boot_option:local'

Procedure

1. Create a custom environment file:

   /home/stack/templates/ceph-config.yaml

2. Add a parameter_defaults: section to the file:

   parameter_defaults:

3. Use parameter_defaults to set all of the parameters that you want to override in /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml. You must set the following parameters at a minimum:

   • CephClientKey: the Ceph client key of your Ceph Storage cluster. This is the value of key you retrieved in Section 2.2, “Configuring the existing Ceph Storage cluster”. For example, AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==.
   • CephClusterFSID: the file system ID of your Ceph Storage cluster. This is the value of fsid in your Ceph Storage cluster configuration file, which you retrieved in Section 2.2, “Configuring the existing Ceph Storage cluster”. For example, 4b5c8c0a-ff60-454b-a1b4-9747aa737d19.

   • CephExternalMonHost: a comma-delimited list of the IPs of all MON hosts in your Ceph Storage cluster, for example, 172.16.1.7, 172.16.1.8.

     For example:

     parameter_defaults:
       CephClientKey: AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ==
       CephClusterFSID: 4b5c8c0a-ff60-454b-a1b4-9747aa737d19
       CephExternalMonHost: 172.16.1.7, 172.16.1.8

4. If necessary, override the default pool names or the name of the Red Hat OpenStack Platform client user to match your Ceph Storage cluster:

   • CephClientUserName: openstack
   • NovaRbdPoolName: vms
   • CinderRbdPoolName: volumes
   • GlanceRbdPoolName: images
   • CinderBackupRbdPoolName: backups
   • GnocchiRbdPoolName: metrics

5. If you are deploying the Shared File Systems service backed by CephFS, set the names of the data and metadata pools:

     ManilaCephFSDataPoolName: manila_data
     ManilaCephFSMetadataPoolName: manila_metadata

   Note

   Ensure that these names match the names of the pools you created.

6. Set the client key that you created for manila and the name of the Ceph user for that key:

     ManilaCephFSCephFSAuthId: 'manila'
     CephManilaClientKey: 'AQDQ991cAAAAABAA0aXFrTnjH9aO39P0iVvYyg=='

   Note

   The default client user name ManilaCephFSCephFSAuthId is manila, unless you overide it. CephManilaClientKey is always required.

7. You can also add overcloud parameters to your custom environment file. For example, to set vxlan as the neutron network type, add the following to parameter_defaults:

     NeutronNetworkType: vxlan

Procedure

1. Create an image with the Image service (glance).

   For more information about how to create an image, see Import an image in the Creating and Managing Images guide.

2. Record the glance image ID for later use.

3. Create a Compute (nova) instance.

   For more information about how to create an instance, see Launch an instance in the Creating and Managing Instances guide.

4. Record the nova instance ID for later use.

5. Create a Block Storage (cinder) volume.

   For more information about how to create a Block Storage volume, see Create a volume in the Storage Guide.

6. Record the cinder volume ID for later use.

Procedure

1. Source the undercloud credentials:

   [stack@undercloud-0 ~]$ source stackrc

2. List the available servers to retrieve the IP addresses of nodes on the system:

   (undercloud) [stack@undercloud-0 ~]$ openstack server list
   
   +---------------+----------------+---------------+
   | ID | Name | Status | Networks | Image | Flavor |
   +---------------+----------------+---------------+
   | d5a621bd-d109-41ae-a381-a42414397802 | compute-0 | ACTIVE | ctlplane=192.168.24.31 | overcloud-full | compute |
   | 496ab196-d6cb-447d-a118-5bafc5166cf2 | controller-0 | ACTIVE | ctlplane=192.168.24.37 | overcloud-full | controller |
   | c01e730d-62f2-426a-a964-b31448f250b3 | controller-2 | ACTIVE | ctlplane=192.168.24.55 | overcloud-full | controller |
   | 36df59b3-66f3-452e-9aec-b7e7f7c54b86 | controller-1 | ACTIVE | ctlplane=192.168.24.39 | overcloud-full | controller |
   | f8f00497-246d-4e40-8a6a-b5a60fa66483 | compute-1 | ACTIVE | ctlplane=192.168.24.10 | overcloud-full | compute |

3. Use SSH to log in to any Compute node:

   (undercloud) [stack@undercloud-0 ~]$ ssh heat-admin@192.168.24.31

4. Switch to the root user:

   [heat-admin@compute-0 ~]$ sudo su -

5. Confirm that the files /etc/ceph/ceph.conf and /etc/ceph/ceph.client.openstack.keyring exist:

   [root@compute-0 ~]# ls -l /etc/ceph/ceph.conf
   
   -rw-r--r--. 1 root root 1170 Sep 29 23:25 /etc/ceph/ceph.conf
   [root@compute-0 ~]# ls -l /etc/ceph/ceph.client.openstack.keyring
   
   -rw-------. 1 ceph ceph 253 Sep 29 23:25 /etc/ceph/ceph.client.openstack.keyring

6. Enter the following command to force the nova_compute container to use the rbd command to list the contents of the appropriate pool.

   # podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls vms

   The pool name must match the pool names of the images, VMs, and volumes that you created when you configured the Ceph Storage cluster. For more information, see Configuring the existing Ceph Storage cluster. The IDs of the image, Compute instance, and volume must match the IDs that you recorded in Section 4.1, “Gathering IDs”.

   Note

   The example command is prefixed with podman exec nova_compute because /usr/bin/rbd, which is provided by the ceph-common package, is not installed on overcloud nodes by default. However, it is available in the nova_compute container. The command lists block device images. For more information, see Listing the block device images in the Ceph Storage Block Device Guide.

   The following examples show how to confirm whether an ID for each service is present for each pool by using the IDs from Section 4.1, “Gathering IDs”.

   # podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls images | grep 4485d4c0-24c3-42ec-a158-4d3950fa020b
   
   # podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls vms | grep 64bcb731-e7a4-4dd5-a807-ee26c669482f
   
   # podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack ls volumes | grep aeac15e8-b67f-454f-9486-46b3d75daff4

Procedure

1. To shorten the amount of typing you must do in this procedure, log in to a Compute node and create an alias for the rbd command:

   # alias rbd="podman exec nova_compute /usr/bin/rbd --conf /etc/ceph/ceph.conf --keyring /etc/ceph/ceph.client.openstack.keyring --cluster ceph --id openstack"

2. Confirm that you can write test data to the pool as a new object:

   # rbd create --size 1024 vms/foo

3. Confirm that you can see the test data:

   # rbd ls vms | grep foo

4. Delete the test data:

   # rbd rm vms/foo