Chapter 24. OSD migration from FileStore to BlueStore

After you complete and verify the upgrade process, you must migrate your FileStore OSDs to BlueStore. You must complete the migration one node at a time. The following procedure uses ceph-ansible to complete the migration. This procedure only applies if the Ceph cluster is deployed by director.

24.1. Checking that your cluster runs FileStore and therefore requires migration

Procedure

  1. Log in as the heat-admin user on a node with Ceph MON containers, such as Controller nodes or standalone Ceph MON nodes. For example, in a standard overcloud deployment, overcloud-controller-1 uses Ceph MON containers.
  2. Query the Ceph cluster to see what driver is in use by the OSDs:

    [heat-admin@overcloud-controller-1 ~]$ sudo -i
    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
    [root@overcloud-controller-1 ~]#
  3. If any line returns "objectstore": "filestore", that node requires OSD migration.
Warning

The migration time can vary depending on the size of your cluster. If you have a very large cluster, the migration time is proportional to the number of OSDs in that cluster and the amount of data stored. Ensure that you complete the migration as soon as possible so that your environment is not in a mixed architecture scenario, which can impact performance.

Warning

Because managing FileStore-based OSDs with Red Hat Ceph Storage (RHCS) 4 versions of ceph-ansible is not supported, complete the migration before you run any stack updates.

24.2. Migrating OSDs from FileStore to BlueStore

To migrate from FileStore to BlueStore, director uses Ansible to delete and recreate all OSDs on the node. Director performs a capacity check before the migration process. Finally, director redploys the OSDs with the BlueStore back end.

Prerequisites

  • A healthy and running Red Hat Ceph Storage (RHCS) 4 cluster. You can check the cluster by entering the following command in a ceph MON container on a Controller or Standalone Ceph MON node:

    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c  "ceph  -s"

Procedure

  1. Ensure that osd_objectstore in the CephAnsibleDisksConfig parameter does not default to filestore. If the osd_objectstore parameter is present in any of your custom heat environment files, you must define the value bluestore explicitly or remove it:

    parameter_defaults:
      CephAnsibleDisksConfig:
        devices:
          - /dev/sdb
          - /dev/sdc
          - /dev/sdd
        osd_scenario: lvm
       osd_objectstore: bluestore
    Note

    If you have any specific FileStore configuration with, for example, journals, ensure that you update the configuration accordingly. For more information about advanced configurations, see Mapping the Ceph Storage Node Disk Layout in the Deploying an overcloud with containerized Red Hat Ceph guide.

  2. Log in to the undercloud as the stack user.
  3. Enter the openstack overcloud external-upgrade run command with the ceph_fstobs tag. Replace <NODE_NAME> with the name of the Ceph OSD node you want to upgrade. You can use the openstack server list command to find the node name.

    [stack@undercloud ~] $ openstack overcloud external-upgrade run --tags ceph_fstobs -e ceph_ansible_limit=<NODE_NAME> | tee oc-fstobs.log
  4. Log in to a node that has Ceph MON services and query the Ceph cluster to check the status of the OSDs of the node you have upgraded. Before you can start the migration of the next OSD node, you must ensure that the one you have upgraded is successfully migrated:

    [heat-admin@overcloud-controller-1 ~]$ sudo -i
    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c '.[] | select(.hostname == "<NODE_NAME>") | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
    [root@overcloud-controller-1 ~]# exit

    Replace <NODE_NAME> with the name of the node that was migrated. If the result shows that the OSDs use BlueStore, its migration is successful.

  5. Optional: To view additional details about a specific OSD, enter the following command:

    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph osd metadata <ID>"

    Replace <ID> with the ID of the OSD you want to query.

  6. Before you can start the migration process on the next node, you must wait for the cluster to synchronize.

    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c  "ceph  -s"
    Review the command output and ensure that the health of the cluster is `HEALTH_OK` and the PGs are in the `active+clean` state.
  7. Before you can start the migration process on the next node, you must wait for the cluster rebalancing process to complete. To follow the status, run the following command:

    [heat-admin@overcloud-controller-0 ~]$ sudo podman exec ceph-mon-<NODE_NAME> ceph -w

    Replace <NODE_NAME> with the name of the node that was migrated.

  8. Repeat the migration process for each node in the storage cluster.

For more information about migration from FileStore to BlueStore, see BlueStore in the Red Hat Ceph Storage Administration Guide.

24.3. Verifying your FileStore to BlueStore migration

You can check the status of an OSD to ensure that you have successfully completed the migration.

Procedure

  1. Log in as the heat-admin user to a ceph-mon container that is hosted on one of the Controller nodes.
  2. Query the Ceph cluster:

    [heat-admin@overcloud-controller-1 ~]$ sudo -i
    [root@overcloud-controller-1 ~]# podman exec -it ceph-mon-overcloud-controller-1 sh -c "ceph -f json osd metadata" | jq -c 'sort_by(.hostname) | .[] | ["host", .hostname, "osd_id", .id, "objectstore", .osd_objectstore]'
    [root@overcloud-controller-1 ~]#

If the configuration shows that all the OSDs across the cluster use BlueStore, the migration is successful.

Important

A recommended best practice is to run an idempotent stack update to ensure that the configuration definition and the actual configuration match. The stack update duration varies depending on the size of your system, so to reduce downtime you can plan to complete the migration during a maintenance window.