Chapter 5. Adding file and object storage to an existing external OpenShift Container Storage cluster

When OpenShift Container Storage is configured in external mode, there are several ways to provide storage for persistent volume claims and object bucket claims.

  • Persistent volume claims for block storage are provided directly from the external Red Hat Ceph Storage cluster.
  • Persistent volume claims for file storage can be provided by adding a Metadata Server (MDS) to the external Red Hat Ceph Storage cluster.
  • Object bucket claims for object storage can be provided either by using the Multicloud Object Gateway or by adding the Ceph Object Gateway to the external Red Hat Ceph Storage cluster.

Use the following process to add file storage (using Metadata Servers) or object storage (using Ceph Object Gateway) or both to an external OpenShift Container Storage cluster that was initially deployed to provide only block storage.

Prerequisites

  • You have OpenShift Container Storage 4.7 installed and running on the OpenShift Container Platform version 4.7 or above. Also, the OpenShift Storage Cluster in external mode is in Ready state.
  • Your external Red Hat Ceph Storage cluster is configured with one or both of the following:

    • a Ceph Object Gateway (RGW) endpoint that can be accessed by the OpenShift Container Platform cluster for object storage
    • a Metadata Server (MDS) pool for file storage
  • Ensure that you know the parameters used with the ceph-external-cluster-details-exporter.py script during external OpenShift Container Storage cluster deployment.

Procedure

  1. Download the OpenShift Container Storage version of the ceph-external-cluster-details-exporter.py python script using the following command:

    oc get csv $(oc get csv -n openshift-storage | grep ocs-operator | awk '{print $1}') -n openshift-storage -o jsonpath='{.metadata.annotations.external\.features\.ocs\.openshift\.io/export-script}' | base64 --decode > ceph-external-cluster-details-exporter.py
  2. Update permission caps on the external Red Hat Ceph Storage cluster by running ceph-external-cluster-details-exporter.py on any client node in the external Red Hat Ceph Storage cluster. You may need to ask your Red Hat Ceph Storage administrator to do this.

    # python3 ceph-external-cluster-details-exporter.py --upgrade \
    --run-as-user=ocs-client-name \
    --rgw-pool-prefix rgw-pool-prefix
    --run-as-user
    The client name used during OpenShift Container Storage cluster deployment. Use the default client name client.healthchecker if a different client name was not set.
    --rgw-pool-prefix
    The prefix used for the Ceph Object Gateway pool. This can be omitted if the default prefix is used.
  3. Generate and save configuration details from the external Red Hat Ceph Storage cluster.

    1. Generate configuration details by running ceph-external-cluster-details-exporter.py on any client node in the external Red Hat Ceph Storage cluster.

      # python3 ceph-external-cluster-details-exporter.py --rbd-data-pool-name rbd-block-pool-name --monitoring-endpoint ceph-mgr-prometheus-exporter-endpoint --monitoring-endpoint-port ceph-mgr-prometheus-exporter-port --run-as-user ocs-client-name  --rgw-endpoint rgw-endpoint --rgw-pool-prefix rgw-pool-prefix
      --monitoring-endpoint
      The IP address of the active Ceph Manager that can be reached from the OpenShift Container Storage cluster.
      --monitoring-endpoint-port
      The port for the Ceph Manager Prometheus Exporter endpoint.
      --run-as-user
      The client name used during OpenShift Container Storage cluster deployment. Use the default client name client.healthchecker if a different client name was not set.
      --rgw-endpoint
      Provide this parameter to provision object storage through Ceph Object Gateway for OpenShift Container Storage. (optional parameter)
      --rgw-pool-prefix
      The prefix used for the Ceph Object Gateway pool. This can be omitted if the default prefix is used.

      User permissions are updated as shown:

      caps: [mgr] allow command config
      caps: [mon] allow r, allow command quorum_status, allow command version
      caps: [osd] allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index
      Note

      Ensure that all the parameters (including the optional arguments) except the Ceph Object Gateway details (if provided), are the same as what was used during the deployment of Openshift Container Storage in external mode.

    2. Save the output of the script in an external-cluster-config.json file.

      The following example output shows the generated configuration changes in bold text.

      [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "xxx.xxx.xxx.xxx:xxxx", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "<fs-id>", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "<user-key>"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "<user-key>"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "ceph-rbd"}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "xxx.xxx.xxx.xxx", "MonitoringPort": "xxxx"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "<user-key>"}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "<admin-key>"}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "<admin-key>"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs_data"}}, {"name": "ceph-rgw", "kind": "StorageClass", "data": {"endpoint": "xxx.xxx.xxx.xxx:xxxx", "poolPrefix": "default"}}]
  4. Upload the generated JSON file.

    1. Log in to the OpenShift web console.
    2. Click WorkloadsSecrets.
    3. Set project to openshift-storage.
    4. Click on rook-ceph-external-cluster-details.
    5. Click Actions (⋮) → Edit Secret
    6. Click Browse and upload the external-cluster-config.json file.
    7. Click Save.

Verification steps

  • Click OverviewHomePersistent Storage and verify the health of the OpenShift Container Storage cluster.
  • If you added a Metadata Server for file storage:

    1. Click WorkloadsPods and verify that csi-cephfsplugin-* pods are created new and are in the Running state.
    2. Click StorageStorage Classes and verify that the ocs-external-storagecluster-cephfs storage class is created.
  • If you added the Ceph Object Gateway for object storage:

    1. Click StorageStorage Classes and verify that the ocs-external-storagecluster-ceph-rgw storage class is created.
    2. Click HomeOverview from the left pane of the OpenShift Web Console and click the Object Service tab. On the Status card, verify that the Object Service displays a green tick icon.