4.2.  Deploying Container-Native Storage

The following section covers deployment of the Container-Native Storage pods using the cns-deploy tool. If you prefer to manually install Container-Native Storage, see Appendix A, Manual Deployment

Note

If you want to enable encryption on the Container Native Storage setup, refer Chapter 12, Enabling Encryption before proceeding with the following steps.
  1. You must first provide a topology file for heketi which describes the topology of the Red Hat Gluster Storage nodes and their attached storage devices. A sample, formatted topology file (topology-sample.json) is installed with the ‘heketi-client’ package in the /usr/share/heketi/ directory.
    {
        "clusters": [
            {
                "nodes": [
                    {
                        "node": {
                            "hostnames": {
                                "manage": [
                                    "node1.example.com"
                                ],
                                "storage": [
                                    "192.168.68.3"
                                ]
                            },
                            "zone": 1
                        },
                        "devices": [
                            "/dev/sdb",
                            "/dev/sdc",
                            "/dev/sdd",
                            "/dev/sde",
                            "/dev/sdf",
                            "/dev/sdg",
                            "/dev/sdh",
                            "/dev/sdi"
                        ]
                    },
                    {
                        "node": {
                            "hostnames": {
                                "manage": [
                                    "node2.example.com"
                                ],
                                "storage": [
                                    "192.168.68.2"
                                ]
                            },
                            "zone": 2
                        },
                        "devices": [
                            "/dev/sdb",
                            "/dev/sdc",
                            "/dev/sdd",
                            "/dev/sde",
                            "/dev/sdf",
                            "/dev/sdg",
                            "/dev/sdh",
                            "/dev/sdi"
                        ]
                    },
    
    .......
    .......
    where,
    • clusters: Array of clusters.
      Each element on the array is a map which describes the cluster as follows.
      • nodes: Array of OpenShift nodes that will host the Red Hat Gluster Storage container
        Each element on the array is a map which describes the node as follows
        • node: It is a map of the following elements:
          • zone: The value represents the zone number that the node belongs to; the zone number is used by heketi for choosing optimum position of bricks by having replicas of bricks in different zones. Hence zone number is similar to a failure domain.
          • hostnames: It is a map which lists the manage and storage addresses
            • manage: It is the hostname/IP Address that is used by Heketi to communicate with the node
            • storage: It is the IP address that is used by other OpenShift nodes to communicate with the node. Storage data traffic will use the interface attached to this IP. This must be the IP address and not the hostname because, in an OpenShift environment, Heketi considers this to be the endpoint too.
        • devices: Name of each disk to be added
    Edit the topology file based on the Red Hat Gluster Storage pod hostname under the node.hostnames.manage section and node.hostnames.storage section with the IP address. For simplicity, the /usr/share/heketi/topology-sample.json file only sets up 4 nodes with 8 drives each.

    Important

    Heketi stores its database on a Red Hat Gluster Storage volume. In cases where the volume is down, the Heketi service does not respond due to the unavailability of the volume served by a disabled trusted storage pool. To resolve this issue, restart the trusted storage pool which contains the Heketi volume.
  2. Execute the following command on the client to deploy the heketi and Red Hat Gluster Storage pods:
    # cns-deploy -n <namespace> -g topology.json
    For example:
    # cns-deploy -n storage-project -g topology.json
    Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.
    
    Before getting started, this script has some requirements of the execution
    environment and of the container platform that you should verify.
    
    The client machine that will run this script must have:
     * Administrative access to an existing Kubernetes or OpenShift cluster
     * Access to a python interpreter 'python'
     * Access to the heketi client 'heketi-cli'
    
    Each of the nodes that will host GlusterFS must also have appropriate firewall
    rules for the required GlusterFS ports:
     * 2222  - sshd (if running GlusterFS in a pod)
     * 24007 - GlusterFS Daemon
     * 24008 - GlusterFS Management
     * 49152 to 49251 - Each brick for every volume on the host requires its own
       port. For every new brick, one new port will be used starting at 49152. We
       recommend a default range of 49152-49251 on each host, though you can adjust
       this to fit your needs.
    
    In addition, for an OpenShift deployment you must:
     * Have 'cluster_admin' role on the administrative account doing the deployment
     * Add the 'default' and 'router' Service Accounts to the 'privileged' SCC
     * Have a router deployed that is configured to allow apps to access services
       running in the cluster
    
    Do you wish to proceed with deployment?
    
    [Y]es, [N]o? [Default: Y]: y
    Using OpenShift CLI.
    NAME              STATUS    AGE
    storage-project   Active    1h
    Using namespace "storage-project".
    Checking that heketi pod is not running ... OK
    template "deploy-heketi" created
    serviceaccount "heketi-service-account" created
    template "heketi" created
    template "glusterfs" created
    role "edit" added:
    "system:serviceaccount:storage-project:heketi-service-account"
    node "dhcp37-64.lab.eng.blr.redhat.com" labeled
    node "dhcp37-79.lab.eng.blr.redhat.com" labeled
    node "dhcp37-100.lab.eng.blr.redhat.com" labeled
    daemonset "glusterfs" created
    Waiting for GlusterFS pods to start ... OK
    service "deploy-heketi" created
    route "deploy-heketi" created
    deploymentconfig "deploy-heketi" created
    Waiting for deploy-heketi pod to start ... OK
    Creating cluster ... ID: 94877b3f72b79273e87c1e94201ecd58
    Creating node dhcp37-64.lab.eng.blr.redhat.com ... ID:
    c718232efcc5f8ee50c91ed8d3e35364
    Adding device /dev/vdc ... OK
    Adding device /dev/vdd ... OK
    Adding device /dev/vde ... OK
    Adding device /dev/vdf ... OK
    Creating node dhcp37-79.lab.eng.blr.redhat.com ... ID:
    279f5caadce331f7d1df35e2697364a8
    Adding device /dev/vdc ... OK
    Adding device /dev/vdd ... OK
    Adding device /dev/vde ... OK
    Adding device /dev/vdf ... OK
    Creating node dhcp37-100.lab.eng.blr.redhat.com ... ID:
    4ae467b5da48d40118bd1992c85d5cdd
    Adding device /dev/vdc ... OK
    Adding device /dev/vdd ... OK
    Adding device /dev/vde ... OK
    Adding device /dev/vdf ... OK
    heketi topology loaded.
    Saving heketi-storage.json
    secret "heketi-storage-secret" created
    endpoints "heketi-storage-endpoints" created
    service "heketi-storage-endpoints" created
    job "heketi-storage-copy-job" created
    deploymentconfig "deploy-heketi" deleted
    route "deploy-heketi" deleted
    service "deploy-heketi" deleted
    job "heketi-storage-copy-job" deleted
    pod "deploy-heketi-1-kd1zn" deleted
    secret "heketi-storage-secret" deleted
    service "heketi" created
    route "heketi" created
    deploymentconfig "heketi" created
    Waiting for heketi pod to start ... OK
    heketi is now running.
    Ready to create and provide GlusterFS volumes.

    Note

    For more information on the cns-deploy commands, refer to the man page of the cns-deploy.
    # cns-deploy --help
  3. Execute the following command to let the client communicate with the container:
    # export  HEKETI_CLI_SERVER=http://heketi-<project_name>.<sub_domain_name>
    For example:
    # export  HEKETI_CLI_SERVER=http://heketi-storage-project.cloudapps.mystorage.com
    To verify if Heketi is loaded with the topology execute the following command:
    # heketi-cli topology info

Note

The cns-deploy tool does not support scaling up of the cluster. To manually scale-up the cluster, refer Chapter 8, Managing Clusters