Red Hat CodeReady Workspaces scalability workaround on OCP 4.1/AWS

Solution In Progress - Updated -

Issue

The following are the issues faced while doing a hands-on workshop of CodeReady Workspaces on OpenShift Client Platform (OCP) 4.1/Amazon Web Services (AWS):

  • Workspace creation on CodeReady Workspaces takes a long time (around 2-3 minutes per workspace).
  • Workspace creation progress is serial (not parallel due to the AWS specification). Creation of attendees' workspaces takes a long time.

Resolution

Method 1: Change pvcStrategy

To reduce the EBS provisioning process, change the pvcStrategy in the CoreReady Workspaces Custom Resource (CR).

Pros:

  • With product support
  • No additional storage configuration required for the OCP infrastructure

Cons:

  • EBS can mount on only one node, so all workspaces must deploy to a single node

Default

  storage:
    pvcStrategy: per-workspace
    pvcClaimSize: 1Gi
    preCreateSubPaths: true

Share single EBS volume for all users

  storage:
    pvcStrategy: common
    pvcClaimSize: 100Gi
    preCreateSubPaths: true

Method 2: Use AWS EFS

To avoid the EBS provisioning bottleneck, use AWS EFS via an external provisioner.

Pros:

  • Creation of workspaces is quicker
  • Scaling of nodes is possible

Cons:

  • Without product support

Procedure

  1. Use the nfs-client provisioner. See nfs-client.

  2. Create a namespace for the external provisioner:

    $ oc new-project nfs-client-provisioner
    $ NAMESPACE=`oc project -q`
    
  3. Create Role-based access control (RBAC) by modifying the namespace:

    $ cat deploy/rbac.yaml 
    kind: ServiceAccount
    apiVersion: v1
    metadata:
    name: nfs-client-provisioner
    
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: nfs-client-provisioner-runner
    rules:
    - apiGroups: [""]
      resources: ["persistentvolumes"]
      verbs: ["get", "list", "watch", "create", "delete"]
    - apiGroups: [""]
      resources: ["persistentvolumeclaims"]
      verbs: ["get", "list", "watch", "update"]
    - apiGroups: ["storage.k8s.io"]
      resources: ["storageclasses"]
      verbs: ["get", "list", "watch"]
    - apiGroups: [""]
      resources: ["events"]
      verbs: ["create", "update", "patch"]
    
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: run-nfs-client-provisioner
    subjects:
    - kind: ServiceAccount
      name: nfs-client-provisioner
      namespace: nfs-client-provisioner
    roleRef:
    kind: ClusterRole
    name: nfs-client-provisioner-runner
    apiGroup: rbac.authorization.k8s.io
    
    kind: Role
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: leader-locking-nfs-client-provisioner
    rules:
    - apiGroups: [""]
      resources: ["endpoints"]
      verbs: ["get", "list", "watch", "create", "update", "patch"]
    
    kind: RoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
    name: leader-locking-nfs-client-provisioner
    subjects:
    - kind: ServiceAccount
      name: nfs-client-provisioner
      # replace with namespace where provisioner is deployed
      namespace: nfs-client-provisioner
    roleRef:
    kind: Role
    name: leader-locking-nfs-client-provisioner
    apiGroup: rbac.authorization.k8s.io
    
    $ oc create -f deploy/rbac.yaml
    $ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:$NAMESPACE:nfs-client-provisioner
    
  4. Create the Elastic File System (EFS) volume from the AWS Console. To confirm the NFS Server hostname and path from the mount options:

    $ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com:/ efs
    
    Caution: Security Groups setting
    You have to open nfs port using Security Group, check if you face "Connection Timeout"
    https://dev.classmethod.jp/etc/20181209-efs/
    
  5. Deploy the external provisioner. Modify the NFS server and path (env and volume sections) as per your environment.

    $ cat deploy/deployment.yaml 
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: nfs-client-provisioner
    
    kind: Deployment
    apiVersion: extensions/v1beta1
    metadata:
    name: nfs-client-provisioner
    spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: nfs-client-provisioner
      spec:
        serviceAccountName: nfs-client-provisioner
        containers:
          - name: nfs-client-provisioner
            image: quay.io/external_storage/nfs-client-provisioner:latest
            volumeMounts:
              - name: nfs-client-root
                mountPath: /persistentvolumes
            env:
              - name: PROVISIONER_NAME
                value: fuseim.pri/ifs
              - name: NFS_SERVER
                value: fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com
              - name: NFS_PATH
                value: /
        volumes:
          - name: nfs-client-root
            nfs:
              server: fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com
              path: /
    
    $ oc create -f deploy/deployment.yaml
    
  6. Test the external provisioner.

    $ oc create -f deploy/test-claim.yaml -f deploy/test-pod.yaml
    $ oc delete -f deploy/test-pod.yaml -f deploy/test-claim.yaml
    
  7. Change the CR for CodeReady Workspaces cluster creation.

    Default

    storage:
    pvcStrategy: per-workspace
    pvcClaimSize: 1Gi
    preCreateSubPaths: true
    

    Use EFS volume for workspaces

    storage:
    pvcStrategy: common
    pvcClaimSize: 1Gi
    preCreateSubPaths: true
    workspacePVCStorageClassName: 'managed-nfs-storage'
    

Root Cause

  • Supported dynamic provisionable volume is only AWS EBS on OCP 4.1.
  • CodeReady Workspaces requires storageclass to do user workspace dynamic provisioning.
  • EBS provisioning takes 2-3 minutes per user workspace and is serialized.
  • The time taken to complete the setup is (2-3 minutes) x (number of users).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.