Red Hat CodeReady Workspaces scalability workaround on OCP 4.1/AWS

Solution In Progress - Updated -

Issue

The following are the issues faced while doing a hands-on workshop of CodeReady Workspaces on OpenShift Client Platform (OCP) 4.1/Amazon Web Services (AWS):

  • Workspace creation on CodeReady Workspaces takes a long time (around 2-3 minutes per workspace).
  • Workspace creation progress is serial (not parallel due to the AWS specification). Creation of attendees' workspaces takes a long time.

Resolution

Method 1: Change pvcStrategy

To reduce the EBS provisioning process, change the pvcStrategy in the CoreReady Workspaces Custom Resource (CR).

Pros:

  • With product support
  • No additional storage configuration required for the OCP infrastructure

Cons:

  • EBS can mount on only one node, so all workspaces must deploy to a single node

Default

  storage:
    pvcStrategy: per-workspace
    pvcClaimSize: 1Gi
    preCreateSubPaths: true

Share single EBS volume for all users

  storage:
    pvcStrategy: common
    pvcClaimSize: 100Gi
    preCreateSubPaths: true

Method 2: Use AWS EFS

To avoid the EBS provisioning bottleneck, use AWS EFS via an external provisioner.

Pros:

  • Creation of workspaces is quicker
  • Scaling of nodes is possible

Cons:

  • Without product support

Procedure

  1. Use the nfs-client provisioner. See nfs-client.

  2. Create a namespace for the external provisioner:

    $ oc new-project nfs-client-provisioner
$ NAMESPACE=`oc project -q`

  3. Create Role-based access control (RBAC) by modifying the namespace:

    $ cat deploy/rbac.yaml 
kind: ServiceAccount
apiVersion: v1
metadata:
name: nfs-client-provisioner

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
  resources: ["storageclasses"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "update", "patch"]

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
  name: nfs-client-provisioner
  namespace: nfs-client-provisioner
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-client-provisioner
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io

$ oc create -f deploy/rbac.yaml
$ oc adm policy add-scc-to-user hostmount-anyuid system:serviceaccount:$NAMESPACE:nfs-client-provisioner

  4. Create the Elastic File System (EFS) volume from the AWS Console. To confirm the NFS Server hostname and path from the mount options:

    $ sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com:/ efs

Caution: Security Groups setting
You have to open nfs port using Security Group, check if you face "Connection Timeout"
https://dev.classmethod.jp/etc/20181209-efs/

  5. Deploy the external provisioner. Modify the NFS server and path (env and volume sections) as per your environment.

    $ cat deploy/deployment.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
  type: Recreate
template:
  metadata:
    labels:
      app: nfs-client-provisioner
  spec:
    serviceAccountName: nfs-client-provisioner
    containers:
      - name: nfs-client-provisioner
        image: quay.io/external_storage/nfs-client-provisioner:latest
        volumeMounts:
          - name: nfs-client-root
            mountPath: /persistentvolumes
        env:
          - name: PROVISIONER_NAME
            value: fuseim.pri/ifs
          - name: NFS_SERVER
            value: fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com
          - name: NFS_PATH
            value: /
    volumes:
      - name: nfs-client-root
        nfs:
          server: fs-c1a1d7e0.efs.ap-northeast-1.amazonaws.com
          path: /

$ oc create -f deploy/deployment.yaml

  6. Test the external provisioner.

    $ oc create -f deploy/test-claim.yaml -f deploy/test-pod.yaml
$ oc delete -f deploy/test-pod.yaml -f deploy/test-claim.yaml

  7. Change the CR for CodeReady Workspaces cluster creation.

    Default

    storage:
pvcStrategy: per-workspace
pvcClaimSize: 1Gi
preCreateSubPaths: true

    Use EFS volume for workspaces

    storage:
pvcStrategy: common
pvcClaimSize: 1Gi
preCreateSubPaths: true
workspacePVCStorageClassName: 'managed-nfs-storage'

Root Cause

  • Supported dynamic provisionable volume is only AWS EBS on OCP 4.1.
  • CodeReady Workspaces requires storageclass to do user workspace dynamic provisioning.
  • EBS provisioning takes 2-3 minutes per user workspace and is serialized.
  • The time taken to complete the setup is (2-3 minutes) x (number of users).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments