Chapter 9. Updating the Registry with Red Hat Openshift Container Storage as the Storage Back-end

OpenShift Container Platform provides an integrated registry with storage using an NFS-backed persistent volume that is automatically setup. Red Hat Openshift Container Storage allows you to replace this with a Gluster persistent volume for registry storage. This provides increased reliability, scalability and failover.

For additional information about OpenShift Container Platform and the docker-registry, see https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/configuring_clusters/setting-up-the-registry.

9.1. Validating the Openshift Container Platform Registry Deployment

To verify that the registry is properly deployed, execute the following commands:

  1. On the master or client, execute the following command to login as the cluster admin user:

    # oc login

    For example:

    # oc login
    
    Authentication required for https://master.example.com:8443 (openshift)
    Username: <cluster-admin-user>
    Password: <password>
    Login successful.
    
    You have access to the following projects and can switch between them with 'oc project <projectname>':
    
      * default
        management-infra
        openshift
        openshift-infra
    
    Using project "default".

    If you are not automatically logged into project default, then switch to it by executing the following command:

    # oc project default
  2. To verify that the pod is created, execute the following command:

    # oc get pods

    For example:

    # oc get pods
    NAME                       READY     STATUS    RESTARTS   AGE
    docker-registry-2-mbu0u    1/1       Running   4          6d
    docker-registry-2-spw0o    1/1       Running   3          6d
    registry-console-1-rblwo   1/1       Running   3          6d
  3. To verify that the endpoints are created, execute the following command:

    # oc get endpoints

    For example:

    # oc get endpoints
    NAME               ENDPOINTS                                                                  AGE
    docker-registry    10.128.0.15:5000,10.129.0.9:5000                                           7d
    kubernetes         192.168.234.143:8443,192.168.234.143:8053,192.168.234.143:8053             7d
    registry-console   10.128.0.17:9090                                                           7d
    router             192.168.234.144:443,192.168.234.145:443,192.168.234.144:1936 + 3 more...   7d
  4. To verify that the persistent volume is created, execute the following command:

    # oc get pv
    NAME   CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM  REASON    AGE
    registry-volume           5Gi        RWX           Retain          Bound       default/registry-claim             7d
  5. To obtain the details of the persistent volume that was created for the NFS registry, execute the following command:

    # oc describe pv registry-volume
    Name:        registry-volume
    Labels:        <none>
    StorageClass:
    Status:        Bound
    Claim:        default/registry-claim
    Reclaim Policy:    Retain
    Access Modes:    RWX
    Capacity:    5Gi
    Message:
    Source:
        Type:    NFS (an NFS mount that lasts the lifetime of a pod)
        Server:    cns30.rh73
        Path:    /exports/registry
        ReadOnly:    false
    No events.

9.2. Converting the Openshift Container Platform Registry with Red Hat Openshift Container Storage

This section provides the steps to create a Red Hat Gluster Storage volume and use it to provide storage for the integrated registry.

Setting up a Red Hat Gluster Storage Persistent Volume

Execute the following commands to create a Red Hat Gluster Storage volume to store the registry data and create a persistent volume.

Note

The commands must be executed in the default project.

  1. Login to the default project:

    # oc project default

    For example:

    # oc project default
    Now using project "default" on server "https://cns30.rh73:8443"
  2. Execute the following command to create the gluster-registry-endpoints.yaml file:

     oc get endpoints <heketi-db-storage-endpoint-name> -o yaml --namespace=<project-name> >  gluster-registry-endpoints.yaml
    Note

    You must create an endpoint for each project from which you want to utilize the Red Hat Gluster Storage registry. Hence, you will have a service and an endpoint in both the default project and the new project (storage-project) created in earlier steps.

  3. Edit the gluster-registry-endpoints.yaml file. Change the name to gluster-registry-endpoints and remove all the other metadata, leaving everything else the same.

    # cat gluster-registry-endpoints.yaml
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: gluster-registry-endpoints
    subsets:
      - addresses:
          - ip: 192.168.124.114
          - ip: 192.168.124.52
          - ip: 192.168.124.83
        ports:
          - port: 1
    protocol: TCP
  4. Execute the following command to create the endpoint:

    # oc create -f gluster-registry-endpoints.yaml
    endpoints "gluster-registry-endpoints" created
  5. To verify the creation of the endpoint, execute the following command:

    # oc get endpoints
    NAME                       ENDPOINTS                                                                 AGE
    docker-registry            10.129.0.8:5000,10.130.0.5:5000                                           28d
    gluster-registry-endpoints  192.168.124.114:1,192.168.124.52:1,192.168.124.83:1                       10s
    kubernetes                 192.168.124.250:8443,192.168.124.250:8053,192.168.124.250:8053            28d
    registry-console           10.131.0.6:9090                                                           28d
    router                     192.168.124.114:443,192.168.124.83:443,192.168.124.114:1936 + 3 more...   28d
  6. Execute the following command to create the gluster-registry-service.yaml file:

     oc get services <heketi-storage-endpoint-name> -o yaml --namespace=<project-name> >  gluster-registry-service.yaml
  7. Edit the gluster-registry-service.yaml file. Change the name to gluster-registry-service and remove all the other metadata. Also, remove the specific cluster IP addresses:

    # cat gluster-registry-service.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: gluster-registry-service
    spec:
      ports:
        - port: 1
          protocol: TCP
          targetPort: 1
      sessionAffinity: None
      type: ClusterIP
    status:
    loadBalancer: {}
  8. Execute the following command to create the service:

    # oc create -f gluster-registry-service.yaml
    services "gluster-registry-service" created
  9. Execute the following command to verify if the service are running:

    # oc get services
    NAME                       CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
    docker-registry            172.30.197.118   <none>        5000/TCP                  28d
    gluster-registry-service   172.30.0.183     <none>        1/TCP                     6s
    kubernetes                 172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     29d
    registry-console           172.30.146.178   <none>        9000/TCP                  28d
    router                     172.30.232.238   <none>        80/TCP,443/TCP,1936/TCP   28d
  10. Execute the following command to obtain the fsGroup GID of the existing docker-registry pods:

    # export GID=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%.0f" ((index .items 0).spec.securityContext.fsGroup)}}')
  11. Execute the following command to create a volume

    # heketi-cli volume create --size=5 --name=gluster-registry-volume --gid=${GID}
  12. Create the persistent volume file for the Red Hat Gluster Storage volume:

    # cat gluster-registry-volume.yaml
    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: gluster-registry-volume
      labels:
        glusterfs: registry-volume
    spec:
      capacity:
        storage: 5Gi
      glusterfs:
        endpoints: gluster-registry-endpoints
        path: gluster-registry-volume
      accessModes:
        - ReadWriteMany
    persistentVolumeReclaimPolicy: Retain
  13. Execute the following command to create the persistent volume:

    # oc create -f gluster-registry-volume.yaml
  14. Execute the following command to verify and get the details of the created persistent volume:

    # oc get pv/gluster-registry-volume
    NAME                      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM     REASON    AGE
    gluster-registry-volume   5Gi        RWX           Retain          Available                       21m
  15. Create a new persistent volume claim. Following is a sample Persistent Volume Claim that will be used to replace the existing registry-storage volume claim.

    # cat gluster-registry-claim.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: gluster-registry-claim
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
      selector:
        matchLabels:
    glusterfs: registry-volume
  16. Create the persistent volume claim by executing the following command:

    # oc create -f gluster-registry-claim.yaml

    For example:

    # oc create -f gluster-registry-claim.yaml
    persistentvolumeclaim "gluster-registry-claim" created
  17. Execute the following command to verify if the claim is bound:

    # oc get pvc/gluster-registry-claim

    For example:

    # oc get pvc/gluster-registry-claim
    NAME                     STATUS    VOLUME                    CAPACITY   ACCESSMODES   AGE
    gluster-registry-claim   Bound     gluster-registry-volume   5Gi        RWX           22s
  18. Make the registry read-only by executing the following command:

    # oc set env -n default dc/docker-registry 'REGISTRY_STORAGE_MAINTENANCE_READONLY={"enabled":true}'

    To confirm the value is set to readonly, execute the following command:

    # oc set env -n default dc/docker-registry --list
  19. If you want to migrate the data from the old registry to the Red Hat Gluster Storage registry, then execute the following commands:

    Note

    These steps are optional.

    1. Add the Red Hat Gluster Storage registry to the old registry deployment configuration (dc) by executing the following command:

      # oc set volume dc/docker-registry --add --name=gluster-registry-storage -m /gluster-registry -t pvc --claim-name=gluster-registry-claim
    2. Save the Registry pod name by executing the following command:

      # export REGISTRY_POD=$(oc get po --selector="docker-registry=default" -o go-template --template='{{printf "%s" ((index .items 0).metadata.name)}}')
    3. Copy the data from the old registry directory to the Red Hat Gluster Storage registry directory by executing the following command:

      # oc rsh $REGISTRY_POD cp -a /registry/ /gluster-registry/
    4. Remove the Red Hat Gluster Storage registry from the old dc registry by executing the following command:

      # oc volume dc/docker-registry --remove --name=gluster-registry-storage
  20. Replace the existing registry-storage volume with the new gluster-registry-claim PVC:

    # oc set volume dc/docker-registry --add --name=registry-storage -t pvc --claim-name=gluster-registry-claim --overwrite
  21. Make the registry read write by executing the following command:

    # oc set env dc/docker-registry REGISTRY_STORAGE_MAINTENANCE_READONLY-

    To validate if the setting is set to read write, execute the following command:

    # oc set env -n default dc/docker-registry --list

For more information about accessing the registry, see https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html/configuring_clusters/setting-up-the-registry#install-config-registry-accessing.