Chapter 8. Backup and recovery

Backing up CodeReady Workspaces involves a combination of the following processes that back up different data:

8.1. Supported Restic-compatible backup servers

CodeReady Workspaces uses the CodeReady Workspaces Operator and integrated Restic to back up and restore CodeReady Workspaces instances from backup snapshots on a configured backup server. The CodeReady Workspaces Operator automates the creation of a Restic backup repository on the backup server. To back up data, the CodeReady Workspaces Operator gathers the data required for a backup snapshot and uses Restic to create and manage the snapshot. To restore data, the CodeReady Workspaces Operator uses Restic to retrieve and decrypt the snapshot, and then the CodeReady Workspaces Operator applies the retrieved data to perform the recovery.

CodeReady Workspaces can use the following backup servers that are compatible with the integrated Restic:

SFTP
See the documentation for the SFTP server solution you plan to use (OpenSSH or a derived commercial product) and the Restic Docs on SFTP.
Amazon S3
See the documentation for Amazon S3 (or the chosen S3 API compatible storage) and the Restic Docs on Amazon S3.
REST
See the README for Rest Server and the Restic Docs on Rest Server.

For testing the backing up and restoring of CodeReady Workspaces instances, CodeReady Workspaces offers the internal backup server.

Additional resources

8.2. Backing up of CodeReady Workspaces instances to an SFTP backup server

You can send backups of CodeReady Workspaces instances to an SFTP backup server with custom resources or crwctl:

8.2.1. Backing up a CodeReady Workspaces instance to an SFTP backup server by using custom resources

Backing up a CodeReady Workspaces instance to an SFTP backup server by using custom resources requires two custom objects:

  • First you create a custom object to configure CodeReady Workspaces to use an SFTP backup server.
  • Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured SFTP backup server.

8.2.1.1. Configuring CodeReady Workspaces with custom resources to use an SFTP backup server

To configure CodeReady Workspaces to use an SFTP backup server:

Prerequisites

Procedure

  1. Create a Secret containing the repo-password key with a password: 

    apiVersion: v1
kind: Secret
metadata:
  name: backup-encryption-password-secret
  namespace: eclipse-che
type: Opaque
stringData:
  repo-password: <password>

    (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.)

    Warning

    The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data.

  2. Create a Secret (for example, name: ssh-key-secret) containing the ssh-privatekey key with a private SSH key for logging in to the SFTP server without a password.

  3. Create the CheBackupServerConfiguration custom object: 

    apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
  namespace: openshift-workspaces
spec: 1
  sftp:
    username: <username> 2
    hostname: <hostname> 3
    port: 1234 4
    repositoryPath: red-hat-codeready-workspaces-backups 5
    repositoryPasswordSecretRef: backup-encryption-password-secret 6
    sshKeySecretRef: ssh-key-secret 7
    1
    Must only contain one section (such as sftp).
    2
    User name on the remote server to log in using the SSH protocol.
    3
    Remote server hostname.
    4
    Optional property that specifies the port on which an SFTP server is running. The default value is 22.
    5
    Absolute or relative path on the server where backup snapshots are stored.
    6
    Secret created in step 1.
    7
    Secret created in step 2.
  4. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server.
Important

The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to previous CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don’t add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.)

8.2.1.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using the CheClusterBackup custom object

You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Create the CheClusterBackup custom object, which creates a backup snapshot: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: red-hat-codeready-workspaces-backup
  namespace: openshift-workspaces
spec:
  backupServerConfigRef: backup-server-configuration 1
  useInternalBackupServer: false 2
    1
    Name of the CheBackupServerConfiguration custom object defining what backup server to use.
    2
    Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST).
    Tip

    If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name. To delete it on the command line, use oc: 

    $ oc delete CheClusterBackup <name> -n openshift-workspaces

  2. Read the status section of the CheClusterBackup custom object to verify the backup process, for example: 

    status:
  message: 'Backup is in progress. Start time: <timestamp>' 1
  stage: Collecting CodeReady Workspaces installation data 2
  state: InProgress 3
    1
    Summary of the latest state of the backup process.
    2
    Current stage of the backup process in a human-readable format.
    3
    Backup process state: InProgress or Succeeded or Failed.

    The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded: 

    status:
  ...
  message: >-
    Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC
    m=+999.553272281 1
  snapshotId: 9f0adce2 2
  state: Succeeded 3
    1
    Summary of the latest state of the backup process.
    2
    ID of the created backup snapshot. This field appears only when state is Succeeded.
    3
    Backup process state: Succeeded (or Failed).

8.2.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl

You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured SFTP backup server. To do so, enter crwctl with the command-line options or set the environment variables.

8.2.2.1. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl with command-line options

To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured SFTP backup server, run crwctl with the command-line options.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Enter the crwctl server:backup command with the following arguments: 

    $ crwctl server:backup \
--repository-url=<repository_url> \ 1
--repository-password=<repository_password> \ 2
--ssh-key-file=<ssh_key_file> 3
    1
    Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: sftp:<user>@<host>:/<repository_path_on_sftp_server>. An example of a repository path on the SFTP server: /srv/restic-repo. For more details about repository URL syntax, see Restic Documentation.
    2
    Backup repository password as an argument using the --repository-password or -p option.
    3
    Path to a private SSH key file for authenticating on the SFTP server.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.2.2.2. Backing up a CodeReady Workspaces instance to an SFTP backup server by using crwctl and a CheBackupServerConfiguration custom object

You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command.

Prerequisites

Procedure

  1. Run the following command on a command line: 

    $ crwctl server:backup \
--backup-server-config-name=<name_of_CheBackupServerConfiguration> 1
    1
    This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.2.2.3. Configuring crwctl for an SFTP backup server with environment variables

As an alternative to using command-line options for crwctl server:backup, you can configure crwctl for an SFTP backup server with environment variables. This enables you to run crwctl server:backup without having to enter those options with it.

Prerequisites

Procedure

Which environment variables you need to set depends on your use case:

  • If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables:

    • BACKUP_REPOSITORY_URL for the backup repository.
    • BACKUP_REPOSITORY_PASSWORD for the backup repository password.

    • One of the two environment variables for the SSH key (for logging in without a password):

      • SSH_KEY_FILE for the path to the SSH key file.
      • SSH_KEY for the SSH key.

  • If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object:

    • BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.)

8.3. Backing up of CodeReady Workspaces instances to Amazon S3

You can send backups of CodeReady Workspaces instances to Amazon S3 (or S3 API compatible storage) with custom resources or crwctl:

8.3.1. Backing up a CodeReady Workspaces instance to Amazon S3 by using custom resources

Backing up a CodeReady Workspaces instance to Amazon S3 (or S3 API compatible storage) by using custom resources requires two custom objects:

  • First you create a custom object to configure CodeReady Workspaces to use Amazon S3.
  • Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured Amazon S3.

8.3.1.1. Configuring CodeReady Workspaces with custom resources to use Amazon S3

To configure CodeReady Workspaces to use Amazon S3:

Prerequisites

Procedure

  1. Create a Secret containing the repo-password key with a password: 

    apiVersion: v1
kind: Secret
metadata:
  name: backup-encryption-password-secret
  namespace: eclipse-che
type: Opaque
stringData:
  repo-password: <password>

    (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.)

    Warning

    The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data.

  2. Create a Secret (for example, name: aws-user-credentials-secret) containing:

    • awsAccessKeyId key with the AWS access key ID of the user
    • awsSecretAccessKey key with the AWS secret access key of the user

  3. Create the CheBackupServerConfiguration custom object: 

    apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
  namespace: openshift-workspaces
spec: 1
  awss3:
    protocol: https 2
    hostname: s3.amazonaws.com 3
    port: 443 4
    repositoryPath: red-hat-codeready-workspaces-backups 5
    repositoryPasswordSecretRef: backup-encryption-password-secret 6
    awsAccessKeySecretRef: aws-user-credentials-secret 7
    1
    Must only contain one section (such as awss3).
    2
    Optional property that specifies the protocol to be used. https is the default value; http is a permitted value.
    3
    Optional property that specifies the S3 hostname. The default value is s3.amazonaws.com.
    4
    Optional property that specifies the port on which the backup server is running. The default value is 443.
    5
    Name of the bucket resource that stores the backup snapshots. You must manually create the bucket resource before configuring it here.
    6
    Secret created in step 1.
    7
    Secret created in step 2.
  4. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server.
Important

The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to previous CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don’t add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.)

8.3.1.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using the CheClusterBackup custom object

You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Create the CheClusterBackup custom object, which creates a backup snapshot: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: red-hat-codeready-workspaces-backup
  namespace: openshift-workspaces
spec:
  backupServerConfigRef: backup-server-configuration 1
  useInternalBackupServer: false 2
    1
    Name of the CheBackupServerConfiguration custom object defining what backup server to use.
    2
    Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST).
    Tip

    If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name. To delete it on the command line, use oc: 

    $ oc delete CheClusterBackup <name> -n openshift-workspaces

  2. Read the status section of the CheClusterBackup custom object to verify the backup process, for example: 

    status:
  message: 'Backup is in progress. Start time: <timestamp>' 1
  stage: Collecting CodeReady Workspaces installation data 2
  state: InProgress 3
    1
    Summary of the latest state of the backup process.
    2
    Current stage of the backup process in a human-readable format.
    3
    Backup process state: InProgress or Succeeded or Failed.

    The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded: 

    status:
  ...
  message: >-
    Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC
    m=+999.553272281 1
  snapshotId: 9f0adce2 2
  state: Succeeded 3
    1
    Summary of the latest state of the backup process.
    2
    ID of the created backup snapshot. This field appears only when state is Succeeded.
    3
    Backup process state: Succeeded (or Failed).

8.3.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl

You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to the configured Amazon S3 (or S3 API compatible storage). To do so, enter crwctl with the command-line options or set the environment variables.

8.3.2.1. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl with command-line options

To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to the configured Amazon S3, run crwctl with the command-line options.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Enter the crwctl server:backup command with the following arguments: 

    $ crwctl server:backup \
--repository-url=<repository_url> \ 1
--repository-password=<repository_password> \ 2
--aws-access-key-id=<aws-access-key-id> \ 3
--aws-secret-access-key==<aws-secret-access-key> 4
    1
    Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: s3:s3.amazonaws.com/<bucket_name>. For more details about repository URL syntax, see Restic Documentation.
    2
    Backup repository password as an argument using the --repository-password or -p option.
    3
    User’s AWS access key ID.
    4
    User’s AWS secret access key.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.3.2.2. Backing up a CodeReady Workspaces instance to Amazon S3 by using crwctl and a CheBackupServerConfiguration custom object

You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command.

Prerequisites

Procedure

  1. Run the following command on a command line: 

    $ crwctl server:backup \
--backup-server-config-name=<name_of_CheBackupServerConfiguration> 1
    1
    This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.3.2.3. Configuring crwctl with environment variables to use Amazon S3

As an alternative to using command-line options for crwctl server:backup, you can configure crwctl for Amazon S3 with environment variables. This enables you to run crwctl server:backup without having to enter those options with it.

Prerequisites

Procedure

Which environment variables you need to set depends on your use case:

  • If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables:

    • BACKUP_REPOSITORY_URL for the backup repository.
    • BACKUP_REPOSITORY_PASSWORD for the backup repository password.
    • AWS_ACCESS_KEY_ID for the user’s AWS access key ID.
    • AWS_SECRET_ACCESS_KEY for the user’s AWS secret access key.

  • If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object:

    • BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.)

8.4. Backing up of CodeReady Workspaces instances to a REST backup server

You can send backups of CodeReady Workspaces instances to a REST backup server with custom resources or crwctl:

8.4.1. Backing up a CodeReady Workspaces instance to a REST backup server by using custom resources

Backing up a CodeReady Workspaces instance to a REST backup server by using custom resources requires two custom objects:

  • First you create a custom object to configure CodeReady Workspaces to use a REST backup server.
  • Then you create a custom object to make and send a backup snapshot of a CodeReady Workspaces instance to the configured REST backup server.

8.4.1.1. Configuring CodeReady Workspaces with custom resources to use a REST backup server

To configure CodeReady Workspaces to use a REST backup server:

Prerequisites

Procedure

  1. Create a Secret containing the repo-password key with a password: 

    apiVersion: v1
kind: Secret
metadata:
  name: backup-encryption-password-secret
  namespace: eclipse-che
type: Opaque
stringData:
  repo-password: <password>

    (The CodeReady Workspaces Operator will set up this password for the backup repository that the CodeReady Workspaces Operator will create from this custom object on the backup server.)

    Warning

    The backup repository password is used to encrypt the backup data. If you lose this password, you will lose the backup data.

  2. Optional: Create a Secret (for example, name: rest-server-auth-secret) containing the username and password keys for the REST server user credentials.

    Note

    Skip this step if not using the REST server user credentials.

  3. Create the CheBackupServerConfiguration custom object: 

    apiVersion: org.eclipse.che/v1
kind: CheBackupServerConfiguration
metadata:
  name: backup-server-configuration
  namespace: openshift-workspaces
spec: 1
  rest:
    protocol: https 2
    hostname: <hostname> 3
    port: 1234 4
    repositoryPath: red-hat-codeready-workspaces-backups 5
    repositoryPasswordSecretRef: backup-encryption-password-secret 6
    credentialsSecretRef: rest-server-auth-secret 7
    1
    Must only contain one section (such as rest).
    2
    Optional property that specifies the protocol to be used: https is the default value; http is a permitted value.
    3
    Backup server hostname.
    4
    Optional property that specifies the port on which the backup server is running. The default value is 8000.
    5
    Path on the backup server where the backup snapshots are stored.
    6
    Secret created in step 1.
    7
    Optional property that specifies the Secret created in step 2.
  4. Optional: To configure multiple backup servers, create a separate CheBackupServerConfiguration custom object for each backup server.
Important

The CodeReady Workspaces Operator automatically backs up the CodeReady Workspaces instance before every CodeReady Workspaces update, permitting rollback to previous CodeReady Workspaces version if needed. If you configure only one backup server, that backup server is automatically used for pre-update backups by default. If you configure multiple backup servers, you must add the che.eclipse.org/backup-before-update: true annotation to the custom object of only one of them to specify it as the default backup server for pre-update backups. (If you don’t add this annotation for one of multiple backup servers, or if you add this annotation for multiple backup servers, then the CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups.)

8.4.1.2. Backing up a CodeReady Workspaces instance to a REST backup server by using the CheClusterBackup custom object

You can use a CheClusterBackup custom object to make a backup snapshot of a CodeReady Workspaces instance and to send the snapshot to the configured backup server. To create each and every backup snapshot requires a new CheClusterBackup custom object; that is, editing an already consumed CheClusterBackup custom object, during or after backing up, has no effect.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Create the CheClusterBackup custom object, which creates a backup snapshot: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: red-hat-codeready-workspaces-backup
  namespace: openshift-workspaces
spec:
  backupServerConfigRef: backup-server-configuration 1
  useInternalBackupServer: false 2
    1
    Name of the CheBackupServerConfiguration custom object defining what backup server to use.
    2
    Configures the Operator through this custom resource to use the CodeReady Workspaces-managed internal backup server or an administrator-managed external backup server (SFTP, Amazon S3 or S3 API compatible storage, or REST).
    Tip

    If you intend to reuse a name for CheClusterBackup custom objects, first delete any existing custom object with the same name. To delete it on the command line, use oc: 

    $ oc delete CheClusterBackup <name> -n openshift-workspaces

  2. Read the status section of the CheClusterBackup custom object to verify the backup process, for example: 

    status:
  message: 'Backup is in progress. Start time: <timestamp>' 1
  stage: Collecting CodeReady Workspaces installation data 2
  state: InProgress 3
    1
    Summary of the latest state of the backup process.
    2
    Current stage of the backup process in a human-readable format.
    3
    Backup process state: InProgress or Succeeded or Failed.

    The CodeReady Workspaces instance is backed up in a snapshot when state is Succeeded: 

    status:
  ...
  message: >-
    Backup successfully finished at 2021-12-03 10:07:51.151090621 +0000 UTC
    m=+999.553272281 1
  snapshotId: 9f0adce2 2
  state: Succeeded 3
    1
    Summary of the latest state of the backup process.
    2
    ID of the created backup snapshot. This field appears only when state is Succeeded.
    3
    Backup process state: Succeeded (or Failed).

8.4.2. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl

You can use crwctl to make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured REST backup server. To do so, enter crwctl with the command-line options or set the environment variables.

8.4.2.1. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl with command-line options

To make a backup snapshot of a CodeReady Workspaces instance and send the snapshot to a configured REST backup server, run crwctl with the command-line options.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

Procedure

  1. Enter the crwctl server:backup command with the following arguments: 

    $ crwctl server:backup \
--repository-url=<repository_url> \ 1
--repository-password=<repository_password> \ 2
--username=<username> \ 3
--password=<password> 4
    1
    Backup repository URL as an argument using the --repository-url or -r option. Syntax for the backup repository URL: rest:http://<host>:8000/ or rest:https://<user>:<password>@<host>:8000/ or rest:https://<user>:<password>@<host>:8000/<repository_path_on_rest_server>/. For more details about repository URL syntax, see Restic Documentation.
    2
    Backup repository password as an argument using the --repository-password or -p option.
    3
    When authentication is required, the username for authenticating to the backup REST server as the --username=<username> argument.
    4
    When authentication is required, the password for authenticating to the backup REST server as the --password=<password> argument.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.4.2.2. Backing up a CodeReady Workspaces instance to a REST backup server by using crwctl and a CheBackupServerConfiguration custom object

You can back up a CodeReady Workspaces instance by entering the name of a CheBackupServerConfiguration custom object as an argument with the crwctl server:backup command.

Prerequisites

Procedure

  1. Run the following command on a command line: 

    $ crwctl server:backup \
--backup-server-config-name=<name_of_CheBackupServerConfiguration> 1
    1
    This option points crwctl to a CheBackupServerConfiguration custom object. You can find the name of the CheBackupServerConfiguration custom object under metadata in the custom object.
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --backup-server-config-name option with the crwctl server:backup command just once and omit this option onward when entering crwctl server:backup.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.4.2.3. Configuring crwctl with environment variables to use a REST backup server

As an alternative to using command-line options for crwctl server:backup, you can configure crwctl for a REST backup server with environment variables. This enables you to run crwctl server:backup without having to enter those options with it.

Prerequisites

Procedure

Which environment variables you need to set depends on your use case:

  • If you do not have a CheBackupServerConfiguration custom object yet, then you can set the following environment variables:

    • BACKUP_REPOSITORY_URL for the backup repository.
    • BACKUP_REPOSITORY_PASSWORD for the backup repository password.

    • When authentication to the REST backup server is required:

      • REST_SERVER_USERNAME for the username.
      • REST_SERVER_PASSWORD for the password.

  • If you already have a CheBackupServerConfiguration custom object, you can set the following environment variable that will point crwctl to that CheBackupServerConfiguration custom object:

    • BACKUP_SERVER_CONFIG_NAME for the name of a CheBackupServerConfiguration custom object to point crwctl to. (You can find name under metadata in the custom object.)

8.5. Backing up of CodeReady Workspaces instances to the internal backup server

To create backups for testing purposes, you can use the internal backup server. The internal backup server is a REST backup server that is managed by the CodeReady Workspaces Operator.

Important

The CodeReady Workspaces Operator defaults to using the internal backup server for pre-update backups (to allow the CodeReady Workspaces version rollback) if multiple backup servers are configured and if the che.eclipse.org/backup-before-update: true annotation is added to more than one of the multiple backup servers or is not added to the custom object of any one of the multiple backup servers.

8.5.1. Backing up a CodeReady Workspaces instance to the internal server by using the CheClusterBackup custom object

You can create a custom object to back up a CodeReady Workspaces instance to the internal backup server.

Note

In this case, the CodeReady Workspaces Operator automatically configures the internal backup server and creates a CheBackupServerConfiguration custom object with name: backup-rest-server-configuration.

Warning

Configuring a backup server inside the same cluster, for example the internal backup server, is not the best practice because a cluster failure can be one of the scenarios when a backed up CodeReady Workspaces instance needs to be restored.

Procedure

  1. Create the CheClusterBackup custom object to send a backup snapshot to the internal backup server: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterBackup
metadata:
  name: red-hat-codeready-workspaces-backup
  namespace: openshift-workspaces
spec:
  useInternalBackupServer: true 1
    1
    Configures the custom resource to back up to the internal backup server.

  2. Read the status section of the CheClusterBackup object to verify the backup process: 

    ...
status:
  message: 'Backup is in progress. Start time: <timestamp>' 1
  stage: Collecting CodeReady Workspaces installation data 2
  state: InProgress 3
...
    1
    Displays the overall state or error message.
    2
    Current phase of the backup process in a human-readable format.
    3
    Backup process state: InProgress or Succeeded or Failed. 
    ...
status:
  cheVersion: 7.37.0 1
  message: >- 2
    Backup successfully finished at 2021-10-07 11:18:28.116103737 +0000 UTC
    m=+2384.430366711
  snapshotId: 40a09756 3
  state: Succeeded 4
...
    1
    The version of CodeReady Workspaces from which the backup snapshot was created. This field appears only when state is Succeeded.
    2
    Displays the overall state or error message.
    3
    The ID of the created backup snapshot. This field appears only when state is Succeeded.
    4
    Backup process state: InProgress or Succeeded or Failed.

8.5.2. Backing up a CodeReady Workspaces instance to the internal server by using crwctl

You can use crwctl to take a backup snapshot of a CodeReady Workspaces instance and send it to the internal backup server.

Note

In this case, the CodeReady Workspaces Operator automatically configures the internal backup server and creates a CheBackupServerConfiguration custom object with name: backup-rest-server-configuration.

Warning

This procedure does not back up the source code stored in users' workspaces. To back up the source code stored in users' workspaces, see Backups of persistent volumes.

Prerequisites

  • Installed crwctl.
  • CodeReady Workspaces has not been configured to use a backup server.

Procedure

  1. On a command line, enter the crwctl server:backup command with no arguments.

  2. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling backup...OK
  ✔ Waiting until backup process finishes...OK
Backup snapshot ID: 9f0adce2
Command server:backup has completed successfully in 00:10.

8.6. Restoring a CodeReady Workspaces instance from a backup

You can restore a CodeReady Workspaces instance from a backup snapshot by using the CheClusterRestore custom object or crwctl:

8.6.1. Restoring a CodeReady Workspaces instance from a backup by using the CheClusterRestore custom object

You can restore a CodeReady Workspaces instance from a backup snapshot by using custom resources. A new CheClusterRestore custom object is required every time you restore a CodeReady Workspaces instance from a backup snapshot. (So editing an already consumed CheClusterRestore custom object, during or after backing up, has no effect.)

Warning

Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster.

Important

Do not use the CheClusterRestore custom object to recover a CodeReady Workspaces instance of an earlier version of CodeReady Workspaces! Use only crwctl to recover a CodeReady Workspaces instance of an earlier version of CodeReady Workspaces; see Section 8.6.2, “Restoring a CodeReady Workspaces instance from a backup by using crwctl”!

Prerequisites

Procedure

  1. Create the CheClusterRestore custom object, which restores a backup: 

    apiVersion: org.eclipse.che/v1
kind: CheClusterRestore
metadata:
  name: red-hat-codeready-workspaces-restore
  namespace: openshift-workspaces
spec:
  backupServerConfigRef: backup-server-configuration 1
  snapshotId: <snapshot_id> 2
    1
    Name of the CheBackupServerConfiguration object that defines what backup server to use. You can find this name as backupServerConfigRef in spec of the CheClusterBackup custom object. (You can also find it as name in metadata of the CheBackupServerConfiguration custom object used to take the backup snapshot you are restoring.)
    2
    Optional parameter defining the Snapshot ID to restore from. The default value is the last snapshot in the backup repository on the backup server.
    Tip

    To reuse a name for CheClusterRestore custom objects, first delete any existing custom object with the same name. To delete it on the command line, use oc: 

    $ oc delete CheClusterRestore <name> -n openshift-workspaces

  2. Read the status section of the CheClusterRestore object to monitor the recovery process, for example: 

    status:
  message: 'Restore is in progress. Start time: <timestamp>' 1
  stage: Restoring CodeReady Workspaces related cluster objects 2
  state: InProgress 3
    1
    Overall state or error message; for example, Restore successfully finished.
    2
    Current phase of the recovery process in a human-readable format.
    3
    Recovery process state. One of InProgress, Succeeded, or Failed.

  3. Check that the CodeReady Workspaces instance has been recovered.

    Tip

    If errors occur in your browser after the recovery, clean up the browser data for the CodeReady Workspaces domain.

  4. Delete the CheClusterRestore object.

8.6.2. Restoring a CodeReady Workspaces instance from a backup by using crwctl

Using crwctl, you can restore a CodeReady Workspaces instance from a backup snapshot by using any of the following options (as applicable):

8.6.2.1. Restoring a CodeReady Workspaces instance from its latest backup by using crwctl

To restore the CodeReady Workspaces instance from its latest backup by using crwctl, you can enter latest as the snapshot ID.

Warning

Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster.

Important

If the backup snapshot was taken in an earlier version of Che, add the --version=<earlier_version_number> option to the crwctl server:restore command to roll back Che to that earlier version.

Prerequisites

Procedure

  1. Run the crwctl server:restore command with the following arguments: 

    $ crwctl server:restore \
--snapshot-id=latest \ 1
--repository-url=<repository-url> \ 2
--repository-password=<repository-password> 3
    1
    The latest snapshot.
    2
    The backup repository URL as an argument using the --repository-url or -r option. (Omit if using the internal backup server!)
    3
    The backup repository password as an argument using the --repository-password or -p option. (Omit if using the internal backup server!)
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore.

  2. When prompted in the output, enter y to confirm: 

    ...
 ✔ Snapshot:          ...
 Asking for restore confirmation: Do you want to proceed? [y/n]
...
    Tip

    To skip this question, add the --batch option to the server:restore command.

  3. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling restore...OK
  ✔ Waiting until restore process finishes...OK
Command server:restore has completed successfully in 05:59.

8.6.2.2. Restoring a CodeReady Workspaces instance from a backup by snapshot ID

To restore a CodeReady Workspaces instance from a particular backup snapshot, you can enter the snapshot ID with crwctl.

Warning

Backup snapshots are bound to a OpenShift cluster. To restore a CodeReady Workspaces instance, you must only use a backup snapshot that was created in the same OpenShift cluster.

Important

If the backup snapshot was taken in an earlier version of Che, add the --version=<earlier_version_number> option to the crwctl server:restore command to roll back Che to that earlier version.

Prerequisites

Procedure

  1. Run the crwctl server:restore command with the backup snapshot ID specified using the --snapshot-id= option: 

    $ crwctl server:restore \
--snapshot-id=<snapshot-id> \ 1
--repository-url=<repository-url> \ 2
--repository-password=<repository-password> 3
    1
    The snapshot ID.
    2
    The backup repository URL as an argument using the --repository-url or -r option. (Omit if using the internal backup server!)
    3
    The backup repository password as an argument using the --repository-password or -p option. (Omit if using the internal backup server!)
    Tip

    Last used backup server information is stored in a Secret inside the CodeReady Workspaces cluster. When consistently using the same backup server, you can enter the --repository-url and --repository-password options with the crwctl server:backup command just once and omit them onward when entering crwctl server:backup or crwctl server:restore.

  2. When prompted in the output, enter y to confirm: 

    ...
 ✔ Snapshot:          ...
 Asking for restore confirmation: Do you want to proceed? [y/n]
...
    Tip

    To skip this question, add the --batch option to the server:restore command.

  3. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling restore...OK
  ✔ Waiting until restore process finishes...OK
Command server:restore has completed successfully in 05:59.

8.6.2.3. Restoring a CodeReady Workspaces instance by using crwctl and a CheClusterBackup custom object

You can restore a CodeReady Workspaces instance by entering the name of the CheClusterBackup custom object as an argument with the crwctl server:restore command.

Important

If the backup snapshot was taken in an earlier version of Che, then crwctl server:restore --backup-cr-name=<name_of_custom_object> rolls back Che to that earlier version, which is specified in the CheClusterBackup custom object.

Prerequisites

Procedure

  1. Run the following command on a command line: 

    $ crwctl server:restore \
--backup-cr-name=<name_of_custom_object> 1
    1
    The name of the CheClusterBackup custom object.

  2. When prompted in the output, enter y to confirm: 

    ...
 Asking for restore confirmation: Do you want to proceed? [y/n]
...
    Tip

    To skip this question, add the --batch option to the server:restore command.

  3. Verify the output of the entered command. For example: 

    ...
  ✔ Scheduling restore...OK
  ✔ Waiting until restore process finishes...OK
Command server:restore has completed successfully in 05:59.

8.7. Backups of persistent volumes

Persistent Volumes (PVs) store workspace data for CodeReady Workspaces similarly to how workspace data are stored for desktop IDEs on a local hard disk drive. PVs must be backed up periodically to prevent data loss. Storage-agnostic tools are available for backing up and restoring OpenShift resources, including PVs.

8.7.1. Velero

An open source tool for backing up OpenShift applications and their PVs is Velero. This tool can perform the following operations:

  • Deploy in the cloud or on premises.
  • Back up the cluster and restore the backed up data in case of data loss.
  • Migrate cluster resources to other clusters.
  • Replicate a production cluster to development and testing clusters.
Note

Alternatively, you can use backup solutions that depend on the underlying storage system: for example, solutions specific to Gluster or Ceph.

Additional resources

8.8. Backups of PostgreSQL

The CodeReady Workspaces server uses a PostgreSQL database for persisting data about the state of CodeReady Workspaces. The database contains information about user accounts, workspaces, preferences, and other details.

8.8.1. External PostgreSQL setup

By default, the CodeReady Workspaces Operator creates and manages deployment of the internal PostgreSQL database. However, the CodeReady Workspaces Operator does not support full lifecycle capabilities such as backups and recovery.

A business-critical setup must use an external PostgreSQL database that is configured:

  • for High Availability (HA) and Point In Time Recovery (PITR)
  • on-premises or using a cloud service such as Amazon Relational Database Service (Amazon RDS)

For example, Amazon RDS enables deployment of production databases in a Multi-Availability Zone configuration for a resilient disaster recovery strategy with daily and on-demand snapshots.

An example configuration is as follows:

ParameterValue

Instance class

db.t2.small

vCPU

1

RAM

2 GB

Multi-az

true, 2 replicas

Engine version

9.6.11

TLS

enabled

Automated backups

enabled (30 days)

You can make workspace metadata and user information persistent by configuring the external PostgreSQL database and configuring CodeReady Workspaces to use the configured external PostgreSQL database:

Additional resources

8.8.2. Configuring the external PostgreSQL

To make workspace metadata and user information persistent, configure an external PostgreSQL database as follows:

Procedure

  1. Define the values of the following placeholders:

    <database-user>
    CodeReady Workspaces server database user name
    <database-password>
    CodeReady Workspaces server database password
    <database>
    CodeReady Workspaces server database name

  2. Use the following SQL script to create a user and database for the CodeReady Workspaces server to make workspace metadata persistent: 

    CREATE USER <database-user> WITH PASSWORD '<database-password>'
CREATE DATABASE <database>
GRANT ALL PRIVILEGES ON DATABASE <database> TO <database-user>
ALTER USER <database-user> WITH SUPERUSER

  3. Define the value of the following placeholder:

    <identity-database-password>
    RH-SSO database password

  4. Use the following SQL script to create a database for the RH-SSO back end to make user information persistent: 

    CREATE USER keycloak WITH PASSWORD '<identity-database-password>'
CREATE DATABASE keycloak
GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak

8.8.3. Configuring CodeReady Workspaces to work with the external PostgreSQL

To make workspace metadata and user information persistent, you must configure CodeReady Workspaces to work with the configured external PostgreSQL database.

Prerequisites

Procedure

  1. Create a project for CodeReady Workspaces: 

    $ oc create namespace openshift-workspaces

  2. Create a Secret to store CodeReady Workspaces server database credentials: 

    $ oc create secret generic <server-database-credentials> \ 1
--from-literal=user=<database-user> \ 2
--from-literal=password=<database-password> \ 3
-n openshift-workspaces
    1
    Secret name to store CodeReady Workspaces server database credentials
    2
    CodeReady Workspaces server database username
    3
    CodeReady Workspaces server database password

  3. Create a Secret to store RH-SSO database credentials: 

    $ oc create secret generic <identity-database-credentials> \ 1
--from-literal=password=<identity-database-password> \ 2
-n openshift-workspaces
    1
    Secret name to store RH-SSO database credentials
    2
    RH-SSO database password

  4. Deploy Red Hat CodeReady Workspaces by executing the crwctl command and applying a patch. For example: 

    $ crwctl server:deploy --che-operator-cr-patch-yaml=patch.yaml ...

  5. Ensure that patch.yaml contains the following lines to make the Operator skip the deployment of a database and pass connection details of an existing database to a CodeReady Workspaces server: 

    spec:
  database:
    externalDb: true
    chePostgresHostName: <hostname> 1
    chePostgresPort: <port> 2
    chePostgresSecret: <server-database-credentials> 3
    chePostgresDb: <database> 4
spec:
  auth:
    identityProviderPostgresSecret: <identity-database-credentials> 5
    1
    External database hostname
    2
    External database port
    3
    Secret name with CodeReady Workspaces server database credentials
    4
    CodeReady Workspaces server database name
    5
    Secret name with RH-SSO database credentials