Chapter 4. Application backup and restore

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Set the BUCKET variable:

   $ BUCKET=<your_bucket>

2. Set the REGION variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION 1

   1

   us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero 1

   1

   If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a velero-policy.json file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the velero user the minimum necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the velero user:

   $ aws iam create-access-key --user-name velero

   Example output

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

8. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - openshift 1
           - aws
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - name: default
         velero:
           provider: aws
           default: true
           objectStorage:
             bucket: <bucket_name> 5
             prefix: <prefix> 6
           config:
             region: <region>
             profile: "default"
           credential:
             key: cloud
             name: cloud-credentials 7
     snapshotLocations: 8
       - name: default
         velero:
           provider: aws
           config:
             region: <region> 9
             profile: "default"

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   6

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   7

   Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Log in to Azure:

   $ az login

2. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

3. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location CentralUS 1

   1

   Specify your location.

4. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

5. Create an Azure storage account:

   $ az storage account create \
       --name $AZURE_STORAGE_ACCOUNT_ID \
       --resource-group $AZURE_RESOURCE_GROUP \
       --sku Standard_GRS \
       --encryption-services blob \
       --https-only true \
       --kind BlobStorage \
       --access-tier Hot

6. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

7. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

8. Obtain the storage account access key:

   $ AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list \
     --account-name $AZURE_STORAGE_ACCOUNT_ID \
     --query "[?keyName == 'key1'].value" -o tsv`

9. Create a custom role that has the minimum required permissions:

   AZURE_ROLE=Velero
   az role definition create --role-definition '{
      "Name": "'$AZURE_ROLE'",
      "Description": "Velero related permissions to perform backups, restores and deletions",
      "Actions": [
          "Microsoft.Compute/disks/read",
          "Microsoft.Compute/disks/write",
          "Microsoft.Compute/disks/endGetAccess/action",
          "Microsoft.Compute/disks/beginGetAccess/action",
          "Microsoft.Compute/snapshots/read",
          "Microsoft.Compute/snapshots/write",
          "Microsoft.Compute/snapshots/delete",
          "Microsoft.Storage/storageAccounts/listkeys/action",
          "Microsoft.Storage/storageAccounts/regeneratekey/action"
      ],
      "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"]
      }'

10. Create a credentials-velero file:

    $ cat << EOF > ./credentials-velero
    AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
    AZURE_TENANT_ID=${AZURE_TENANT_ID}
    AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
    AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
    AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
    AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} 1
    AZURE_CLOUD_NAME=AzurePublicCloud
    EOF

    1

    Mandatory. You cannot back up internal images if the credentials-velero file contains only the service principal credentials.

    You use the credentials-velero file to create a Secret object for Azure before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - azure
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             resourceGroup: <azure_resource_group> 5
             storageAccount: <azure_storage_account_id> 6
             subscriptionId: <azure_subscription_id> 7
             storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
           credential:
             key: cloud
             name: cloud-credentials-azure  8
           provider: azure
           default: true
           objectStorage:
             bucket: <bucket_name> 9
             prefix: <prefix> 10
     snapshotLocations: 11
       - velero:
           config:
             resourceGroup: <azure_resource_group>
             subscriptionId: <azure_subscription_id>
             incremental: "true"
           name: default
           provider: azure

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the Azure resource group.

   6

   Specify the Azure storage account ID.

   7

   Specify the Azure subscription ID.

   8

   If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.

   9

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   10

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   11

   You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Log in to GCP:

   $ gcloud auth login

2. Set the BUCKET variable:

   $ BUCKET=<bucket> 1

   1

   Specify your bucket name.

3. Create the storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero service account"

6. List your service accounts:

   $ gcloud iam service-accounts list

7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:Velero service account" \
       --format 'value(email)')

8. Attach the policies to give the velero user the minimum necessary permissions:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
       storage.objects.create
       storage.objects.delete
       storage.objects.get
       storage.objects.list
       iam.serviceAccounts.signBlob
   )

9. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server

11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - gcp
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           provider: gcp
           default: true
           credential:
             key: cloud
             name: cloud-credentials-gcp 5
           objectStorage:
             bucket: <bucket_name> 6
             prefix: <prefix> 7
     snapshotLocations: 8
       - velero:
           provider: gcp
           default: true
           config:
             project: <project>
             snapshotLocation: us-west1 9

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.

   6

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   7

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

2. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - aws
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: minio
             s3Url: <url> 5
             insecureSkipTLSVerify: "true"
             s3ForcePathStyle: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name: cloud-credentials 6
           objectStorage:
             bucket: <bucket_name> 7
             prefix: <prefix> 8

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the URL of the S3 endpoint.

   6

   If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   7

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   8

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
3. Click YAML View and update the parameters of the DataProtectionApplication manifest:
4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Retrieve the backupStorageLocations CRs by entering the following command:

   $ oc get backupStorageLocations -n openshift-adp

   Example output

   NAMESPACE       NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   openshift-adp   velero-sample-1   Available   11s              31m

2. Create a Backup CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: <backup>
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     hooks: {}
     includedNamespaces:
     - <namespace> 1
     includedResources: [] 2
     excludedResources: [] 3
     storageLocation: <velero-sample-1> 4
     ttl: 720h0m0s
     labelSelector: 5
       matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>
     orLabelSelectors: 6
     - matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>

   1

   Specify an array of namespaces to back up.

   2

   Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.

   3

   Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.

   4

   Specify the name of the backupStorageLocations CR.

   5

   Map of {key,value} pairs of backup resources that have all of the specified labels.

   6

   Map of {key,value} pairs of backup resources that have one or more of the specified labels.

3. Verify that the status of the Backup CR is Completed:

   $ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'

Procedure

1. Configure a Restic secret by creating a .yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name>
   type: Opaque
   stringData:
   # The repository encryption key
     RESTIC_PASSWORD: my-secure-restic-password

2. Create a DPA CR similar to the following example. The default plugins include CSI.

3. Add the restic secret name from the step above to your DPA CR as spec.features.dataMover.credentialName. If this step is not completed, then it will default to the secret name dm-credential.

   Note

   In this DPA, the CSI and VSM are included as defaultPlugins. Also included is the dataMover.enable flag.

   Example Data Protection Application (DPA) CR

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: velero-sample
     namespace: openshift-adp
   spec:
     features:
       dataMover:
         enable: true
         credentialName: <secret-name>
         maxConcurrentBackupVolumes: "3" 1
         maxConcurrentRestoreVolumes: "3" 2
         pruneInterval: "14" 3
         volumeOptionsForStorageClasses: 4
           gp2-csi-copy-1:
             destinationVolumeOptions:
               storageClassName: csi-copy-2
             sourceVolumeOptions:
               storageClassName: csi-copy-1
     backupLocations:
       - velero:
           config:
             profile: default
             region: us-east-1
           credential:
             key: cloud
             name: cloud-credentials
           default: true
           objectStorage:
             bucket: <bucket_name>
             prefix: <bucket-prefix>
           provider: aws
     configuration:
       restic:
         enable: false
       velero:
         defaultPlugins:
           - openshift
           - aws
           - csi
           - vsm 5

   1

   OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for backup. The default value is 10.

   2

   OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for restore. The default value is 10.

   3

   OADP 1.2 only. Optional: Specify the number of days, between running Restic pruning on the repository. The prune operation repacks the data to free space, but it can also generate significant I/O traffic as a part of the process. Setting this option allows a trade-off between storage consumption, from no longer referenced data, and access costs.

   4

   OADP 1.2 only. Optional: Specify VolumeSync volume options for backup and restore.

   5

   OADP 1.2 only.

   The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

   Example VolumeSnapshotBackup CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotBackup
   metadata:
     name: <vsb_name>
     namespace: <namespace_name> 1
   spec:
     volumeSnapshotContent:
       name: <snapcontent_name>
     protectedNamespace: <adp_namespace>
     resticSecretRef:
       name: <restic_secret_name>

   1

   Specify the namespace where the volume snapshot exists.

   Example VolumeSnapshotRestore CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotRestore
   metadata:
     name: <vsr_name>
     namespace: <namespace_name> 1
   spec:
     protectedNamespace: <protected_ns> 2
     resticSecretRef:
       name: <restic_secret_name>
     volumeSnapshotMoverBackupRef:
       sourcePVCData:
         name: <source_pvc_name>
         size: <source_pvc_size>
       resticrepository: <your_restic_repo>
       volumeSnapshotClassName: <vsclass_name>

   1

   Specify the namespace where the volume snapshot exists.

   2

   Specify the namespace where the Operator is installed. The default is openshift-adp.

4. You can back up a volume snapshot by performing the following steps:

   1. Create a backup CR:

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: <backup_name>
        namespace: <protected_ns> 1
      spec:
        includedNamespaces:
        - <app_ns>
        storageLocation: velero-sample-1

      1

      Specify the namespace where the Operator is installed. The default namespace is openshift-adp.

   2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands:

      $ oc get vsb -n <app_ns>

      $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"

      A snapshot is created in the object store was configured in the DPA.

      Note

      If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.

5. You can restore a volume snapshot by performing the following steps:

   1. Delete the application namespace and the volumeSnapshotContent that was created by the Velero CSI plugin.

   2. Create a Restore CR and set restorePVs to true.

      Example Restore CR

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: <restore_name>
        namespace: <protected_ns>
      spec:
        backupName: <previous_backup_name>
        restorePVs: true

   3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command:

      $ oc get vsr -n <app_ns>

      $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"

   4. Check whether your application data and resources have been restored.

      Note

      If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

Procedure

1. Retrieve the backupStorageLocations CRs:

   $ oc get backupStorageLocations -n openshift-adp

   Example output

   NAMESPACE       NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   openshift-adp   velero-sample-1   Available   11s              31m

2. Create a Schedule CR, as in the following example:

   $ cat << EOF | oc apply -f -
   apiVersion: velero.io/v1
   kind: Schedule
   metadata:
     name: <schedule>
     namespace: openshift-adp
   spec:
     schedule: 0 7 * * * 1
     template:
       hooks: {}
       includedNamespaces:
       - <namespace> 2
       storageLocation: <velero-sample-1> 3
       defaultVolumesToRestic: true 4
       ttl: 720h0m0s
   EOF

   1

   cron expression to schedule the backup, for example, 0 7 * * * to perform a backup every day at 7:00.

   2

   Array of namespaces to back up.

   3

   Name of the backupStorageLocations CR.

   4

   Optional: Add the defaultVolumesToRestic: true key-value pair if you are backing up volumes with Restic.

3. Verify that the status of the Schedule CR is Completed after the scheduled backup runs:

   $ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'

Procedure

1. Create a Restore CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: <restore>
     namespace: openshift-adp
   spec:
     backupName: <backup> 1
     includedResources: [] 2
     excludedResources:
     - nodes
     - events
     - events.events.k8s.io
     - backups.velero.io
     - restores.velero.io
     - resticrepositories.velero.io
     restorePVs: true 3

   1

   Name of the Backup CR.

   2

   Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, po for pods) or fully-qualified. If unspecified, all resources are included.

   3

   Optional: The restorePVs parameter can be set to false in order to turn off restore of PersistentVolumes from VolumeSnapshot of Container Storage Interface (CSI) snapshots, or from native snapshots when VolumeSnapshotLocation is configured.

2. Verify that the status of the Restore CR is Completed by entering the following command:

   $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

3. Verify that the backup resources have been restored by entering the following command:

   $ oc get all -n <namespace> 1

   1

   Namespace that you backed up.

4. If you use Restic to restore DeploymentConfig objects or if you use post-restore hooks, run the dc-restic-post-restore.sh cleanup script by entering the following command:

   $ bash dc-restic-post-restore.sh <restore-name>

   Note

   In the course of the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow Restic and post-restore hooks to complete their actions on the restored pods. The cleanup script removes these disconnected pods and scale any DeploymentConfig objects back up to the appropriate number of replicas.

   Example 4.1. dc-restic-post-restore.sh cleanup script

   #!/bin/bash
   set -e
   
   # if sha256sum exists, use it to check the integrity of the file
   if command -v sha256sum >/dev/null 2>&1; then
     CHECKSUM_CMD="sha256sum"
   else
     CHECKSUM_CMD="shasum -a 256"
   fi
   
   label_name () {
       if [ "${#1}" -le "63" ]; then
   	echo $1
   	return
       fi
       sha=$(echo -n $1|$CHECKSUM_CMD)
       echo "${1:0:57}${sha:0:6}"
   }
   
   OADP_NAMESPACE=${OADP_NAMESPACE:=openshift-adp}
   
   if [[ $# -ne 1 ]]; then
       echo "usage: ${BASH_SOURCE} restore-name"
       exit 1
   fi
   
   echo using OADP Namespace $OADP_NAMESPACE
   echo restore: $1
   
   label=$(label_name $1)
   echo label: $label
   
   echo Deleting disconnected restore pods
   oc delete pods -l oadp.openshift.io/disconnected-from-dc=$label
   
   for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
   do
       IFS=',' read -ra dc_arr <<< "$dc"
       if [ ${#dc_arr[0]} -gt 0 ]; then
   	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
   	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
   spec:
     replicas: ${dc_arr[2]}
     paused: ${dc_arr[3]}
   EOF
       fi
   done

Solution

1. Edit the value of the spec.snapshotLocations.velero.config.region key in the DataProtectionApplication manifest so that the snapshot location is in the same region as the PV.
2. Create a new Backup CR.

Solution

1. Retrieve the details of the Backup CR:

   $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
     backup describe <backup>

2. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

   You do not need to clean up the backup location because a Backup CR in progress has not uploaded files to object storage.

3. Create a new Backup CR.

Solution

1. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

2. If required, clean up the stored data on the BackupStorageLocation to free up space.

3. Apply label velero.io/csi-volumesnapshot-class=true to the VolumeSnapshotClass object:

   $ oc label volumesnapshotclass/<snapclass_name> velero.io/csi-volumesnapshot-class=true

4. Create a new Backup CR.

1. Create a supplemental group for Restic on the NFS data volume.
2. Set the setgid bit on the NFS directories so that group ownership is inherited.

3. Add the spec.configuration.restic.supplementalGroups parameter and the group ID to the DataProtectionApplication manifest, as in the following example:

   spec:
     configuration:
       restic:
         enable: true
         supplementalGroups:
         - <group_id> 1

   1

   Specify the supplemental group ID.

4. Wait for the Restic pods to restart so that the changes are applied.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

2. Add or enable the enableUserWorkload option in the data section’s config.yaml field:

   apiVersion: v1
   data:
     config.yaml: |
       enableUserWorkload: true 1
   kind: ConfigMap
   metadata:
   # ...

   1

   Add this option or set to true

3. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the openshift-user-workload-monitoring namespace:

   $ oc get pods -n openshift-user-workload-monitoring

   Example output

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-6844b4b99c-b57j9   2/2     Running   0          43s
   prometheus-user-workload-0             5/5     Running   0          32s
   prometheus-user-workload-1             5/5     Running   0          32s
   thanos-ruler-user-workload-0           3/3     Running   0          32s
   thanos-ruler-user-workload-1           3/3     Running   0          32s

4. Verify the existence of the user-workload-monitoring-config ConfigMap in the openshift-user-workload-monitoring. If it exists, skip the remaining steps in this procedure.

   $ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring

   Example output

   Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

5. Create a user-workload-monitoring-config ConfigMap object for the User Workload Monitoring, and save it under the 2_configure_user_workload_monitoring.yaml file name:

   Example output

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |

6. Apply the 2_configure_user_workload_monitoring.yaml file:

   $ oc apply -f 2_configure_user_workload_monitoring.yaml
   configmap/user-workload-monitoring-config created

Procedure

1. Ensure the openshift-adp-velero-metrics-svc service exists. It should contain app.kubernetes.io/name=velero label, which will be used as selector for the ServiceMonitor object.

   $ oc get svc -n openshift-adp -l app.kubernetes.io/name=velero

   Example output

   NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   openshift-adp-velero-metrics-svc   ClusterIP   172.30.38.244   <none>        8085/TCP   1h

2. Create a ServiceMonitor YAML file that matches the existing service label, and save the file as 3_create_oadp_service_monitor.yaml. The service monitor is created in the openshift-adp namespace where the openshift-adp-velero-metrics-svc service resides.

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       app: oadp-service-monitor
     name: oadp-service-monitor
     namespace: openshift-adp
   spec:
     endpoints:
     - interval: 30s
       path: /metrics
       targetPort: 8085
       scheme: http
     selector:
       matchLabels:
         app.kubernetes.io/name: "velero"

3. Apply the 3_create_oadp_service_monitor.yaml file:

   $ oc apply -f 3_create_oadp_service_monitor.yaml

   Example output

   servicemonitor.monitoring.coreos.com/oadp-service-monitor created

Procedure

1. Create a PrometheusRule YAML file with the sample OADPBackupFailing alert and save it as 4_create_oadp_alert_rule.yaml.

   Sample OADPBackupFailing alert

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: sample-oadp-alert
     namespace: openshift-adp
   spec:
     groups:
     - name: sample-oadp-backup-alert
       rules:
       - alert: OADPBackupFailing
         annotations:
           description: 'OADP had {{$value | humanize}} backup failures over the last 2 hours.'
           summary: OADP has issues creating backups
         expr: |
           increase(velero_backup_failure_total{job="openshift-adp-velero-metrics-svc"}[2h]) > 0
         for: 5m
         labels:
           severity: warning

   In this sample, the Alert displays under the following conditions:

   • There is an increase of new failing backups during the 2 last hours that is greater than 0 and the state persists for at least 5 minutes.
   • If the time of the first increase is less than 5 minutes, the Alert will be in a Pending state, after which it will turn into a Firing state.

2. Apply the 4_create_oadp_alert_rule.yaml file, which creates the PrometheusRule object in the openshift-adp namespace:

   $ oc apply -f 4_create_oadp_alert_rule.yaml

   Example output

   prometheusrule.monitoring.coreos.com/sample-oadp-alert created

1. Preferred version of the destination cluster
2. Preferred version of the source_ cluster
3. Common non-preferred supported version with the highest Kubernetes version priority