Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Set the BUCKET variable:

   $ BUCKET=<your_bucket>

2. Set the REGION variable:

   $ REGION=<your_region>

3. Create an AWS S3 bucket:

   $ aws s3api create-bucket \
       --bucket $BUCKET \
       --region $REGION \
       --create-bucket-configuration LocationConstraint=$REGION 1

   1

   us-east-1 does not support a LocationConstraint. If your region is us-east-1, omit --create-bucket-configuration LocationConstraint=$REGION.

4. Create an IAM user:

   $ aws iam create-user --user-name velero 1

   1

   If you want to use Velero to back up multiple clusters with multiple S3 buckets, create a unique user name for each cluster.

5. Create a velero-policy.json file:

   $ cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation",
                   "s3:ListBucketMultipartUploads"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF

6. Attach the policies to give the velero user the minimum necessary permissions:

   $ aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json

7. Create an access key for the velero user:

   $ aws iam create-access-key --user-name velero

   Example output

   {
     "AccessKey": {
           "UserName": "velero",
           "Status": "Active",
           "CreateDate": "2017-07-31T22:24:41.576Z",
           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,
           "AccessKeyId": <AWS_ACCESS_KEY_ID>
     }
   }

8. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object for AWS before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - openshift 1
           - aws
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - name: default
         velero:
           provider: aws
           default: true
           objectStorage:
             bucket: <bucket_name> 5
             prefix: <prefix> 6
           config:
             region: <region>
             profile: "default"
           credential:
             key: cloud
             name: cloud-credentials 7
     snapshotLocations: 8
       - name: default
         velero:
           provider: aws
           config:
             region: <region> 9
             profile: "default"

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   6

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   7

   Specify the name of the Secret object that you created. If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Log in to Azure:

   $ az login

2. Set the AZURE_RESOURCE_GROUP variable:

   $ AZURE_RESOURCE_GROUP=Velero_Backups

3. Create an Azure resource group:

   $ az group create -n $AZURE_RESOURCE_GROUP --location CentralUS 1

   1

   Specify your location.

4. Set the AZURE_STORAGE_ACCOUNT_ID variable:

   $ AZURE_STORAGE_ACCOUNT_ID="velero$(uuidgen | cut -d '-' -f5 | tr '[A-Z]' '[a-z]')"

5. Create an Azure storage account:

   $ az storage account create \
       --name $AZURE_STORAGE_ACCOUNT_ID \
       --resource-group $AZURE_RESOURCE_GROUP \
       --sku Standard_GRS \
       --encryption-services blob \
       --https-only true \
       --kind BlobStorage \
       --access-tier Hot

6. Set the BLOB_CONTAINER variable:

   $ BLOB_CONTAINER=velero

7. Create an Azure Blob storage container:

   $ az storage container create \
     -n $BLOB_CONTAINER \
     --public-access off \
     --account-name $AZURE_STORAGE_ACCOUNT_ID

8. Obtain the storage account access key:

   $ AZURE_STORAGE_ACCOUNT_ACCESS_KEY=`az storage account keys list \
     --account-name $AZURE_STORAGE_ACCOUNT_ID \
     --query "[?keyName == 'key1'].value" -o tsv`

9. Create a custom role that has the minimum required permissions:

   AZURE_ROLE=Velero
   az role definition create --role-definition '{
      "Name": "'$AZURE_ROLE'",
      "Description": "Velero related permissions to perform backups, restores and deletions",
      "Actions": [
          "Microsoft.Compute/disks/read",
          "Microsoft.Compute/disks/write",
          "Microsoft.Compute/disks/endGetAccess/action",
          "Microsoft.Compute/disks/beginGetAccess/action",
          "Microsoft.Compute/snapshots/read",
          "Microsoft.Compute/snapshots/write",
          "Microsoft.Compute/snapshots/delete",
          "Microsoft.Storage/storageAccounts/listkeys/action",
          "Microsoft.Storage/storageAccounts/regeneratekey/action"
      ],
      "AssignableScopes": ["/subscriptions/'$AZURE_SUBSCRIPTION_ID'"]
      }'

10. Create a credentials-velero file:

    $ cat << EOF > ./credentials-velero
    AZURE_SUBSCRIPTION_ID=${AZURE_SUBSCRIPTION_ID}
    AZURE_TENANT_ID=${AZURE_TENANT_ID}
    AZURE_CLIENT_ID=${AZURE_CLIENT_ID}
    AZURE_CLIENT_SECRET=${AZURE_CLIENT_SECRET}
    AZURE_RESOURCE_GROUP=${AZURE_RESOURCE_GROUP}
    AZURE_STORAGE_ACCOUNT_ACCESS_KEY=${AZURE_STORAGE_ACCOUNT_ACCESS_KEY} 1
    AZURE_CLOUD_NAME=AzurePublicCloud
    EOF

    1

    Mandatory. You cannot back up internal images if the credentials-velero file contains only the service principal credentials.

    You use the credentials-velero file to create a Secret object for Azure before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - azure
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             resourceGroup: <azure_resource_group> 5
             storageAccount: <azure_storage_account_id> 6
             subscriptionId: <azure_subscription_id> 7
             storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
           credential:
             key: cloud
             name: cloud-credentials-azure  8
           provider: azure
           default: true
           objectStorage:
             bucket: <bucket_name> 9
             prefix: <prefix> 10
     snapshotLocations: 11
       - velero:
           config:
             resourceGroup: <azure_resource_group>
             subscriptionId: <azure_subscription_id>
             incremental: "true"
           name: default
           provider: azure

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the Azure resource group.

   6

   Specify the Azure storage account ID.

   7

   Specify the Azure subscription ID.

   8

   If you do not specify this value, the default name, cloud-credentials-azure, is used. If you specify a custom name, the custom name is used for the backup location.

   9

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   10

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   11

   You do not need to specify a snapshot location if you use CSI snapshots or Restic to back up PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Log in to GCP:

   $ gcloud auth login

2. Set the BUCKET variable:

   $ BUCKET=<bucket> 1

   1

   Specify your bucket name.

3. Create the storage bucket:

   $ gsutil mb gs://$BUCKET/

4. Set the PROJECT_ID variable to your active project:

   $ PROJECT_ID=$(gcloud config get-value project)

5. Create a service account:

   $ gcloud iam service-accounts create velero \
       --display-name "Velero service account"

6. List your service accounts:

   $ gcloud iam service-accounts list

7. Set the SERVICE_ACCOUNT_EMAIL variable to match its email value:

   $ SERVICE_ACCOUNT_EMAIL=$(gcloud iam service-accounts list \
       --filter="displayName:Velero service account" \
       --format 'value(email)')

8. Attach the policies to give the velero user the minimum necessary permissions:

   $ ROLE_PERMISSIONS=(
       compute.disks.get
       compute.disks.create
       compute.disks.createSnapshot
       compute.snapshots.get
       compute.snapshots.create
       compute.snapshots.useReadOnly
       compute.snapshots.delete
       compute.zones.get
       storage.objects.create
       storage.objects.delete
       storage.objects.get
       storage.objects.list
       iam.serviceAccounts.signBlob
   )

9. Create the velero.server custom role:

   $ gcloud iam roles create velero.server \
       --project $PROJECT_ID \
       --title "Velero Server" \
       --permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

10. Add IAM policy binding to the project:

    $ gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
        --role projects/$PROJECT_ID/roles/velero.server

11. Update the IAM service account:

    $ gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

12. Save the IAM service account keys to the credentials-velero file in the current directory:

    $ gcloud iam service-accounts keys create credentials-velero \
        --iam-account $SERVICE_ACCOUNT_EMAIL

    You use the credentials-velero file to create a Secret object for GCP before you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - gcp
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           provider: gcp
           default: true
           credential:
             key: cloud
             name: cloud-credentials-gcp 5
           objectStorage:
             bucket: <bucket_name> 6
             prefix: <prefix> 7
     snapshotLocations: 8
       - velero:
           provider: gcp
           default: true
           config:
             project: <project>
             snapshotLocation: us-west1 9

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   If you do not specify this value, the default name, cloud-credentials-gcp, is used. If you specify a custom name, the custom name is used for the backup location.

   6

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   7

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

   8

   Specify a snapshot location, unless you use CSI snapshots or Restic to back up PVs.

   9

   The snapshot location must be in the same region as the PVs.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Obtain the S3 endpoint, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY by running the describe command on the NooBaa custom resource.

2. Create a credentials-velero file:

   $ cat << EOF > ./credentials-velero
   [default]
   aws_access_key_id=<AWS_ACCESS_KEY_ID>
   aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
   EOF

   You use the credentials-velero file to create a Secret object when you install the Data Protection Application.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.

3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: <dpa_sample>
     namespace: openshift-adp
   spec:
     configuration:
       velero:
         defaultPlugins:
           - aws
           - openshift 1
         resourceTimeout: 10m 2
       restic:
         enable: true 3
         podConfig:
           nodeSelector: <node_selector> 4
     backupLocations:
       - velero:
           config:
             profile: "default"
             region: minio
             s3Url: <url> 5
             insecureSkipTLSVerify: "true"
             s3ForcePathStyle: "true"
           provider: aws
           default: true
           credential:
             key: cloud
             name: cloud-credentials 6
           objectStorage:
             bucket: <bucket_name> 7
             prefix: <prefix> 8

   1

   The openshift plugin is mandatory.

   2

   Specify how many minutes to wait for several Velero resources before timeout occurs, such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability. The default is 10m.

   3

   Set to false, if you want to disable the Restic installation. Restic deploys a daemon set, which means that each worker node has Restic pods running. You can configure Restic for backups by adding spec.defaultVolumesToRestic: true to the Backup CR.

   4

   Specify on which nodes Restic is available. By default, Restic runs on all nodes.

   5

   Specify the URL of the S3 endpoint.

   6

   If you do not specify this value, the default name, cloud-credentials, is used. If you specify a custom name, the custom name is used for the backup location.

   7

   Specify a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.

   8

   Specify a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.

4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Use the Filter by keyword field to find the OADP Operator.
3. Select the OADP Operator and click Install.
4. Click Install to install the Operator in the openshift-adp project.
5. Click Operators → Installed Operators to verify the installation.

Procedure

1. Click Operators → Installed Operators and select the OADP Operator.
2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
3. Click YAML View and update the parameters of the DataProtectionApplication manifest:
4. Click Create.

5. Verify the installation by viewing the OADP resources:

   $ oc get all -n openshift-adp

   Example output

   NAME                                                     READY   STATUS    RESTARTS   AGE
   pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
   pod/restic-9cq4q                                         1/1     Running   0          94s
   pod/restic-m4lts                                         1/1     Running   0          94s
   pod/restic-pv4kr                                         1/1     Running   0          95s
   pod/velero-588db7f655-n842v                              1/1     Running   0          95s
   
   NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
   service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
   
   NAME                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/restic   3         3         3       3            3           <none>          96s
   
   NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
   deployment.apps/velero                              1/1     1            1           96s
   
   NAME                                                           DESIRED   CURRENT   READY   AGE
   replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
   replicaset.apps/velero-588db7f655                              1         1         1       96s

Procedure

1. Retrieve the backupStorageLocations CRs by entering the following command:

   $ oc get backupStorageLocations -n openshift-adp

   Example output

   NAMESPACE       NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   openshift-adp   velero-sample-1   Available   11s              31m

2. Create a Backup CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: <backup>
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     hooks: {}
     includedNamespaces:
     - <namespace> 1
     includedResources: [] 2
     excludedResources: [] 3
     storageLocation: <velero-sample-1> 4
     ttl: 720h0m0s
     labelSelector: 5
       matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>
     orLabelSelectors: 6
     - matchLabels:
         app=<label_1>
         app=<label_2>
         app=<label_3>

   1

   Specify an array of namespaces to back up.

   2

   Optional: Specify an array of resources to include in the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified. If unspecified, all resources are included.

   3

   Optional: Specify an array of resources to exclude from the backup. Resources might be shortcuts (for example, 'po' for 'pods') or fully-qualified.

   4

   Specify the name of the backupStorageLocations CR.

   5

   Map of {key,value} pairs of backup resources that have all of the specified labels.

   6

   Map of {key,value} pairs of backup resources that have one or more of the specified labels.

3. Verify that the status of the Backup CR is Completed:

   $ oc get backup -n openshift-adp <backup> -o jsonpath='{.status.phase}'

Procedure

1. Configure a Restic secret by creating a .yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <secret_name>
   type: Opaque
   stringData:
   # The repository encryption key
     RESTIC_PASSWORD: my-secure-restic-password

2. Create a DPA CR similar to the following example. The default plugins include CSI.

3. Add the restic secret name from the step above to your DPA CR as spec.features.dataMover.credentialName. If this step is not completed, then it will default to the secret name dm-credential.

   Note

   In this DPA, the CSI and VSM are included as defaultPlugins. Also included is the dataMover.enable flag.

   Example Data Protection Application (DPA) CR

   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
     name: velero-sample
     namespace: openshift-adp
   spec:
     features:
       dataMover:
         enable: true
         credentialName: <secret-name>
         maxConcurrentBackupVolumes: "3" 1
         maxConcurrentRestoreVolumes: "3" 2
         pruneInterval: "14" 3
         volumeOptionsForStorageClasses: 4
           gp2-csi-copy-1:
             destinationVolumeOptions:
               storageClassName: csi-copy-2
             sourceVolumeOptions:
               storageClassName: csi-copy-1
     backupLocations:
       - velero:
           config:
             profile: default
             region: us-east-1
           credential:
             key: cloud
             name: cloud-credentials
           default: true
           objectStorage:
             bucket: <bucket_name>
             prefix: <bucket-prefix>
           provider: aws
     configuration:
       restic:
         enable: false
       velero:
         defaultPlugins:
           - openshift
           - aws
           - csi
           - vsm 5

   1

   OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for backup. The default value is 10.

   2

   OADP 1.2 only. Optional: Specify the upper limit of the number of snapshots allowed to be queued for restore. The default value is 10.

   3

   OADP 1.2 only. Optional: Specify the number of days, between running Restic pruning on the repository. The prune operation repacks the data to free space, but it can also generate significant I/O traffic as a part of the process. Setting this option allows a trade-off between storage consumption, from no longer referenced data, and access costs.

   4

   OADP 1.2 only. Optional: Specify VolumeSync volume options for backup and restore.

   5

   OADP 1.2 only.

   The OADP Operator installs two custom resource definitions (CRDs), VolumeSnapshotBackup and VolumeSnapshotRestore.

   Example VolumeSnapshotBackup CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotBackup
   metadata:
     name: <vsb_name>
     namespace: <namespace_name> 1
   spec:
     volumeSnapshotContent:
       name: <snapcontent_name>
     protectedNamespace: <adp_namespace>
     resticSecretRef:
       name: <restic_secret_name>

   1

   Specify the namespace where the volume snapshot exists.

   Example VolumeSnapshotRestore CRD

   apiVersion: datamover.oadp.openshift.io/v1alpha1
   kind: VolumeSnapshotRestore
   metadata:
     name: <vsr_name>
     namespace: <namespace_name> 1
   spec:
     protectedNamespace: <protected_ns> 2
     resticSecretRef:
       name: <restic_secret_name>
     volumeSnapshotMoverBackupRef:
       sourcePVCData:
         name: <source_pvc_name>
         size: <source_pvc_size>
       resticrepository: <your_restic_repo>
       volumeSnapshotClassName: <vsclass_name>

   1

   Specify the namespace where the volume snapshot exists.

   2

   Specify the namespace where the Operator is installed. The default is openshift-adp.

4. You can back up a volume snapshot by performing the following steps:

   1. Create a backup CR:

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: <backup_name>
        namespace: <protected_ns> 1
      spec:
        includedNamespaces:
        - <app_ns>
        storageLocation: velero-sample-1

      1

      Specify the namespace where the Operator is installed. The default namespace is openshift-adp.

   2. Wait up to 10 minutes and check whether the VolumeSnapshotBackup CR status is Completed by entering the following commands:

      $ oc get vsb -n <app_ns>

      $ oc get vsb <vsb_name> -n <app_ns> -o jsonpath="{.status.phase}"

      A snapshot is created in the object store was configured in the DPA.

      Note

      If the status of the VolumeSnapshotBackup CR becomes Failed, refer to the Velero logs for troubleshooting.

5. You can restore a volume snapshot by performing the following steps:

   1. Delete the application namespace and the volumeSnapshotContent that was created by the Velero CSI plugin.

   2. Create a Restore CR and set restorePVs to true.

      Example Restore CR

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: <restore_name>
        namespace: <protected_ns>
      spec:
        backupName: <previous_backup_name>
        restorePVs: true

   3. Wait up to 10 minutes and check whether the VolumeSnapshotRestore CR status is Completed by entering the following command:

      $ oc get vsr -n <app_ns>

      $ oc get vsr <vsr_name> -n <app_ns> -o jsonpath="{.status.phase}"

   4. Check whether your application data and resources have been restored.

      Note

      If the status of the VolumeSnapshotRestore CR becomes 'Failed', refer to the Velero logs for troubleshooting.

Procedure

1. Retrieve the backupStorageLocations CRs:

   $ oc get backupStorageLocations -n openshift-adp

   Example output

   NAMESPACE       NAME              PHASE       LAST VALIDATED   AGE   DEFAULT
   openshift-adp   velero-sample-1   Available   11s              31m

2. Create a Schedule CR, as in the following example:

   $ cat << EOF | oc apply -f -
   apiVersion: velero.io/v1
   kind: Schedule
   metadata:
     name: <schedule>
     namespace: openshift-adp
   spec:
     schedule: 0 7 * * * 1
     template:
       hooks: {}
       includedNamespaces:
       - <namespace> 2
       storageLocation: <velero-sample-1> 3
       defaultVolumesToRestic: true 4
       ttl: 720h0m0s
   EOF

   1

   cron expression to schedule the backup, for example, 0 7 * * * to perform a backup every day at 7:00.

   2

   Array of namespaces to back up.

   3

   Name of the backupStorageLocations CR.

   4

   Optional: Add the defaultVolumesToRestic: true key-value pair if you are backing up volumes with Restic.

3. Verify that the status of the Schedule CR is Completed after the scheduled backup runs:

   $ oc get schedule -n openshift-adp <schedule> -o jsonpath='{.status.phase}'

Procedure

1. Create a Restore CR, as in the following example:

   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: <restore>
     namespace: openshift-adp
   spec:
     backupName: <backup> 1
     includedResources: [] 2
     excludedResources:
     - nodes
     - events
     - events.events.k8s.io
     - backups.velero.io
     - restores.velero.io
     - resticrepositories.velero.io
     restorePVs: true 3

   1

   Name of the Backup CR.

   2

   Optional: Specify an array of resources to include in the restore process. Resources might be shortcuts (for example, po for pods) or fully-qualified. If unspecified, all resources are included.

   3

   Optional: The restorePVs parameter can be set to false in order to turn off restore of PersistentVolumes from VolumeSnapshot of Container Storage Interface (CSI) snapshots, or from native snapshots when VolumeSnapshotLocation is configured.

2. Verify that the status of the Restore CR is Completed by entering the following command:

   $ oc get restore -n openshift-adp <restore> -o jsonpath='{.status.phase}'

3. Verify that the backup resources have been restored by entering the following command:

   $ oc get all -n <namespace> 1

   1

   Namespace that you backed up.

4. If you use Restic to restore DeploymentConfig objects or if you use post-restore hooks, run the dc-restic-post-restore.sh cleanup script by entering the following command:

   $ bash dc-restic-post-restore.sh <restore-name>

   Note

   In the course of the restore process, the OADP Velero plug-ins scale down the DeploymentConfig objects and restore the pods as standalone pods to prevent the cluster from deleting the restored DeploymentConfig pods immediately on restore and to allow Restic and post-restore hooks to complete their actions on the restored pods. The cleanup script removes these disconnected pods and scale any DeploymentConfig objects back up to the appropriate number of replicas.

   Example 4.1. dc-restic-post-restore.sh cleanup script

   #!/bin/bash
   set -e
   
   # if sha256sum exists, use it to check the integrity of the file
   if command -v sha256sum >/dev/null 2>&1; then
     CHECKSUM_CMD="sha256sum"
   else
     CHECKSUM_CMD="shasum -a 256"
   fi
   
   label_name () {
       if [ "${#1}" -le "63" ]; then
   	echo $1
   	return
       fi
       sha=$(echo -n $1|$CHECKSUM_CMD)
       echo "${1:0:57}${sha:0:6}"
   }
   
   OADP_NAMESPACE=${OADP_NAMESPACE:=openshift-adp}
   
   if [[ $# -ne 1 ]]; then
       echo "usage: ${BASH_SOURCE} restore-name"
       exit 1
   fi
   
   echo using OADP Namespace $OADP_NAMESPACE
   echo restore: $1
   
   label=$(label_name $1)
   echo label: $label
   
   echo Deleting disconnected restore pods
   oc delete pods -l oadp.openshift.io/disconnected-from-dc=$label
   
   for dc in $(oc get dc --all-namespaces -l oadp.openshift.io/replicas-modified=$label -o jsonpath='{range .items[*]}{.metadata.namespace}{","}{.metadata.name}{","}{.metadata.annotations.oadp\.openshift\.io/original-replicas}{","}{.metadata.annotations.oadp\.openshift\.io/original-paused}{"\n"}')
   do
       IFS=',' read -ra dc_arr <<< "$dc"
       if [ ${#dc_arr[0]} -gt 0 ]; then
   	echo Found deployment ${dc_arr[0]}/${dc_arr[1]}, setting replicas: ${dc_arr[2]}, paused: ${dc_arr[3]}
   	cat <<EOF | oc patch dc  -n ${dc_arr[0]} ${dc_arr[1]} --patch-file /dev/stdin
   spec:
     replicas: ${dc_arr[2]}
     paused: ${dc_arr[3]}
   EOF
       fi
   done

Solution

1. Edit the value of the spec.snapshotLocations.velero.config.region key in the DataProtectionApplication manifest so that the snapshot location is in the same region as the PV.
2. Create a new Backup CR.

Solution

1. Retrieve the details of the Backup CR:

   $ oc -n {namespace} exec deployment/velero -c velero -- ./velero \
     backup describe <backup>

2. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

   You do not need to clean up the backup location because a Backup CR in progress has not uploaded files to object storage.

3. Create a new Backup CR.

Solution

1. Delete the Backup CR:

   $ oc delete backup <backup> -n openshift-adp

2. If required, clean up the stored data on the BackupStorageLocation to free up space.

3. Apply label velero.io/csi-volumesnapshot-class=true to the VolumeSnapshotClass object:

   $ oc label volumesnapshotclass/<snapclass_name> velero.io/csi-volumesnapshot-class=true

4. Create a new Backup CR.

1. Create a supplemental group for Restic on the NFS data volume.
2. Set the setgid bit on the NFS directories so that group ownership is inherited.

3. Add the spec.configuration.restic.supplementalGroups parameter and the group ID to the DataProtectionApplication manifest, as in the following example:

   spec:
     configuration:
       restic:
         enable: true
         supplementalGroups:
         - <group_id> 1

   1

   Specify the supplemental group ID.

4. Wait for the Restic pods to restart so that the changes are applied.

Procedure

1. Edit the cluster-monitoring-config ConfigMap object in the openshift-monitoring namespace:

   $ oc edit configmap cluster-monitoring-config -n openshift-monitoring

2. Add or enable the enableUserWorkload option in the data section’s config.yaml field:

   apiVersion: v1
   data:
     config.yaml: |
       enableUserWorkload: true 1
   kind: ConfigMap
   metadata:
   # ...

   1

   Add this option or set to true

3. Wait a short period of time to verify the User Workload Monitoring Setup by checking if the following components are up and running in the openshift-user-workload-monitoring namespace:

   $ oc get pods -n openshift-user-workload-monitoring

   Example output

   NAME                                   READY   STATUS    RESTARTS   AGE
   prometheus-operator-6844b4b99c-b57j9   2/2     Running   0          43s
   prometheus-user-workload-0             5/5     Running   0          32s
   prometheus-user-workload-1             5/5     Running   0          32s
   thanos-ruler-user-workload-0           3/3     Running   0          32s
   thanos-ruler-user-workload-1           3/3     Running   0          32s

4. Verify the existence of the user-workload-monitoring-config ConfigMap in the openshift-user-workload-monitoring. If it exists, skip the remaining steps in this procedure.

   $ oc get configmap user-workload-monitoring-config -n openshift-user-workload-monitoring

   Example output

   Error from server (NotFound): configmaps "user-workload-monitoring-config" not found

5. Create a user-workload-monitoring-config ConfigMap object for the User Workload Monitoring, and save it under the 2_configure_user_workload_monitoring.yaml file name:

   Example output

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |

6. Apply the 2_configure_user_workload_monitoring.yaml file:

   $ oc apply -f 2_configure_user_workload_monitoring.yaml
   configmap/user-workload-monitoring-config created

Procedure

1. Ensure the openshift-adp-velero-metrics-svc service exists. It should contain app.kubernetes.io/name=velero label, which will be used as selector for the ServiceMonitor object.

   $ oc get svc -n openshift-adp -l app.kubernetes.io/name=velero

   Example output

   NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   openshift-adp-velero-metrics-svc   ClusterIP   172.30.38.244   <none>        8085/TCP   1h

2. Create a ServiceMonitor YAML file that matches the existing service label, and save the file as 3_create_oadp_service_monitor.yaml. The service monitor is created in the openshift-adp namespace where the openshift-adp-velero-metrics-svc service resides.

   Example ServiceMonitor object

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     labels:
       app: oadp-service-monitor
     name: oadp-service-monitor
     namespace: openshift-adp
   spec:
     endpoints:
     - interval: 30s
       path: /metrics
       targetPort: 8085
       scheme: http
     selector:
       matchLabels:
         app.kubernetes.io/name: "velero"

3. Apply the 3_create_oadp_service_monitor.yaml file:

   $ oc apply -f 3_create_oadp_service_monitor.yaml

   Example output

   servicemonitor.monitoring.coreos.com/oadp-service-monitor created

Procedure

1. Create a PrometheusRule YAML file with the sample OADPBackupFailing alert and save it as 4_create_oadp_alert_rule.yaml.

   Sample OADPBackupFailing alert

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     name: sample-oadp-alert
     namespace: openshift-adp
   spec:
     groups:
     - name: sample-oadp-backup-alert
       rules:
       - alert: OADPBackupFailing
         annotations:
           description: 'OADP had {{$value | humanize}} backup failures over the last 2 hours.'
           summary: OADP has issues creating backups
         expr: |
           increase(velero_backup_failure_total{job="openshift-adp-velero-metrics-svc"}[2h]) > 0
         for: 5m
         labels:
           severity: warning

   In this sample, the Alert displays under the following conditions:

   • There is an increase of new failing backups during the 2 last hours that is greater than 0 and the state persists for at least 5 minutes.
   • If the time of the first increase is less than 5 minutes, the Alert will be in a Pending state, after which it will turn into a Firing state.

2. Apply the 4_create_oadp_alert_rule.yaml file, which creates the PrometheusRule object in the openshift-adp namespace:

   $ oc apply -f 4_create_oadp_alert_rule.yaml

   Example output

   prometheusrule.monitoring.coreos.com/sample-oadp-alert created

1. Preferred version of the destination cluster
2. Preferred version of the source_ cluster
3. Common non-preferred supported version with the highest Kubernetes version priority

Procedure

1. Remove the unhealthy member.

   1. Choose a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc -n openshift-etcd get pods -l k8s-app=etcd

      Example output

      etcd-ip-10-0-131-183.ec2.internal                3/3     Running     0          123m
      etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
      etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m

   2. Connect to the running etcd container, passing in the name of a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal

   3. View the member list:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
      |        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
      +------------------+---------+------------------------------+---------------------------+---------------------------+
      | 6fc1e7c9db35841d | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
      | 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
      | ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
      +------------------+---------+------------------------------+---------------------------+---------------------------+

      Take note of the ID and the name of the unhealthy etcd member, because these values are needed later in the procedure. The $ etcdctl endpoint health command will list the removed member until the procedure of replacement is finished and a new member is added.

   4. Remove the unhealthy etcd member by providing the ID to the etcdctl member remove command:

      sh-4.2# etcdctl member remove 6fc1e7c9db35841d

      Example output

      Member 6fc1e7c9db35841d removed from cluster ead669ce1fbfb346

   5. View the member list again and verify that the member was removed:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
      |        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
      +------------------+---------+------------------------------+---------------------------+---------------------------+
      | 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
      | ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
      +------------------+---------+------------------------------+---------------------------+---------------------------+

      You can now exit the node shell.

      Important

      After you remove the member, the cluster might be unreachable for a short time while the remaining etcd instances reboot.

2. Turn off the quorum guard by entering the following command:

   $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'

   This command ensures that you can successfully re-create secrets and roll out the static pods.

3. Remove the old secrets for the unhealthy etcd member that was removed.

   1. List the secrets for the unhealthy etcd member that was removed.

      $ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal 1

      1

      Pass in the name of the unhealthy etcd member that you took note of earlier in this procedure.

      There is a peer, serving, and metrics secret as shown in the following output:

      Example output

      etcd-peer-ip-10-0-131-183.ec2.internal              kubernetes.io/tls                     2      47m
      etcd-serving-ip-10-0-131-183.ec2.internal           kubernetes.io/tls                     2      47m
      etcd-serving-metrics-ip-10-0-131-183.ec2.internal   kubernetes.io/tls                     2      47m

   2. Delete the secrets for the unhealthy etcd member that was removed.

      1. Delete the peer secret:

         $ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal

      2. Delete the serving secret:

         $ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal

      3. Delete the metrics secret:

         $ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal

4. Delete and recreate the control plane machine. After this machine is recreated, a new revision is forced and etcd scales up automatically.

   If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new master using the same method that was used to originally create it.

   1. Obtain the machine for the unhealthy member.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc get machines -n openshift-machine-api -o wide

      Example output

      NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
      clustername-8qw5l-master-0                  Running   m4.xlarge   us-east-1   us-east-1a   3h37m   ip-10-0-131-183.ec2.internal   aws:///us-east-1a/i-0ec2782f8287dfb7e   stopped 1
      clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
      clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
      clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
      clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
      clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

      1

      This is the control plane machine for the unhealthy node, ip-10-0-131-183.ec2.internal.

   2. Save the machine configuration to a file on your file system:

      $ oc get machine clustername-8qw5l-master-0 \ 1
          -n openshift-machine-api \
          -o yaml \
          > new-master-machine.yaml

      1

      Specify the name of the control plane machine for the unhealthy node.

   3. Edit the new-master-machine.yaml file that was created in the previous step to assign a new name and remove unnecessary fields.

      1. Remove the entire status section:

         status:
           addresses:
           - address: 10.0.131.183
             type: InternalIP
           - address: ip-10-0-131-183.ec2.internal
             type: InternalDNS
           - address: ip-10-0-131-183.ec2.internal
             type: Hostname
           lastUpdated: "2020-04-20T17:44:29Z"
           nodeRef:
             kind: Node
             name: ip-10-0-131-183.ec2.internal
             uid: acca4411-af0d-4387-b73e-52b2484295ad
           phase: Running
           providerStatus:
             apiVersion: awsproviderconfig.openshift.io/v1beta1
             conditions:
             - lastProbeTime: "2020-04-20T16:53:50Z"
               lastTransitionTime: "2020-04-20T16:53:50Z"
               message: machine successfully created
               reason: MachineCreationSucceeded
               status: "True"
               type: MachineCreation
             instanceId: i-0fdb85790d76d0c3f
             instanceState: stopped
             kind: AWSMachineProviderStatus

      2. Change the metadata.name field to a new name.

         It is recommended to keep the same base name as the old machine and change the ending number to the next available number. In this example, clustername-8qw5l-master-0 is changed to clustername-8qw5l-master-3.

         For example:

         apiVersion: machine.openshift.io/v1beta1
         kind: Machine
         metadata:
           ...
           name: clustername-8qw5l-master-3
           ...

      3. Remove the spec.providerID field:

           providerID: aws:///us-east-1a/i-0fdb85790d76d0c3f

   4. Delete the machine of the unhealthy member:

      $ oc delete machine -n openshift-machine-api clustername-8qw5l-master-0 1

      1

      Specify the name of the control plane machine for the unhealthy node.

   5. Verify that the machine was deleted:

      $ oc get machines -n openshift-machine-api -o wide

      Example output

      NAME                                        PHASE     TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
      clustername-8qw5l-master-1                  Running   m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
      clustername-8qw5l-master-2                  Running   m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
      clustername-8qw5l-worker-us-east-1a-wbtgd   Running   m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
      clustername-8qw5l-worker-us-east-1b-lrdxb   Running   m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
      clustername-8qw5l-worker-us-east-1c-pkg26   Running   m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

   6. Create the new machine using the new-master-machine.yaml file:

      $ oc apply -f new-master-machine.yaml

   7. Verify that the new machine has been created:

      $ oc get machines -n openshift-machine-api -o wide

      Example output

      NAME                                        PHASE          TYPE        REGION      ZONE         AGE     NODE                           PROVIDERID                              STATE
      clustername-8qw5l-master-1                  Running        m4.xlarge   us-east-1   us-east-1b   3h37m   ip-10-0-154-204.ec2.internal   aws:///us-east-1b/i-096c349b700a19631   running
      clustername-8qw5l-master-2                  Running        m4.xlarge   us-east-1   us-east-1c   3h37m   ip-10-0-164-97.ec2.internal    aws:///us-east-1c/i-02626f1dba9ed5bba   running
      clustername-8qw5l-master-3                  Provisioning   m4.xlarge   us-east-1   us-east-1a   85s     ip-10-0-133-53.ec2.internal    aws:///us-east-1a/i-015b0888fe17bc2c8   running 1
      clustername-8qw5l-worker-us-east-1a-wbtgd   Running        m4.large    us-east-1   us-east-1a   3h28m   ip-10-0-129-226.ec2.internal   aws:///us-east-1a/i-010ef6279b4662ced   running
      clustername-8qw5l-worker-us-east-1b-lrdxb   Running        m4.large    us-east-1   us-east-1b   3h28m   ip-10-0-144-248.ec2.internal   aws:///us-east-1b/i-0cb45ac45a166173b   running
      clustername-8qw5l-worker-us-east-1c-pkg26   Running        m4.large    us-east-1   us-east-1c   3h28m   ip-10-0-170-181.ec2.internal   aws:///us-east-1c/i-06861c00007751b0a   running

      1

      The new machine, clustername-8qw5l-master-3 is being created and is ready once the phase changes from Provisioning to Running.

      It might take a few minutes for the new machine to be created. The etcd cluster Operator will automatically sync when the machine or node returns to a healthy state.

5. Turn the quorum guard back on by entering the following command:

   $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'

6. You can verify that the unsupportedConfigOverrides section is removed from the object by entering this command:

   $ oc get etcd/cluster -oyaml

7. If you are using single-node OpenShift, restart the node. Otherwise, you might encounter the following error in the etcd cluster Operator:

   Example output

   EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]

Verification

1. Verify that all etcd pods are running properly.

   In a terminal that has access to the cluster as a cluster-admin user, run the following command:

   $ oc -n openshift-etcd get pods -l k8s-app=etcd

   Example output

   etcd-ip-10-0-133-53.ec2.internal                 3/3     Running     0          7m49s
   etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          123m
   etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          124m

   If the output from the previous command only lists two pods, you can manually force an etcd redeployment. In a terminal that has access to the cluster as a cluster-admin user, run the following command:

   $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 1

   1

   The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

2. Verify that there are exactly three etcd members.

   1. Connect to the running etcd container, passing in the name of a pod that was not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal

   2. View the member list:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
      |        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
      +------------------+---------+------------------------------+---------------------------+---------------------------+
      | 5eb0d6b8ca24730c | started |  ip-10-0-133-53.ec2.internal |  https://10.0.133.53:2380 |  https://10.0.133.53:2379 |
      | 757b6793e2408b6c | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
      | ca8c2990a0aa29d1 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
      +------------------+---------+------------------------------+---------------------------+---------------------------+

      If the output from the previous command lists more than three etcd members, you must carefully remove the unwanted member.

      Warning

      Be sure to remove the correct etcd member; removing a good etcd member might lead to quorum loss.

Procedure

1. Stop the crashlooping etcd pod.

   1. Debug the node that is crashlooping.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc debug node/ip-10-0-131-183.ec2.internal 1

      1

      Replace this with the name of the unhealthy node.

   2. Change your root directory to /host:

      sh-4.2# chroot /host

   3. Move the existing etcd pod file out of the kubelet manifest directory:

      sh-4.2# mkdir /var/lib/etcd-backup

      sh-4.2# mv /etc/kubernetes/manifests/etcd-pod.yaml /var/lib/etcd-backup/

   4. Move the etcd data directory to a different location:

      sh-4.2# mv /var/lib/etcd/ /tmp

      You can now exit the node shell.

2. Remove the unhealthy member.

   1. Choose a pod that is not on the affected node.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc -n openshift-etcd get pods -l k8s-app=etcd

      Example output

      etcd-ip-10-0-131-183.ec2.internal                2/3     Error       7          6h9m
      etcd-ip-10-0-164-97.ec2.internal                 3/3     Running     0          6h6m
      etcd-ip-10-0-154-204.ec2.internal                3/3     Running     0          6h6m

   2. Connect to the running etcd container, passing in the name of a pod that is not on the affected node.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc rsh -n openshift-etcd etcd-ip-10-0-154-204.ec2.internal

   3. View the member list:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
      |        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
      +------------------+---------+------------------------------+---------------------------+---------------------------+
      | 62bcf33650a7170a | started | ip-10-0-131-183.ec2.internal | https://10.0.131.183:2380 | https://10.0.131.183:2379 |
      | b78e2856655bc2eb | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
      | d022e10b498760d5 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
      +------------------+---------+------------------------------+---------------------------+---------------------------+

      Take note of the ID and the name of the unhealthy etcd member, because these values are needed later in the procedure.

   4. Remove the unhealthy etcd member by providing the ID to the etcdctl member remove command:

      sh-4.2# etcdctl member remove 62bcf33650a7170a

      Example output

      Member 62bcf33650a7170a removed from cluster ead669ce1fbfb346

   5. View the member list again and verify that the member was removed:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+------------------------------+---------------------------+---------------------------+
      |        ID        | STATUS  |             NAME             |        PEER ADDRS         |       CLIENT ADDRS        |
      +------------------+---------+------------------------------+---------------------------+---------------------------+
      | b78e2856655bc2eb | started |  ip-10-0-164-97.ec2.internal |  https://10.0.164.97:2380 |  https://10.0.164.97:2379 |
      | d022e10b498760d5 | started | ip-10-0-154-204.ec2.internal | https://10.0.154.204:2380 | https://10.0.154.204:2379 |
      +------------------+---------+------------------------------+---------------------------+---------------------------+

      You can now exit the node shell.

3. Turn off the quorum guard by entering the following command:

   $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'

   This command ensures that you can successfully re-create secrets and roll out the static pods.

4. Remove the old secrets for the unhealthy etcd member that was removed.

   1. List the secrets for the unhealthy etcd member that was removed.

      $ oc get secrets -n openshift-etcd | grep ip-10-0-131-183.ec2.internal 1

      1

      Pass in the name of the unhealthy etcd member that you took note of earlier in this procedure.

      There is a peer, serving, and metrics secret as shown in the following output:

      Example output

      etcd-peer-ip-10-0-131-183.ec2.internal              kubernetes.io/tls                     2      47m
      etcd-serving-ip-10-0-131-183.ec2.internal           kubernetes.io/tls                     2      47m
      etcd-serving-metrics-ip-10-0-131-183.ec2.internal   kubernetes.io/tls                     2      47m

   2. Delete the secrets for the unhealthy etcd member that was removed.

      1. Delete the peer secret:

         $ oc delete secret -n openshift-etcd etcd-peer-ip-10-0-131-183.ec2.internal

      2. Delete the serving secret:

         $ oc delete secret -n openshift-etcd etcd-serving-ip-10-0-131-183.ec2.internal

      3. Delete the metrics secret:

         $ oc delete secret -n openshift-etcd etcd-serving-metrics-ip-10-0-131-183.ec2.internal

5. Force etcd redeployment.

   In a terminal that has access to the cluster as a cluster-admin user, run the following command:

   $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "single-master-recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge 1

   1

   The forceRedeploymentReason value must be unique, which is why a timestamp is appended.

   When the etcd cluster Operator performs a redeployment, it ensures that all control plane nodes have a functioning etcd pod.

6. Turn the quorum guard back on by entering the following command:

   $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'

7. You can verify that the unsupportedConfigOverrides section is removed from the object by entering this command:

   $ oc get etcd/cluster -oyaml

8. If you are using single-node OpenShift, restart the node. Otherwise, you might encounter the following error in the etcd cluster Operator:

   Example output

   EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]

Procedure

1. Verify and remove the unhealthy member.

   1. Choose a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc -n openshift-etcd get pods -l k8s-app=etcd -o wide

      Example output

      etcd-openshift-control-plane-0   5/5   Running   11   3h56m   192.168.10.9   openshift-control-plane-0  <none>           <none>
      etcd-openshift-control-plane-1   5/5   Running   0    3h54m   192.168.10.10   openshift-control-plane-1   <none>           <none>
      etcd-openshift-control-plane-2   5/5   Running   0    3h58m   192.168.10.11   openshift-control-plane-2   <none>           <none>

   2. Connect to the running etcd container, passing in the name of a pod that is not on the affected node:

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc rsh -n openshift-etcd etcd-openshift-control-plane-0

   3. View the member list:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+--------------------+---------------------------+---------------------------+---------------------+
      | ID               | STATUS  | NAME                      | PEER ADDRS                  | CLIENT ADDRS                | IS LEARNER |
      +------------------+---------+--------------------+---------------------------+---------------------------+---------------------+
      | 7a8197040a5126c8 | started | openshift-control-plane-2 | https://192.168.10.11:2380/ | https://192.168.10.11:2379/ | false |
      | 8d5abe9669a39192 | started | openshift-control-plane-1 | https://192.168.10.10:2380/ | https://192.168.10.10:2379/ | false |
      | cc3830a72fc357f9 | started | openshift-control-plane-0 | https://192.168.10.9:2380/ | https://192.168.10.9:2379/   | false |
      +------------------+---------+--------------------+---------------------------+---------------------------+---------------------+

      Take note of the ID and the name of the unhealthy etcd member, because these values are required later in the procedure. The etcdctl endpoint health command will list the removed member until the replacement procedure is completed and the new member is added.

   4. Remove the unhealthy etcd member by providing the ID to the etcdctl member remove command:

      Warning

      Be sure to remove the correct etcd member; removing a good etcd member might lead to quorum loss.

      sh-4.2# etcdctl member remove 7a8197040a5126c8

      Example output

      Member 7a8197040a5126c8 removed from cluster b23536c33f2cdd1b

   5. View the member list again and verify that the member was removed:

      sh-4.2# etcdctl member list -w table

      Example output

      +------------------+---------+--------------------+---------------------------+---------------------------+-------------------------+
      | ID               | STATUS  | NAME                      | PEER ADDRS                  | CLIENT ADDRS                | IS LEARNER |
      +------------------+---------+--------------------+---------------------------+---------------------------+-------------------------+
      | 7a8197040a5126c8 | started | openshift-control-plane-2 | https://192.168.10.11:2380/ | https://192.168.10.11:2379/ | false |
      | 8d5abe9669a39192 | started | openshift-control-plane-1 | https://192.168.10.10:2380/ | https://192.168.10.10:2379/ | false |
      +------------------+---------+--------------------+---------------------------+---------------------------+-------------------------+

      You can now exit the node shell.

      Important

      After you remove the member, the cluster might be unreachable for a short time while the remaining etcd instances reboot.

2. Turn off the quorum guard by entering the following command:

   $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}'

   This command ensures that you can successfully re-create secrets and roll out the static pods.

3. Remove the old secrets for the unhealthy etcd member that was removed by running the following commands.

   1. List the secrets for the unhealthy etcd member that was removed.

      $ oc get secrets -n openshift-etcd | grep openshift-control-plane-2

      Pass in the name of the unhealthy etcd member that you took note of earlier in this procedure.

      There is a peer, serving, and metrics secret as shown in the following output:

      etcd-peer-openshift-control-plane-2             kubernetes.io/tls   2   134m
      etcd-serving-metrics-openshift-control-plane-2  kubernetes.io/tls   2   134m
      etcd-serving-openshift-control-plane-2          kubernetes.io/tls   2   134m

   2. Delete the secrets for the unhealthy etcd member that was removed.

      1. Delete the peer secret:

         $ oc delete secret etcd-peer-openshift-control-plane-2 -n openshift-etcd
         
         secret "etcd-peer-openshift-control-plane-2" deleted

      2. Delete the serving secret:

         $ oc delete secret etcd-serving-metrics-openshift-control-plane-2 -n openshift-etcd
         
         secret "etcd-serving-metrics-openshift-control-plane-2" deleted

      3. Delete the metrics secret:

         $ oc delete secret etcd-serving-openshift-control-plane-2 -n openshift-etcd
         
         secret "etcd-serving-openshift-control-plane-2" deleted

4. Delete the control plane machine.

   If you are running installer-provisioned infrastructure, or you used the Machine API to create your machines, follow these steps. Otherwise, you must create the new control plane node using the same method that was used to originally create it.

   1. Obtain the machine for the unhealthy member.

      In a terminal that has access to the cluster as a cluster-admin user, run the following command:

      $ oc get machines -n openshift-machine-api -o wide

      Example output

      NAME                              PHASE     TYPE   REGION   ZONE   AGE     NODE                               PROVIDERID                                                                                              STATE
      examplecluster-control-plane-0    Running                          3h11m   openshift-control-plane-0   baremetalhost:///openshift-machine-api/openshift-control-plane-0/da1ebe11-3ff2-41c5-b099-0aa41222964e   externally provisioned 1
      examplecluster-control-plane-1    Running                          3h11m   openshift-control-plane-1   baremetalhost:///openshift-machine-api/openshift-control-plane-1/d9f9acbc-329c-475e-8d81-03b20280a3e1   externally provisioned
      examplecluster-control-plane-2    Running                          3h11m   openshift-control-plane-2   baremetalhost:///openshift-machine-api/openshift-control-plane-2/3354bdac-61d8-410f-be5b-6a395b056135   externally provisioned
      examplecluster-compute-0          Running                          165m    openshift-compute-0         baremetalhost:///openshift-machine-api/openshift-compute-0/3d685b81-7410-4bb3-80ec-13a31858241f         provisioned
      examplecluster-compute-1          Running                          165m    openshift-compute-1         baremetalhost:///openshift-machine-api/openshift-compute-1/0fdae6eb-2066-4241-91dc-e7ea72ab13b9         provisioned

      1

      This is the control plane machine for the unhealthy node, examplecluster-control-plane-2.

   2. Save the machine configuration to a file on your file system:

      $ oc get machine examplecluster-control-plane-2 \ 1
          -n openshift-machine-api \
          -o yaml \
          > new-master-machine.yaml

      1

      Specify the name of the control plane machine for the unhealthy node.

   3. Edit the new-master-machine.yaml file that was created in the previous step to assign a new name and remove unnecessary fields.

      1. Remove the entire status section:

         status:
           addresses:
           - address: ""
             type: InternalIP
           - address: fe80::4adf:37ff:feb0:8aa1%ens1f1.373
             type: InternalDNS
           - address: fe80::4adf:37ff:feb0:8aa1%ens1f1.371
             type: Hostname
           lastUpdated: "2020-04-20T17:44:29Z"
           nodeRef:
             kind: Machine
             name: fe80::4adf:37ff:feb0:8aa1%ens1f1.372
             uid: acca4411-af0d-4387-b73e-52b2484295ad
           phase: Running
           providerStatus:
             apiVersion: machine.openshift.io/v1beta1
             conditions:
             - lastProbeTime: "2020-04-20T16:53:50Z"
               lastTransitionTime: "2020-04-20T16:53:50Z"
               message: machine successfully created
               reason: MachineCreationSucceeded
               status: "True"
               type: MachineCreation
             instanceId: i-0fdb85790d76d0c3f
             instanceState: stopped
             kind: Machine

5. Change the metadata.name field to a new name.

   It is recommended to keep the same base name as the old machine and change the ending number to the next available number. In this example, examplecluster-control-plane-2 is changed to examplecluster-control-plane-3.

   For example:

   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     ...
     name: examplecluster-control-plane-3
     ...

   1. Remove the spec.providerID field:

        providerID: baremetalhost:///openshift-machine-api/openshift-control-plane-2/3354bdac-61d8-410f-be5b-6a395b056135

   2. Remove the metadata.annotations and metadata.generation fields:

        annotations:
          machine.openshift.io/instance-state: externally provisioned
        ...
        generation: 2

   3. Remove the spec.conditions, spec.lastUpdated, spec.nodeRef and spec.phase fields:

        lastTransitionTime: "2022-08-03T08:40:36Z"
      message: 'Drain operation currently blocked by: [{Name:EtcdQuorumOperator Owner:clusteroperator/etcd}]'
      reason: HookPresent
      severity: Warning
      status: "False"
      
      type: Drainable
      lastTransitionTime: "2022-08-03T08:39:55Z"
      status: "True"
      type: InstanceExists
      
      lastTransitionTime: "2022-08-03T08:36:37Z"
      status: "True"
      type: Terminable
      lastUpdated: "2022-08-03T08:40:36Z"
      nodeRef:
      kind: Node
      name: openshift-control-plane-2
      uid: 788df282-6507-4ea2-9a43-24f237ccbc3c
      phase: Running

6. Ensure that the Bare Metal Operator is available by running the following command:

   $ oc get clusteroperator baremetal

   Example output

   NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   baremetal   4.11.3    True        False         False      3d15h

7. Remove the old BareMetalHost object by running the following command:

   $ oc delete bmh openshift-control-plane-2 -n openshift-machine-api

   Example output

   baremetalhost.metal3.io "openshift-control-plane-2" deleted

8. Delete the machine of the unhealthy member by running the following command:

   $ oc delete machine -n openshift-machine-api examplecluster-control-plane-2

   After you remove the BareMetalHost and Machine objects, then the Machine controller automatically deletes the Node object.

   If deletion of the machine is delayed for any reason or the command is obstructed and delayed, you can force deletion by removing the machine object finalizer field.

   Important

   Do not interrupt machine deletion by pressing Ctrl+c. You must allow the command to proceed to completion. Open a new terminal window to edit and delete the finalizer fields.

   1. Edit the machine configuration by running the following command:

      $ oc edit machine -n openshift-machine-api examplecluster-control-plane-2

   2. Delete the following fields in the Machine custom resource, and then save the updated file:

      finalizers:
      - machine.machine.openshift.io

      Example output

      machine.machine.openshift.io/examplecluster-control-plane-2 edited

9. Verify that the machine was deleted by running the following command:

   $ oc get machines -n openshift-machine-api -o wide

   Example output

   NAME                              PHASE     TYPE   REGION   ZONE   AGE     NODE                                 PROVIDERID                                                                                       STATE
   examplecluster-control-plane-0    Running                          3h11m   openshift-control-plane-0   baremetalhost:///openshift-machine-api/openshift-control-plane-0/da1ebe11-3ff2-41c5-b099-0aa41222964e   externally provisioned
   examplecluster-control-plane-1    Running                          3h11m   openshift-control-plane-1   baremetalhost:///openshift-machine-api/openshift-control-plane-1/d9f9acbc-329c-475e-8d81-03b20280a3e1   externally provisioned
   examplecluster-compute-0          Running                          165m    openshift-compute-0         baremetalhost:///openshift-machine-api/openshift-compute-0/3d685b81-7410-4bb3-80ec-13a31858241f         provisioned
   examplecluster-compute-1          Running                          165m    openshift-compute-1         baremetalhost:///openshift-machine-api/openshift-compute-1/0fdae6eb-2066-4241-91dc-e7ea72ab13b9         provisioned

10. Verify that the node has been deleted by running the following command:

    $ oc get nodes
    
    NAME                     STATUS ROLES   AGE   VERSION
    openshift-control-plane-0 Ready master 3h24m v1.24.0+9546431
    openshift-control-plane-1 Ready master 3h24m v1.24.0+9546431
    openshift-compute-0       Ready worker 176m v1.24.0+9546431
    openshift-compute-1       Ready worker 176m v1.24.0+9546431

11. Create the new BareMetalHost object and the secret to store the BMC credentials:

    $ cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: openshift-control-plane-2-bmc-secret
      namespace: openshift-machine-api
    data:
      password: <password>
      username: <username>
    type: Opaque
    ---
    apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      name: openshift-control-plane-2
      namespace: openshift-machine-api
    spec:
      automatedCleaningMode: disabled
      bmc:
        address: redfish://10.46.61.18:443/redfish/v1/Systems/1
        credentialsName: openshift-control-plane-2-bmc-secret
        disableCertificateVerification: true
      bootMACAddress: 48:df:37:b0:8a:a0
      bootMode: UEFI
      externallyProvisioned: false
      online: true
      rootDeviceHints:
        deviceName: /dev/sda
      userData:
        name: master-user-data-managed
        namespace: openshift-machine-api
    EOF

    Note

    The username and password can be found from the other bare metal host’s secrets. The protocol to use in bmc:address can be taken from other bmh objects.

    Important

    If you reuse the BareMetalHost object definition from an existing control plane host, do not leave the externallyProvisioned field set to true.

    Existing control plane BareMetalHost objects may have the externallyProvisioned flag set to true if they were provisioned by the OpenShift Container Platform installation program.

    After the inspection is complete, the BareMetalHost object is created and available to be provisioned.

12. Verify the creation process using available BareMetalHost objects:

    $ oc get bmh -n openshift-machine-api
    
    NAME                      STATE                  CONSUMER                      ONLINE ERROR   AGE
    openshift-control-plane-0 externally provisioned examplecluster-control-plane-0 true         4h48m
    openshift-control-plane-1 externally provisioned examplecluster-control-plane-1 true         4h48m
    openshift-control-plane-2 available              examplecluster-control-plane-3 true         47m
    openshift-compute-0       provisioned            examplecluster-compute-0       true         4h48m
    openshift-compute-1       provisioned            examplecluster-compute-1       true         4h48m

    1. Create the new control plane machine using the new-master-machine.yaml file:

       $ oc apply -f new-master-machine.yaml

    2. Verify that the new machine has been created:

       $ oc get machines -n openshift-machine-api -o wide

       Example output

       NAME                                   PHASE     TYPE   REGION   ZONE   AGE     NODE                              PROVIDERID                                                                                            STATE
       examplecluster-control-plane-0         Running                          3h11m   openshift-control-plane-0   baremetalhost:///openshift-machine-api/openshift-control-plane-0/da1ebe11-3ff2-41c5-b099-0aa41222964e   externally provisioned 1
       examplecluster-control-plane-1         Running                          3h11m   openshift-control-plane-1   baremetalhost:///openshift-machine-api/openshift-control-plane-1/d9f9acbc-329c-475e-8d81-03b20280a3e1   externally provisioned
       examplecluster-control-plane-2         Running                          3h11m   openshift-control-plane-2   baremetalhost:///openshift-machine-api/openshift-control-plane-2/3354bdac-61d8-410f-be5b-6a395b056135   externally provisioned
       examplecluster-compute-0               Running                          165m    openshift-compute-0         baremetalhost:///openshift-machine-api/openshift-compute-0/3d685b81-7410-4bb3-80ec-13a31858241f         provisioned
       examplecluster-compute-1               Running                          165m    openshift-compute-1         baremetalhost:///openshift-machine-api/openshift-compute-1/0fdae6eb-2066-4241-91dc-e7ea72ab13b9         provisioned

       1

       The new machine, clustername-8qw5l-master-3 is being created and is ready after the phase changes from Provisioning to Running.

       It should take a few minutes for the new machine to be created. The etcd cluster Operator will automatically sync when the machine or node returns to a healthy state.

    3. Verify that the bare metal host becomes provisioned and no error reported by running the following command:

       $ oc get bmh -n openshift-machine-api

       Example output

       $ oc get bmh -n openshift-machine-api
       NAME                      STATE                  CONSUMER                       ONLINE ERROR AGE
       openshift-control-plane-0 externally provisioned examplecluster-control-plane-0 true         4h48m
       openshift-control-plane-1 externally provisioned examplecluster-control-plane-1 true         4h48m
       openshift-control-plane-2 provisioned            examplecluster-control-plane-3 true          47m
       openshift-compute-0       provisioned            examplecluster-compute-0       true         4h48m
       openshift-compute-1       provisioned            examplecluster-compute-1       true         4h48m

    4. Verify that the new node is added and in a ready state by running this command:

       $ oc get nodes

       Example output

       $ oc get nodes
       NAME                     STATUS ROLES   AGE   VERSION
       openshift-control-plane-0 Ready master 4h26m v1.24.0+9546431
       openshift-control-plane-1 Ready master 4h26m v1.24.0+9546431
       openshift-control-plane-2 Ready master 12m   v1.24.0+9546431
       openshift-compute-0       Ready worker 3h58m v1.24.0+9546431
       openshift-compute-1       Ready worker 3h58m v1.24.0+9546431

13. Turn the quorum guard back on by entering the following command:

    $ oc patch etcd/cluster --type=merge -p '{"spec": {"unsupportedConfigOverrides": null}}'

14. You can verify that the unsupportedConfigOverrides section is removed from the object by entering this command:

    $ oc get etcd/cluster -oyaml

15. If you are using single-node OpenShift, restart the node. Otherwise, you might encounter the following error in the etcd cluster Operator:

    Example output

    EtcdCertSignerControllerDegraded: [Operation cannot be fulfilled on secrets "etcd-peer-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-sno-0": the object has been modified; please apply your changes to the latest version and try again, Operation cannot be fulfilled on secrets "etcd-serving-metrics-sno-0": the object has been modified; please apply your changes to the latest version and try again]