17.10. 使用 Topology Aware Lifecycle Manager 更新受管集群

流程

1. 将 ClusterGroupUpgrade CR 的内容保存到 cgu-a.yaml、cgu-b.yaml 和 cgu-c.yaml 文件中。

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-a
     namespace: default
   spec:
     blockingCRs: 1
     - name: cgu-c
       namespace: default
     clusters:
     - spoke1
     - spoke2
     - spoke3
     enable: false
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     remediationStrategy:
       canaries:
       - spoke1
       maxConcurrency: 2
       timeout: 240
   status:
     conditions:
     - message: The ClusterGroupUpgrade CR is not enabled
       reason: UpgradeNotStarted
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy2-common-pao-sub-policy
       namespace: default
     - name: policy3-common-ptp-sub-policy
       namespace: default
     placementBindings:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     placementRules:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     remediationPlan:
     - - spoke1
     - - spoke2

   1

   定义阻塞 CR。cgu-a 更新无法启动，直到 cgu-c 完成后。

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-b
     namespace: default
   spec:
     blockingCRs: 1
     - name: cgu-a
       namespace: default
     clusters:
     - spoke4
     - spoke5
     enable: false
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     - policy4-common-sriov-sub-policy
     remediationStrategy:
       maxConcurrency: 1
       timeout: 240
   status:
     conditions:
     - message: The ClusterGroupUpgrade CR is not enabled
       reason: UpgradeNotStarted
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy2-common-pao-sub-policy
       namespace: default
     - name: policy3-common-ptp-sub-policy
       namespace: default
     - name: policy4-common-sriov-sub-policy
       namespace: default
     placementBindings:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     placementRules:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     remediationPlan:
     - - spoke4
     - - spoke5
     status: {}

   1

   cgu-b 更新无法启动，直到 cgu-a 完成后。

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-c
     namespace: default
   spec: 1
     clusters:
     - spoke6
     enable: false
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     - policy4-common-sriov-sub-policy
     remediationStrategy:
       maxConcurrency: 1
       timeout: 240
   status:
     conditions:
     - message: The ClusterGroupUpgrade CR is not enabled
       reason: UpgradeNotStarted
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     managedPoliciesCompliantBeforeUpgrade:
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy4-common-sriov-sub-policy
       namespace: default
     placementBindings:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     placementRules:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     remediationPlan:
     - - spoke6
     status: {}

   1

   cgu-c 更新没有任何阻塞 CR。当 enable 字段设为 true 时，TALM 会启动 cgu-c 更新。

2. 通过为每个相关 CR 运行以下命令创建 ClusterGroupUpgrade CR：

   $ oc apply -f <name>.yaml

3. 通过为每个相关 CR 运行以下命令启动更新过程：

   $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/<name> \
   --type merge -p '{"spec":{"enable":true}}'

   以下示例显示 enable 字段设为 true 的 ClusterGroupUpgrade CR：

   带有阻塞 CR 的 cgu-a 示例

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-a
     namespace: default
   spec:
     blockingCRs:
     - name: cgu-c
       namespace: default
     clusters:
     - spoke1
     - spoke2
     - spoke3
     enable: true
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     remediationStrategy:
       canaries:
       - spoke1
       maxConcurrency: 2
       timeout: 240
   status:
     conditions:
     - message: 'The ClusterGroupUpgrade CR is blocked by other CRs that have not yet
         completed: [cgu-c]' 1
       reason: UpgradeCannotStart
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy2-common-pao-sub-policy
       namespace: default
     - name: policy3-common-ptp-sub-policy
       namespace: default
     placementBindings:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     placementRules:
     - cgu-a-policy1-common-cluster-version-policy
     - cgu-a-policy2-common-pao-sub-policy
     - cgu-a-policy3-common-ptp-sub-policy
     remediationPlan:
     - - spoke1
     - - spoke2
     status: {}

   1

   显示阻塞 CR 的列表。

   带有阻塞 CR 的 cgu-b 示例

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-b
     namespace: default
   spec:
     blockingCRs:
     - name: cgu-a
       namespace: default
     clusters:
     - spoke4
     - spoke5
     enable: true
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     - policy4-common-sriov-sub-policy
     remediationStrategy:
       maxConcurrency: 1
       timeout: 240
   status:
     conditions:
     - message: 'The ClusterGroupUpgrade CR is blocked by other CRs that have not yet
         completed: [cgu-a]' 1
       reason: UpgradeCannotStart
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy2-common-pao-sub-policy
       namespace: default
     - name: policy3-common-ptp-sub-policy
       namespace: default
     - name: policy4-common-sriov-sub-policy
       namespace: default
     placementBindings:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     placementRules:
     - cgu-b-policy1-common-cluster-version-policy
     - cgu-b-policy2-common-pao-sub-policy
     - cgu-b-policy3-common-ptp-sub-policy
     - cgu-b-policy4-common-sriov-sub-policy
     remediationPlan:
     - - spoke4
     - - spoke5
     status: {}

   1

   显示阻塞 CR 的列表。

   带有阻塞 CR 的 cgu-c 示例

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-c
     namespace: default
   spec:
     clusters:
     - spoke6
     enable: true
     managedPolicies:
     - policy1-common-cluster-version-policy
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     - policy4-common-sriov-sub-policy
     remediationStrategy:
       maxConcurrency: 1
       timeout: 240
   status:
     conditions:
     - message: The ClusterGroupUpgrade CR has upgrade policies that are still non compliant 1
       reason: UpgradeNotCompleted
       status: "False"
       type: Ready
     copiedPolicies:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     managedPoliciesCompliantBeforeUpgrade:
     - policy2-common-pao-sub-policy
     - policy3-common-ptp-sub-policy
     managedPoliciesForUpgrade:
     - name: policy1-common-cluster-version-policy
       namespace: default
     - name: policy4-common-sriov-sub-policy
       namespace: default
     placementBindings:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     placementRules:
     - cgu-c-policy1-common-cluster-version-policy
     - cgu-c-policy4-common-sriov-sub-policy
     remediationPlan:
     - - spoke6
     status:
       currentBatch: 1
       remediationPlanForBatch:
         spoke6: 0

   1

   cgu-c 更新没有任何阻塞 CR。

步骤

1. 将 status.state.AtLatestKnown 字段添加到 Operator 的 Subscription CR 中：

   Subscription CR 示例

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: cluster-logging
     namespace: openshift-logging
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: "stable"
     name: cluster-logging
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown 1

   1

   status.state: AtLatestKnown 字段用于 Operator 目录中可用的最新 Operator 版本。

   注意

   当 registry 中有新版本的 Operator 时，相关的策略将变为不合规。

2. 使用 ClusterGroupUpgrade CR 将更改的 Subscription 策略应用到受管集群。

流程

1. 将 ClusterGroupUpgrade CR 的内容保存到 cgu-1.yaml 文件中。

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: cgu-1
     namespace: default
   spec:
     managedPolicies: 1
       - policy1-common-cluster-version-policy
       - policy2-common-nto-sub-policy
       - policy3-common-ptp-sub-policy
       - policy4-common-sriov-sub-policy
     enable: false
     clusters: 2
     - spoke1
     - spoke2
     - spoke5
     - spoke6
     remediationStrategy:
       maxConcurrency: 2 3
       timeout: 240 4
     batchTimeoutAction: 5

   1

   要应用的策略的名称。

   2

   要更新的集群列表。

   3

   maxConcurrency 字段表示同时更新的集群数量。

   4

   更新超时（以分钟为单位）。

   5

   控制批处理超时时会发生什么。可能的值有 abort 或 continue。如果未指定，则默认为 continue。

2. 运行以下命令来创建 ClusterGroupUpgrade CR：

   $ oc create -f cgu-1.yaml

   1. 运行以下命令，检查 hub 集群中是否已创建 ClusterGroupUpgrade CR：

      $ oc get cgu --all-namespaces

      输出示例

      NAMESPACE   NAME  AGE  STATE      DETAILS
      default     cgu-1 8m55 NotEnabled Not Enabled

   2. 运行以下命令检查更新的状态：

      $ oc get cgu -n default cgu-1 -ojsonpath='{.status}' | jq

      输出示例

      {
        "computedMaxConcurrency": 2,
        "conditions": [
          {
            "lastTransitionTime": "2022-02-25T15:34:07Z",
            "message": "Not enabled", 1
            "reason": "NotEnabled",
            "status": "False",
            "type": "Progressing"
          }
        ],
        "copiedPolicies": [
          "cgu-policy1-common-cluster-version-policy",
          "cgu-policy2-common-nto-sub-policy",
          "cgu-policy3-common-ptp-sub-policy",
          "cgu-policy4-common-sriov-sub-policy"
        ],
        "managedPoliciesContent": {
          "policy1-common-cluster-version-policy": "null",
          "policy2-common-nto-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"node-tuning-operator\",\"namespace\":\"openshift-cluster-node-tuning-operator\"}]",
          "policy3-common-ptp-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"ptp-operator-subscription\",\"namespace\":\"openshift-ptp\"}]",
          "policy4-common-sriov-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"sriov-network-operator-subscription\",\"namespace\":\"openshift-sriov-network-operator\"}]"
        },
        "managedPoliciesForUpgrade": [
          {
            "name": "policy1-common-cluster-version-policy",
            "namespace": "default"
          },
          {
            "name": "policy2-common-nto-sub-policy",
            "namespace": "default"
          },
          {
            "name": "policy3-common-ptp-sub-policy",
            "namespace": "default"
          },
          {
            "name": "policy4-common-sriov-sub-policy",
            "namespace": "default"
          }
        ],
        "managedPoliciesNs": {
          "policy1-common-cluster-version-policy": "default",
          "policy2-common-nto-sub-policy": "default",
          "policy3-common-ptp-sub-policy": "default",
          "policy4-common-sriov-sub-policy": "default"
        },
        "placementBindings": [
          "cgu-policy1-common-cluster-version-policy",
          "cgu-policy2-common-nto-sub-policy",
          "cgu-policy3-common-ptp-sub-policy",
          "cgu-policy4-common-sriov-sub-policy"
        ],
        "placementRules": [
          "cgu-policy1-common-cluster-version-policy",
          "cgu-policy2-common-nto-sub-policy",
          "cgu-policy3-common-ptp-sub-policy",
          "cgu-policy4-common-sriov-sub-policy"
        ],
        "precaching": {
          "spec": {}
        },
        "remediationPlan": [
          [
            "spoke1",
            "spoke2"
          ],
          [
            "spoke5",
            "spoke6"
          ]
        ],
        "status": {}
      }

      1

      ClusterGroupUpgrade CR 中的 spec.enable 字段设置为 false。

   3. 运行以下命令，检查策略的状态：

      $ oc get policies -A

      输出示例

      NAMESPACE   NAME                                                 REMEDIATION ACTION   COMPLIANCE STATE   AGE
      default     cgu-policy1-common-cluster-version-policy            enforce                                 17m 1
      default     cgu-policy2-common-nto-sub-policy                    enforce                                 17m
      default     cgu-policy3-common-ptp-sub-policy                    enforce                                 17m
      default     cgu-policy4-common-sriov-sub-policy                  enforce                                 17m
      default     policy1-common-cluster-version-policy                inform               NonCompliant       15h
      default     policy2-common-nto-sub-policy                        inform               NonCompliant       15h
      default     policy3-common-ptp-sub-policy                        inform               NonCompliant       18m
      default     policy4-common-sriov-sub-policy                      inform               NonCompliant       18m

      1

      目前在集群中应用的策略的 spec.remediationAction 字段被设置为 enforce。在更新过程中，来自 ClusterGroupUpgrade CR 的 inform 模式的受管策略会处于 inform 模式。

3. 运行以下命令，将 spec.enable 字段的值更改为 true ：

   $ oc --namespace=default patch clustergroupupgrade.ran.openshift.io/cgu-1 \
   --patch '{"spec":{"enable":true}}' --type=merge

验证

1. 运行以下命令，再次检查更新的状态：

   $ oc get cgu -n default cgu-1 -ojsonpath='{.status}' | jq

   输出示例

   {
     "computedMaxConcurrency": 2,
     "conditions": [ 1
       {
         "lastTransitionTime": "2022-02-25T15:33:07Z",
         "message": "All selected clusters are valid",
         "reason": "ClusterSelectionCompleted",
         "status": "True",
         "type": "ClustersSelected",
         "lastTransitionTime": "2022-02-25T15:33:07Z",
         "message": "Completed validation",
         "reason": "ValidationCompleted",
         "status": "True",
         "type": "Validated",
         "lastTransitionTime": "2022-02-25T15:34:07Z",
         "message": "Remediating non-compliant policies",
         "reason": "InProgress",
         "status": "True",
         "type": "Progressing"
       }
     ],
     "copiedPolicies": [
       "cgu-policy1-common-cluster-version-policy",
       "cgu-policy2-common-nto-sub-policy",
       "cgu-policy3-common-ptp-sub-policy",
       "cgu-policy4-common-sriov-sub-policy"
     ],
     "managedPoliciesContent": {
       "policy1-common-cluster-version-policy": "null",
       "policy2-common-nto-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"node-tuning-operator\",\"namespace\":\"openshift-cluster-node-tuning-operator\"}]",
       "policy3-common-ptp-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"ptp-operator-subscription\",\"namespace\":\"openshift-ptp\"}]",
       "policy4-common-sriov-sub-policy": "[{\"kind\":\"Subscription\",\"name\":\"sriov-network-operator-subscription\",\"namespace\":\"openshift-sriov-network-operator\"}]"
     },
     "managedPoliciesForUpgrade": [
       {
         "name": "policy1-common-cluster-version-policy",
         "namespace": "default"
       },
       {
         "name": "policy2-common-nto-sub-policy",
         "namespace": "default"
       },
       {
         "name": "policy3-common-ptp-sub-policy",
         "namespace": "default"
       },
       {
         "name": "policy4-common-sriov-sub-policy",
         "namespace": "default"
       }
     ],
     "managedPoliciesNs": {
       "policy1-common-cluster-version-policy": "default",
       "policy2-common-nto-sub-policy": "default",
       "policy3-common-ptp-sub-policy": "default",
       "policy4-common-sriov-sub-policy": "default"
     },
     "placementBindings": [
       "cgu-policy1-common-cluster-version-policy",
       "cgu-policy2-common-nto-sub-policy",
       "cgu-policy3-common-ptp-sub-policy",
       "cgu-policy4-common-sriov-sub-policy"
     ],
     "placementRules": [
       "cgu-policy1-common-cluster-version-policy",
       "cgu-policy2-common-nto-sub-policy",
       "cgu-policy3-common-ptp-sub-policy",
       "cgu-policy4-common-sriov-sub-policy"
     ],
     "precaching": {
       "spec": {}
     },
     "remediationPlan": [
       [
         "spoke1",
         "spoke2"
       ],
       [
         "spoke5",
         "spoke6"
       ]
     ],
     "status": {
       "currentBatch": 1,
       "currentBatchStartedAt": "2022-02-25T15:54:16Z",
       "remediationPlanForBatch": {
         "spoke1": 0,
         "spoke2": 1
       },
       "startedAt": "2022-02-25T15:54:16Z"
     }
   }

   1

   反映当前批处理的更新进度。再次运行该命令以接收有关进度的更新信息。

2. 如果策略包含 Operator 订阅，您可以在单节点集群中直接检查安装进度。

   1. 运行以下命令，导出用于检查安装的单节点集群的 KUBECONFIG 文件：

      $ export KUBECONFIG=<cluster_kubeconfig_absolute_path>

   2. 运行以下命令，检查单节点集群中存在的所有订阅，并在您要通过 ClusterGroupUpgrade CR 安装的策略中查找您要通过 ClusterGroupUpgrade CR 安装的订阅：

      $ oc get subs -A | grep -i <subscription_name>

      cluster-logging 策略的输出示例

      NAMESPACE                              NAME                         PACKAGE                      SOURCE             CHANNEL
      openshift-logging                      cluster-logging              cluster-logging              redhat-operators   stable

3. 如果其中一个受管策略包含 ClusterVersion CR，则根据 spoke 集群运行以下命令来检查当前批处理中的平台更新状态：

   $ oc get clusterversion

   输出示例

   NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.4.12.5     True        True          43s     Working towards 4.4.12.7: 71 of 735 done (9% complete)

4. 运行以下命令检查 Operator 订阅：

   $ oc get subs -n <operator-namespace> <operator-subscription> -ojsonpath="{.status}"

5. 运行以下命令，检查与所需订阅关联的单节点集群中是否存在安装计划：

   $ oc get installplan -n <subscription_namespace>

   cluster-logging Operator 的输出示例

   NAMESPACE                              NAME            CSV                                 APPROVAL   APPROVED
   openshift-logging                      install-6khtw   cluster-logging.5.3.3-4             Manual     true 1

   1

   安装计划在 TALM 批准安装计划后将其 Approval 字段设置为 Manual，其 Approved 字段会从 false 改为 true。

   注意

   当 TALM 修复包含订阅的策略时，它会自动批准附加到该订阅的任何安装计划。如果需要多个安装计划将 Operator 升级到最新的已知版本，TALM 可能会批准多个安装计划，通过一个或多个中间版本进行升级以进入最终版本。

6. 运行以下命令，检查正在安装 ClusterGroupUpgrade 的策略的 Operator 的集群服务版本是否已进入 Succeeded 阶段：

   $ oc get csv -n <operator_namespace>

   OpenShift Logging Operator 的输出示例

   NAME                    DISPLAY                     VERSION   REPLACES   PHASE
   cluster-logging.5.4.2   Red Hat OpenShift Logging   5.4.2                Succeeded

流程

1. 在 clustergroupupgrades-group-du.yaml 文件中保存 ClusterGroupUpgrade CR 的内容，并 backup 和 enable 字段设置为 true：

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: du-upgrade-4918
     namespace: ztp-group-du-sno
   spec:
     preCaching: true
     backup: true
     clusters:
     - cnfdb1
     - cnfdb2
     enable: true
     managedPolicies:
     - du-upgrade-platform-upgrade
     remediationStrategy:
       maxConcurrency: 2
       timeout: 240

2. 要启动更新，请运行以下命令来应用 ClusterGroupUpgrade CR：

   $ oc apply -f clustergroupupgrades-group-du.yaml

流程

1. 运行以下命令来删除之前创建的 ClusterGroupUpgrade 自定义资源 (CR)：

   $ oc delete cgu/du-upgrade-4918 -n ztp-group-du-sno

2. 登录到要恢复的集群。

3. 运行以下命令，检查平台操作系统部署的状态：

   $ ostree admin status

   输出示例

   [root@lab-test-spoke2-node-0 core]# ostree admin status
   * rhcos c038a8f08458bbed83a77ece033ad3c55597e3f64edad66ea12fda18cbdceaf9.0
       Version: 49.84.202202230006-0
       Pinned: yes 1
       origin refspec: c038a8f08458bbed83a77ece033ad3c55597e3f64edad66ea12fda18cbdceaf9

   1

   当前部署已被固定。不需要平台操作系统部署回滚。

   [root@lab-test-spoke2-node-0 core]# ostree admin status
   * rhcos f750ff26f2d5550930ccbe17af61af47daafc8018cd9944f2a3a6269af26b0fa.0
       Version: 410.84.202204050541-0
       origin refspec: f750ff26f2d5550930ccbe17af61af47daafc8018cd9944f2a3a6269af26b0fa
   rhcos ad8f159f9dc4ea7e773fd9604c9a16be0fe9b266ae800ac8470f63abc39b52ca.0 (rollback) 1
       Version: 410.84.202203290245-0
       Pinned: yes 2
       origin refspec: ad8f159f9dc4ea7e773fd9604c9a16be0fe9b266ae800ac8470f63abc39b52ca

   1

   此平台操作系统部署标记为回滚。

   2

   以前的部署已被固定，可以回滚。

4. 要触发平台操作系统部署的回滚，请运行以下命令：

   $ rpm-ostree rollback -r

5. 恢复的第一阶段会关闭容器，并将文件从备份分区恢复到目标目录。要开始恢复，请运行以下命令：

   $ /var/recovery/upgrade-recovery.sh

6. 提示时，运行以下命令重启集群：

   $ systemctl reboot

7. 重新引导后，运行以下命令重启恢复：

   $ /var/recovery/upgrade-recovery.sh  --resume

流程

1. 在 clustergroupupgrades-group-du.yaml 文件中将 preCaching 字段设置为 true 来保存 ClusterGroupUpgrade CR 的内容：

   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: du-upgrade-4918
     namespace: ztp-group-du-sno
   spec:
     preCaching: true 1
     clusters:
     - cnfdb1
     - cnfdb2
     enable: false
     managedPolicies:
     - du-upgrade-platform-upgrade
     remediationStrategy:
       maxConcurrency: 2
       timeout: 240

   1

   preCaching 字段设为 true，它允许 TALM 在开始更新前拉取容器镜像。

2. 当您要启动预缓存时，请运行以下命令应用 ClusterGroupUpgrade CR：

   $ oc apply -f clustergroupupgrades-group-du.yaml

验证

1. 运行以下命令，检查 hub 集群中是否存在 ClusterGroupUpgrade CR：

   $ oc get cgu -A

   输出示例

   NAMESPACE          NAME              AGE   STATE        DETAILS
   ztp-group-du-sno   du-upgrade-4918   10s   InProgress   Precaching is required and not done 1

   1

   CR 被创建。

2. 运行以下命令，检查预缓存任务的状态：

   $ oc get cgu -n ztp-group-du-sno du-upgrade-4918 -o jsonpath='{.status}'

   输出示例

   {
     "conditions": [
       {
         "lastTransitionTime": "2022-01-27T19:07:24Z",
         "message": "Precaching is required and not done",
         "reason": "InProgress",
         "status": "False",
         "type": "PrecachingSucceeded"
       },
       {
         "lastTransitionTime": "2022-01-27T19:07:34Z",
         "message": "Pre-caching spec is valid and consistent",
         "reason": "PrecacheSpecIsWellFormed",
         "status": "True",
         "type": "PrecacheSpecValid"
       }
     ],
     "precaching": {
       "clusters": [
         "cnfdb1" 1
         "cnfdb2"
       ],
       "spec": {
         "platformImage": "image.example.io"},
       "status": {
         "cnfdb1": "Active"
         "cnfdb2": "Succeeded"}
       }
   }

   1

   显示已识别的集群列表。

3. 在 spoke 集群中运行以下命令来检查预缓存作业的状态：

   $ oc get jobs,pods -n openshift-talo-pre-cache

   输出示例

   NAME                  COMPLETIONS   DURATION   AGE
   job.batch/pre-cache   0/1           3m10s      3m10s
   
   NAME                     READY   STATUS    RESTARTS   AGE
   pod/pre-cache--1-9bmlr   1/1     Running   0          3m10s

4. 运行以下命令，检查 ClusterGroupUpgrade CR 的状态：

   $ oc get cgu -n ztp-group-du-sno du-upgrade-4918 -o jsonpath='{.status}'

   输出示例

   "conditions": [
       {
         "lastTransitionTime": "2022-01-27T19:30:41Z",
         "message": "The ClusterGroupUpgrade CR has all clusters compliant with all the managed policies",
         "reason": "UpgradeCompleted",
         "status": "True",
         "type": "Ready"
       },
       {
         "lastTransitionTime": "2022-01-27T19:28:57Z",
         "message": "Precaching is completed",
         "reason": "PrecachingCompleted",
         "status": "True",
         "type": "PrecachingSucceeded" 1
       }

   1

   预缓存任务已完成。

1. 创建 ClusterGroupUpgrade CR，并将 spec.enable 字段设置为 false。
2. 等待状态更新，再完成故障排除问题。
3. 如果所有内容都如预期，在 ClusterGroupUpgrade CR 中将 spec.enable 字段设置为 true。

1. 在 spoke 集群中运行以下命令来检查预缓存的状态：

   $ oc get jobs,pods -n openshift-talo-pre-cache