12.6. 正常でないクラスターへのプライマリーコントロールプレーンノードのインストール
この手順では、正常でない OpenShift Container Platform クラスターにプライマリーコントロールプレーンノードをインストールする方法について説明します。
前提条件
手順
クラスターの初期状態を確認します。
$ oc get nodes
出力例
NAME STATUS ROLES AGE VERSION worker-1 Ready worker 20h v1.24.0+3882f8f master-2 NotReady master 20h v1.24.0+3882f8f master-3 Ready master 20h v1.24.0+3882f8f worker-4 Ready worker 20h v1.24.0+3882f8f master-5 Ready master 15h v1.24.0+3882f8f
etcd-operatorがクラスターを正常ではないと検出していることを確認します。$ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf
出力例
E0927 08:24:23.983733 1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, worker-2 is unhealthy
etcdctlメンバーを確認します。$ oc rsh -n openshift-etcd etcd-worker-3 etcdctl member list -w table
出力例
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |2c18942f| started |worker-3|192.168.111.26|192.168.111.26| false | |61e2a860| started |worker-2|192.168.111.25|192.168.111.25| false | |ead4f280| started |worker-5|192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+
etcdctlがクラスターの正常でないメンバーを報告していることを確認します。$ etcdctl endpoint health
出力例
{"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 12.465641ms 192.168.111.26 is healthy: committed proposal: took = 12.297059ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusterMachineカスタムリソースを削除して、正常でないコントロールプレーンを削除します。$ oc delete machine -n openshift-machine-api test-day2-1-6qv96-master-2
注記正常でないクラスターを正常に実行できない場合、
MachineおよびNodeのカスタムリソース (CR) は削除されません。etcd-operatorが正常でないマシンを削除していないことを確認します。$ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf -f
出力例
I0927 08:58:41.249222 1 machinedeletionhooks.go:135] skip removing the deletion hook from machine test-day2-1-6qv96-master-2 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.26}]正常でない
etcdctlメンバーを手動で削除します。$ oc rsh -n openshift-etcd etcd-worker-3\ etcdctl member list -w table
出力例
+--------+---------+--------+--------------+--------------+---------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | LEARNER | +--------+---------+--------+--------------+--------------+---------+ |2c18942f| started |worker-3|192.168.111.26|192.168.111.26| false | |61e2a860| started |worker-2|192.168.111.25|192.168.111.25| false | |ead4f280| started |worker-5|192.168.111.28|192.168.111.28| false | +--------+---------+--------+--------------+--------------+---------+
etcdctlがクラスターの正常でないメンバーを報告していることを確認します。$ etcdctl endpoint health
出力例
{"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 13.038278ms 192.168.111.26 is healthy: committed proposal: took = 12.950355ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusteretcdctlメンバーのカスタムリソースを削除して正常でないクラスターを削除します。$ etcdctl member remove 61e2a86084aafa62
出力例
Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7
次のコマンドを実行して
etcdctlのメンバーを確認します。$ etcdctl member list -w table
出力例
+----------+---------+--------+--------------+--------------+-------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |LEARNER| +----------+---------+--------+--------------+--------------+-------+ | 2c18942f | started |worker-3|192.168.111.26|192.168.111.26| false | | ead4f280 | started |worker-5|192.168.111.28|192.168.111.28| false | +----------+---------+--------+--------------+--------------+-------+
証明書署名要求を確認して承認します。
証明書署名要求 (CSR) を確認します。
$ oc get csr | grep Pending
出力例
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:worker-6 <none> Pending
保留中の 全 CSR を承認します。
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve注記インストールを完了するには、CSR を承認する必要があります。
コントロールプレーンノードが Ready ステータスであることを確認します。
$ oc get nodes
出力例
NAME STATUS ROLES AGE VERSION worker-1 Ready worker 22h v1.24.0+3882f8f master-3 Ready master 22h v1.24.0+3882f8f worker-4 Ready worker 22h v1.24.0+3882f8f master-5 Ready master 17h v1.24.0+3882f8f master-6 Ready master 2m52s v1.24.0+3882f8f
Machine、Node、およびBareMetalHostカスタムリソースを検証します。機能する Machine API を使用してクラスターを実行する場合、
etcd-operatorにMachineCR が存在する必要があります。MachineCR は、存在する場合はRunningフェーズ中に表示されます。BareMetalHostおよびNodeにリンクされたMachineカスタムリソースを作成します。新しく追加されたノードを参照する
MachineCR があることを確認します。重要Boot-it-yourself では
BareMetalHostおよびMachineCR は作成されないため、これらを作成する必要があります。BareMetalHostおよびMachineCR の作成に失敗すると、etcd-operatorの実行時にエラーが生成されます。BareMetalHostカスタムリソースを追加します。$ oc create bmh -n openshift-machine-api custom-master3
Machineカスタムリソースを追加します。$ oc create machine -n openshift-machine-api custom-master3
link-machine-and-node.shスクリプトを実行して、BareMetalHost、Machine、およびNodeをリンクします。#!/bin/bash # Credit goes to https://bugzilla.redhat.com/show_bug.cgi?id=1801238. # This script will link Machine object and Node object. This is needed # in order to have IP address of the Node present in the status of the Machine. set -x set -e machine="$1" node="$2" if [ -z "$machine" -o -z "$node" ]; then echo "Usage: $0 MACHINE NODE" exit 1 fi uid=$(echo $node | cut -f1 -d':') node_name=$(echo $node | cut -f2 -d':') oc proxy & proxy_pid=$! function kill_proxy { kill $proxy_pid } trap kill_proxy EXIT SIGINT HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts" function wait_for_json() { local name local url local curl_opts local timeout local start_time local curr_time local time_diff name="$1" url="$2" timeout="$3" shift 3 curl_opts="$@" echo -n "Waiting for $name to respond" start_time=$(date +%s) until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do echo -n "." curr_time=$(date +%s) time_diff=$(($curr_time - $start_time)) if [[ $time_diff -gt $timeout ]]; then echo "\nTimed out waiting for $name" return 1 fi sleep 5 done echo " Success!" return 0 } wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json" addresses=$(oc get node -n openshift-machine-api ${node_name} -o json | jq -c '.status.addresses') machine_data=$(oc get machine -n openshift-machine-api -o json ${machine}) host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g') if [ -z "$host" ]; then echo "Machine $machine is not linked to a host yet." 1>&2 exit 1 fi # The address structure on the host doesn't match the node, so extract # the values we want into separate variables so we can build the patch # we need. hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g') ipaddr=$(echo "${addresses}" | jq '.[] | select(. | .type == "InternalIP") | .address' | sed 's/"//g') host_patch=' { "status": { "hardware": { "hostname": "'${hostname}'", "nics": [ { "ip": "'${ipaddr}'", "mac": "00:00:00:00:00:00", "model": "unknown", "speedGbps": 10, "vlanId": 0, "pxe": true, "name": "eth1" } ], "systemVendor": { "manufacturer": "Red Hat", "productName": "product name", "serialNumber": "" }, "firmware": { "bios": { "date": "04/01/2014", "vendor": "SeaBIOS", "version": "1.11.0-2.el7" } }, "ramMebibytes": 0, "storage": [], "cpu": { "arch": "x86_64", "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz", "clockMegahertz": 2199.998, "count": 4, "flags": [] } } } } ' echo "PATCHING HOST" echo "${host_patch}" | jq . curl -s \ -X PATCH \ ${HOST_PROXY_API_PATH}/${host}/status \ -H "Content-type: application/merge-patch+json" \ -d "${host_patch}" oc get baremetalhost -n openshift-machine-api -o yaml "${host}"$ bash link-machine-and-node.sh custom-master3 worker-3
次のコマンドを実行して
etcdctlのメンバーを確認します。$ oc rsh -n openshift-etcd etcd-worker-3 etcdctl member list -w table
出力例
+---------+-------+--------+--------------+--------------+-------+ | ID | STATUS| NAME | PEER ADDRS | CLIENT ADDRS |LEARNER| +---------+-------+--------+--------------+--------------+-------+ | 2c18942f|started|worker-3|192.168.111.26|192.168.111.26| false | | ead4f280|started|worker-5|192.168.111.28|192.168.111.28| false | | 79153c5a|started|worker-6|192.168.111.29|192.168.111.29| false | +---------+-------+--------+--------------+--------------+-------+
etcdOperator がすべてのノードを設定したことを確認します。$ oc get clusteroperator etcd
出力例
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE etcd 4.11.5 True False False 22h
etcdctlの正常性を確認します。$ oc rsh -n openshift-etcd etcd-worker-3 etcdctl endpoint health
出力例
192.168.111.26 is healthy: committed proposal: took = 9.105375ms 192.168.111.28 is healthy: committed proposal: took = 9.15205ms 192.168.111.29 is healthy: committed proposal: took = 10.277577ms
ノードの正常性を確認します。
$ oc get Nodes
出力例
NAME STATUS ROLES AGE VERSION worker-1 Ready worker 22h v1.24.0+3882f8f master-3 Ready master 22h v1.24.0+3882f8f worker-4 Ready worker 22h v1.24.0+3882f8f master-5 Ready master 18h v1.24.0+3882f8f master-6 Ready master 40m v1.24.0+3882f8f
ClusterOperatorsの正常性を確認します。$ oc get ClusterOperators
出力例
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.11.5 True False False 150m baremetal 4.11.5 True False False 22h cloud-controller-manager 4.11.5 True False False 22h cloud-credential 4.11.5 True False False 22h cluster-autoscaler 4.11.5 True False False 22h config-operator 4.11.5 True False False 22h console 4.11.5 True False False 145m csi-snapshot-controller 4.11.5 True False False 22h dns 4.11.5 True False False 22h etcd 4.11.5 True False False 22h image-registry 4.11.5 True False False 22h ingress 4.11.5 True False False 22h insights 4.11.5 True False False 22h kube-apiserver 4.11.5 True False False 22h kube-controller-manager 4.11.5 True False False 22h kube-scheduler 4.11.5 True False False 22h kube-storage-version-migrator 4.11.5 True False False 148m machine-api 4.11.5 True False False 22h machine-approver 4.11.5 True False False 22h machine-config 4.11.5 True False False 110m marketplace 4.11.5 True False False 22h monitoring 4.11.5 True False False 22h network 4.11.5 True False False 22h node-tuning 4.11.5 True False False 22h openshift-apiserver 4.11.5 True False False 163m openshift-controller-manager 4.11.5 True False False 22h openshift-samples 4.11.5 True False False 22h operator-lifecycle-manager 4.11.5 True False False 22h operator-lifecycle-manager-catalog 4.11.5 True False False 22h operator-lifecycle-manager-pkgsvr 4.11.5 True False False 22h service-ca 4.11.5 True False False 22h storage 4.11.5 True False False 22h
ClusterVersionを確認します。$ oc get ClusterVersion
出力例
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 22h Cluster version is 4.11.5