Modify vSphere configuration of OpenShift 4 cluster installed by Assisted Installer
Environment
- Red Hat Openshift Container Platform (RHOCP)
- 4
- Assisted Installer
- VMware vSphere
Issue
- If a cluster is installed using the Assisted Installer with the platform integration feature enabled, the vSphere configuration needs to be updated manually on the installed cluster. This has to happen only once the installation is fully finished and the cluster is connected to the Red Hat Hybrid Cloud Console.
Resolution
Note: This document only applies to Assisted Installer installation method. For other installation methods, refer to modifying the vSphere configuration in RHOCP4.
Steps to modify vSphere configuration in OpenShift clusters installed via Assisted Installer
Note: It is necessary to change the username, passwords, vcenter address, datacenter, datastore (
YOURVCENTERADDRESS
,YOURVCENTERDATACENTER
,YOURVCENTERDATASTORE
,YOURVCENTERUSERVNAME
,YOURVCENTERPASSWORD
,YOURVCENTERCLUSTER
)
1) Create backups
# oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml
# oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config_backup.yaml
Additional backup needed on OpenShift version 4.13 or higher:
# oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backup
2) Generate the base64-encoded passwords for vCenter user and password.
NOTE: The vSphere username must be in the format 'USERNAME@DOMAIN'
# echo -n "YOURVCENTERPASSWORD" | base64 -w0
**YOURVCENTERPASSWORD_ENCODED**
# echo -n "YOURVCENTERUSERNAME@YOURVCENTERDOMAIN" | base64 -w0
**YOURVCENTERUSERVNAME_ENCODED**
3) Edit creds.yaml
# cp creds_backup.yaml vsphere-creds.yaml
# vi vsphere-creds.yaml
apiVersion: v1
data:
YOURVCENTERADDRESS.password: YOURVCENTERPASSWORD_ENCODED**
YOURVCENTERADDRESS.username: YOURVCENTERUSERVNAME_ENCODED**
kind: Secret
metadata:
annotations:
cloudcredential.openshift.io/mode: passthrough
creationTimestamp: "2022-01-25T17:39:50Z"
name: vsphere-creds
namespace: kube-system
resourceVersion: "2437"
uid: 06971978-e3a5-4741-87f9-2ca3602f2658
type: Opaque
# oc replace -f vsphere-creds.yaml
secret/vsphere-creds replaced
or
# oc edit -f vsphere-creds.yaml --namespace=kube-system
secret/vsphere-creds replaced
4) Force a redeployment of the kube-controller-manager pods
# oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
kubecontrollermanager.operator.openshift.io/cluster patched
5) Edit cloud-provider-config.yaml
# cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
# vi cloud-provider-config.yaml
apiVersion: v1
data:
config: |
[Global]
secret-name = "vsphere-creds"
secret-namespace = "kube-system"
insecure-flag = "1"
[Workspace]
server = "YOURVCENTERADDRESS"
datacenter = "YOURVCENTERDATACENTER"
default-datastore = "YOURVCENTERDATASTORE"
folder = "/YOURVCENTERDATACENTER/path/to/folder"
[VirtualCenter "YOURVCENTERADDRESS"]
datacenters = "YOURVCENTERDATACENTER"
kind: ConfigMap
metadata:
creationTimestamp: "2022-01-25T17:40:49Z"
name: cloud-provider-config
namespace: openshift-config
resourceVersion: "2070"
uid: 80bb8618-bf25-442b-b023-b31311918507
# oc apply -f cloud-provider-config.yaml
Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
configmap/cloud-provider-config configured
Steps required for OpenShift version 4.13 or higher
6) Taint the nodes with the uninitialized taint:
For each node, replace
# oc get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready control-plane,master 45h v1.26.3+379cd9f
master-1 Ready control-plane,master 45h v1.26.3+379cd9f
worker-0 Ready worker 45h v1.26.3+379cd9f
worker-1 Ready worker 45h v1.26.3+379cd9f
master-2 Ready control-plane,master 45h v1.26.3+379cd9f
# oc adm taint node master-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node master-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node master-3 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node worker-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node worker-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
7) Update the infrastructures.config
# cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml
# vi infrastructures.config.openshift.io.yaml
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
creationTimestamp: "2023-05-07T10:19:55Z"
generation: 1
name: cluster
resourceVersion: "536"
uid: e8a5742c-6d15-44e6-8a9e-064b26ab347d
spec:
cloudConfig:
key: config
name: cloud-provider-config
platformSpec:
type: VSphere
vsphere:
failureDomains:
- name: assisted-generated-failure-domain
region: assisted-generated-region
server: YOURVCENTERADDRESS
topology:
computeCluster: /YOURVCENTERDATACENTER/host/YOURVCENTERCLUSTER
datacenter: YOURVCENTERDATACENTER
datastore: /YOURVCENTERDATACENTER/datastore/YOURVCENTERDATASTORE
folder: "/YOURVCENTERDATACENTER/path/to/folder"
networks:
- "VM Network"
resourcePool: /YOURVCENTERDATACENTER/host/YOURVCENTERCLUSTER/Resources
zone: assisted-generated-zone
nodeNetworking:
external: {}
internal: {}
vcenters:
- datacenters:
- YOURVCENTERDATACENTER
server: YOURVCENTERADDRESS
kind: List
metadata:
resourceVersion: ""
# oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true
Warning: resource infrastructures/cluster is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
infrastructure.config.openshift.io/cluster configured
Diagnostic Steps
- Steps to verify vSphere in Openshift
1) StorageClass vsphere-ds
# vi vsphere-sc.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: vsphere-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
datastore: YOURVCENTERDATASTORE
diskformat: thin
reclaimPolicy: Delete
volumeBindingMode: Immediate
# oc create -f vsphere-sc.yaml
storageclass.storage.k8s.io/vsphere-sc created
2) Create PersistentVolumeClaim and PersistentVolume to test connection
# vi test-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-pvc
namespace: openshift-config
annotations:
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: vsphere-sc
volumeMode: Filesystem
# oc create -f test-pvc.yaml
persistentvolumeclaim/my-test-pvc created
# oc get pvc -n openshift-config
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-pvc Bound pvc-e91314a3-f225-46b9-9fdb-c2adcea934ad 10Gi RWO vsphere-sc 2m4s
# oc get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e91314a3-f225-46b9-9fdb-c2adcea934ad 10Gi RWO Delete Bound openshift-config/test-pvc vsphere-sc 10m
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments