Modify vSphere configuration of OpenShift 4 cluster installed by Assisted Installer

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (RHOCP)
    • 4
  • Assisted Installer
  • VMware vSphere

Issue

  • If a cluster is installed using the Assisted Installer with the platform integration feature enabled, the vSphere configuration needs to be updated manually on the installed cluster. This has to happen only once the installation is fully finished and the cluster is connected to the Red Hat Hybrid Cloud Console.

Resolution

Note: This document only applies to Assisted Installer installation method. For other installation methods, refer to modifying the vSphere configuration in RHOCP4.

Steps to modify vSphere configuration in OpenShift clusters installed via Assisted Installer

Note: It is necessary to change the username, passwords, vcenter address, datacenter, datastore (YOURVCENTERADDRESS, YOURVCENTERDATACENTER, YOURVCENTERDATASTORE, YOURVCENTERUSERVNAME, YOURVCENTERPASSWORD, YOURVCENTERCLUSTER)

1) Create backups

# oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml
# oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config_backup.yaml

Additional backup needed on OpenShift version 4.13 or higher:

# oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backup

2) Generate the base64-encoded passwords for vCenter user and password.
NOTE: The vSphere username must be in the format 'USERNAME@DOMAIN'

# echo -n "YOURVCENTERPASSWORD" | base64 -w0
**YOURVCENTERPASSWORD_ENCODED**

# echo -n "YOURVCENTERUSERNAME@YOURVCENTERDOMAIN" | base64 -w0
**YOURVCENTERUSERVNAME_ENCODED**

3) Edit creds.yaml

# cp creds_backup.yaml vsphere-creds.yaml
# vi vsphere-creds.yaml

apiVersion: v1
data:
  YOURVCENTERADDRESS.password: YOURVCENTERPASSWORD_ENCODED**
  YOURVCENTERADDRESS.username: YOURVCENTERUSERVNAME_ENCODED**
kind: Secret
metadata:
  annotations:
    cloudcredential.openshift.io/mode: passthrough
  creationTimestamp: "2022-01-25T17:39:50Z"
  name: vsphere-creds
  namespace: kube-system
  resourceVersion: "2437"
  uid: 06971978-e3a5-4741-87f9-2ca3602f2658
type: Opaque

# oc replace -f vsphere-creds.yaml
secret/vsphere-creds replaced

or 
# oc edit -f vsphere-creds.yaml --namespace=kube-system
secret/vsphere-creds replaced

4) Force a redeployment of the kube-controller-manager pods

# oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
kubecontrollermanager.operator.openshift.io/cluster patched

5) Edit cloud-provider-config.yaml

# cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
# vi cloud-provider-config.yaml

apiVersion: v1
data:
  config: |
    [Global]
    secret-name = "vsphere-creds"
    secret-namespace = "kube-system"
    insecure-flag = "1"

    [Workspace]
    server = "YOURVCENTERADDRESS"
    datacenter = "YOURVCENTERDATACENTER"
    default-datastore = "YOURVCENTERDATASTORE"
    folder = "/YOURVCENTERDATACENTER/path/to/folder"

    [VirtualCenter "YOURVCENTERADDRESS"]
    datacenters = "YOURVCENTERDATACENTER"
kind: ConfigMap
metadata:
  creationTimestamp: "2022-01-25T17:40:49Z"
  name: cloud-provider-config
  namespace: openshift-config
  resourceVersion: "2070"
  uid: 80bb8618-bf25-442b-b023-b31311918507

# oc apply -f cloud-provider-config.yaml
Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
configmap/cloud-provider-config configured

Steps required for OpenShift version 4.13 or higher

6) Taint the nodes with the uninitialized taint:

For each node, replace and execute the following (once for each node):

#  oc get nodes
NAME                STATUS   ROLES                  AGE   VERSION
master-0   Ready    control-plane,master   45h   v1.26.3+379cd9f
master-1   Ready    control-plane,master   45h   v1.26.3+379cd9f
worker-0   Ready    worker                 45h   v1.26.3+379cd9f
worker-1   Ready    worker                 45h   v1.26.3+379cd9f
master-2   Ready    control-plane,master   45h   v1.26.3+379cd9f

# oc adm taint node master-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node master-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node master-3 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node worker-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
# oc adm taint node worker-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

7) Update the infrastructures.config

# cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml
# vi infrastructures.config.openshift.io.yaml

apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Infrastructure
  metadata:
    creationTimestamp: "2023-05-07T10:19:55Z"
    generation: 1
    name: cluster
    resourceVersion: "536"
    uid: e8a5742c-6d15-44e6-8a9e-064b26ab347d
  spec:
    cloudConfig:
      key: config
      name: cloud-provider-config
    platformSpec:
      type: VSphere
      vsphere:
        failureDomains:
        - name: assisted-generated-failure-domain
          region: assisted-generated-region
          server: YOURVCENTERADDRESS
          topology:
            computeCluster: /YOURVCENTERDATACENTER/host/YOURVCENTERCLUSTER
            datacenter: YOURVCENTERDATACENTER
            datastore: /YOURVCENTERDATACENTER/datastore/YOURVCENTERDATASTORE
            folder: "/YOURVCENTERDATACENTER/path/to/folder"
            networks:
            - "VM Network"
            resourcePool: /YOURVCENTERDATACENTER/host/YOURVCENTERCLUSTER/Resources
          zone: assisted-generated-zone
        nodeNetworking:
          external: {}
          internal: {}
        vcenters:
        - datacenters:
          - YOURVCENTERDATACENTER
          server: YOURVCENTERADDRESS

kind: List
metadata:
  resourceVersion: ""

# oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true
Warning: resource infrastructures/cluster is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
infrastructure.config.openshift.io/cluster configured

Diagnostic Steps

  • Steps to verify vSphere in Openshift

1) StorageClass vsphere-ds

# vi vsphere-sc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: vsphere-sc
provisioner: kubernetes.io/vsphere-volume
parameters:
 datastore: YOURVCENTERDATASTORE
 diskformat: thin
reclaimPolicy: Delete
volumeBindingMode: Immediate

# oc create -f vsphere-sc.yaml
storageclass.storage.k8s.io/vsphere-sc created

2) Create PersistentVolumeClaim and PersistentVolume to test connection

# vi test-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: test-pvc
 namespace: openshift-config
 annotations:
   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
 finalizers:
   - kubernetes.io/pvc-protection
spec:
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
    storage: 10Gi
 storageClassName: vsphere-sc
 volumeMode: Filesystem


# oc create -f test-pvc.yaml
persistentvolumeclaim/my-test-pvc created

# oc get pvc -n openshift-config
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-pvc   Bound    pvc-e91314a3-f225-46b9-9fdb-c2adcea934ad   10Gi     RWO         vsphere-sc  2m4s

# oc get pv
NAME                                    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
pvc-e91314a3-f225-46b9-9fdb-c2adcea934ad   10Gi     RWO         Delete          Bound   openshift-config/test-pvc   vsphere-sc              10m

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments