1. Configure install-config.yaml. Change the appropriate variables to match the environment, including pullSecret and sshKey.

   apiVersion: v1
   baseDomain: <domain>
   metadata:
     name: <cluster-name>
   networking:
     machineCIDR: <public-cidr>
     networkType: OVNKubernetes
   compute:
   - name: worker
     replicas: 2 1
   controlPlane:
     name: master
     replicas: 3
     platform:
       baremetal: {}
   platform:
     baremetal:
       apiVIP: <api-ip>
       ingressVIP: <wildcard-ip>
       provisioningNetworkInterface: <NIC1>
       provisioningNetworkCIDR: <CIDR>
       hosts:
         - name: openshift-master-0
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 2
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           hardwareProfile: default
         - name: <openshift-master-1>
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 3
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           hardwareProfile: default
         - name: <openshift-master-2>
           role: master
           bmc:
             address: ipmi://<out-of-band-ip> 4
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           hardwareProfile: default
         - name: <openshift-worker-0>
           role: worker
           bmc:
             address: ipmi://<out-of-band-ip> 5
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           hardwareProfile: unknown
         - name: <openshift-worker-1>
           role: worker
           bmc:
             address: ipmi://<out-of-band-ip>
             username: <user>
             password: <password>
           bootMACAddress: <NIC1-mac-address>
           hardwareProfile: unknown
   pullSecret: '<pull_secret>'
   sshKey: '<ssh_pub_key>'

   1

   Scale the worker machines based on the number of worker nodes that are part of the OpenShift Container Platform cluster.

   2 3 4 5

   Refer to the BMC addressing sections for more options.

2. Create a directory to store cluster configs.

   $ mkdir ~/clusterconfigs
   $ cp install-config.yaml ~/clusterconfigs

3. Ensure all bare metal nodes are powered off prior to installing the OpenShift Container Platform cluster.

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

4. Remove old bootstrap resources if any are left over from a previous deployment attempt.

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

1. Create the OpenShift Container Platform manifests.

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
   WARNING Discarding the OpenShift Manifest that was provided in the target directory because its dependencies are dirty and it needs to be regenerated

Procedure

1. Open the firewall port on the registry node.

   $ sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent
   $ sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent
   $ sudo firewall-cmd --reload

2. Install the required packages for the registry node.

   $ sudo yum -y install python3 podman httpd httpd-tools jq

3. Create the directory structure where the repository information will be held.

   $ sudo mkdir -p /opt/registry/{auth,certs,data}

Procedure

1. Adjust the certificate information as appropriate.

   $ host_fqdn=$( hostname --long )
   $ cert_c="<Country Name>"   # Country Name (C, 2 letter code)
   $ cert_s="<State>"          # Certificate State (S)
   $ cert_l="<Locality>"       # Certificate Locality (L)
   $ cert_o="<Organization>"   # Certificate Organization (O)
   $ cert_ou="<Org Unit>"      # Certificate Organizational Unit (OU)
   $ cert_cn="${host_fqdn}"    # Certificate Common Name (CN)
   
   $ openssl req \
       -newkey rsa:4096 \
       -nodes \
       -sha256 \
       -keyout /opt/registry/certs/domain.key \
       -x509 \
       -days 365 \
       -out /opt/registry/certs/domain.crt \
       -addext "subjectAltName = DNS:${host_fqdn}" \
       -subj "/C=${cert_c}/ST=${cert_s}/L=${cert_l}/O=${cert_o}/OU=${cert_ou}/CN=${cert_cn}"

   Note

   When replacing <Country Name>, ensure that it only contains two letters. For example, US.

2. Update the registry node’s ca-trust with the new certificate.

   $ sudo cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
   $ sudo update-ca-trust extract

Procedure

1. Create an htpasswd file in /opt/registry/auth for the container to use.

   $ htpasswd -bBc /opt/registry/auth/htpasswd <user> <passwd>

   Replace <user> with the user name and <passwd> with the password.

2. Create and start the registry container.

   $ podman create \
     --name ocpdiscon-registry \
     -p 5000:5000 \
     -e "REGISTRY_AUTH=htpasswd" \
     -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" \
     -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" \
     -e "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd" \
     -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
     -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
     -e "REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true" \
     -v /opt/registry/data:/var/lib/registry:z \
     -v /opt/registry/auth:/auth:z \
     -v /opt/registry/certs:/certs:z \
     docker.io/library/registry:2

   $ podman start ocpdiscon-registry

Procedure

1. Copy the pull-secret.txt file.

   $ scp kni@provisioner:/home/kni/pull-secret.txt pull-secret.txt

2. Update the host_fqdn environment variable with the fully qualified domain name of the registry node.

   $ host_fqdn=$( hostname --long )

3. Update the b64auth environment variable with the base64 encoding of the http credentials used to create the htpasswd file.

   $ b64auth=$( echo -n '<username>:<passwd>' | openssl base64 )

   Replace <username> with the user name and <passwd> with the password.

4. Set the AUTHSTRING environment variable to use the base64 authorization string. The $USER variable is an environment variable containing the name of the current user.

   $ AUTHSTRING="{\"$host_fqdn:5000\": {\"auth\": \"$b64auth\",\"email\": \"$USER@redhat.com\"}}"

5. Update the pull-secret.txt file.

   $ jq ".auths += $AUTHSTRING" < pull-secret.txt > pull-secret-update.txt

Procedure

1. Copy the oc binary from the provisioner node to the registry node.

   $ sudo scp kni@provisioner:/usr/local/bin/oc /usr/local/bin

2. Set the required environment variables.

   1. Set the release version:

      $ VERSION=<release_version>

      For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.6.

   2. Set the local registry name and host port:

      $ LOCAL_REG='<local_registry_host_name>:<local_registry_host_port>'

      For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3. Set the local repository name:

      $ LOCAL_REPO='<local_repository_name>'

      For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

3. Mirror the remote install images to the local repository.

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.txt \
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

Procedure

1. Add the disconnected registry node’s certificate to the install-config.yaml file. The certificate should follow the "additionalTrustBundle: |" line and be properly indented, usually by two spaces.

   $ echo "additionalTrustBundle: |" >> install-config.yaml
   $ sed -e 's/^/  /' /opt/registry/certs/domain.crt >> install-config.yaml

2. Add the mirror information for the registry to the install-config.yaml file.

   $ echo "imageContentSources:" >> install-config.yaml
   $ echo "- mirrors:" >> install-config.yaml
   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
   $ echo "  source: quay.io/openshift-release-dev/ocp-release" >> install-config.yaml
   $ echo "- mirrors:" >> install-config.yaml
   $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
   $ echo "  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev" >> install-config.yaml

   Note

   Replace registry.example.com with the registry’s fully qualified domain name.

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. To check the container logs, execute the following:

   [core@localhost ~]$ sudo podman logs -f <container-name>

   Replace <container-name> with one of ironic-api, ironic-conductor, or ironic-inspector. If you encounter an issue where the control plane nodes are not booting up via PXE, check the ironic-conductor pod. The ironic-conductor pod contains the most detail about the attempt to boot the cluster nodes, because it attempts to log in to the node over IPMI.

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

2. Check the status of the ipa-downloader and coreos-downloader containers within the bootstrap VM:

   [core@localhost ~]$ sudo podman logs -f ipa-downloader

   [core@localhost ~]$ sudo podman logs -f coreos-downloader

   If the bootstrap VM cannot access the URL to the images, use the curl command to verify that the VM can access the images.

3. To inspect the bootkube logs that indicate if all the containers launched during the deployment phase, execute the following:

   [core@localhost ~]$ journalctl -xe

   [core@localhost ~]$ journalctl -b -f -u bootkube.service

4. Verify all the pods, including dnsmasq, mariadb, httpd, and ironic, are running:

   [core@localhost ~]$ sudo podman ps

5. If there are issues with the pods, check the logs of the containers with issues. To check the log of the ironic-api, execute the following:

   [core@localhost ~]$ sudo podman logs <ironic-api>

Procedure

1. Inspect the pods in the openshift-network-operator namespace:

   $ oc get all -n openshift-network-operator

   NAME                                    READY STATUS            RESTARTS   AGE
   pod/network-operator-69dfd7b577-bg89v   0/1   ContainerCreating 0          149m

2. On the provisioner node, determine that the network configuration exists:

   $ kubectl get network.config.openshift.io cluster -oyaml

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNetwork:
     - 172.30.0.0/16
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     networkType: OpenShiftSDN

   If it does not exist, the installer did not create it. To determine why the installer did not create it, execute the following:

   $ openshift-install create manifests

3. Check that the network-operator is running:

   $ kubectl -n openshift-network-operator get pods

4. Retrieve the logs:

   $ kubectl -n openshift-network-operator logs -l "name=network-operator"

   On high availability clusters with three or more control plane (master) nodes, the Operator will perform leader election and all other Operators will sleep. For additional details, see Troubleshooting.

1. Ensure the reserved IPv6 addresses reside outside the DHCP range.

2. In the IP address reservation on the DHCP server, ensure the reservation specifies the correct DHCP Unique Identifier (DUID). For example:

   # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC
   id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6]

3. Ensure that route announcements are working.
4. Ensure that the DHCP server is listening on the required interfaces serving the IP address ranges.

Procedure

1. Retrieve the hostname:

   [core@master-X ~]$ hostname

   If the hostname is localhost, proceed with the following steps.

   Note

   Where X is the control plane node (also known as the master node) number.

2. Force the cluster node to renew the DHCP lease:

   [core@master-X ~]$ sudo nmcli con up "<bare-metal-nic>"

   Replace <bare-metal-nic> with the wired connection corresponding to the baremetal network.

3. Check hostname again:

   [core@master-X ~]$ hostname

4. If the hostname is still localhost.localdomain, restart NetworkManager:

   [core@master-X ~]$ sudo systemctl restart NetworkManager

5. If the hostname is still localhost.localdomain, wait a few minutes and check again. If the hostname remains localhost.localdomain, repeat the previous steps.

6. Restart the nodeip-configuration service:

   [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service

   This service will reconfigure the kubelet service with the correct hostname references.

7. Reload the unit files definition since the kubelet changed in the previous step:

   [core@master-X ~]$ sudo systemctl daemon-reload

8. Restart the kubelet service:

   [core@master-X ~]$ sudo systemctl restart kubelet.service

9. Ensure kubelet booted with the correct hostname:

   [core@master-X ~]$ sudo journalctl -fu kubelet.service

Addressing a csr

1. Get CSRs on the cluster:

   $ oc get csr

2. Verify if a pending csr contains Subject Name: localhost.localdomain:

   $ oc get csr <pending_csr> -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text

3. Remove any csr that contains Subject Name: localhost.localdomain:

   $ oc delete csr <wrong_csr>

1. Get the route:

   $ oc get route oauth-openshift

2. Check the service endpoint:

   $ oc get svc oauth-openshift

   NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   oauth-openshift   ClusterIP   172.30.19.162   <none>        443/TCP   59m

3. Attempt to reach the service from a control plane (master) node:

   [core@master0 ~]$ curl -k https://172.30.19.162

   {
     "kind": "Status",
     "apiVersion": "v1",
     "metadata": {
     },
     "status": "Failure",
     "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
     "reason": "Forbidden",
     "details": {
     },
     "code": 403

4. Identify the authentication-operator errors from the provisioner node:

   $ oc logs deployment/authentication-operator -n openshift-authentication-operator

   Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting"

Solution

1. Ensure that the cluster name for every deployment is unique, ensuring no conflict.
2. Turn off all the rogue nodes which are not part of the cluster deployment that are using the same cluster name. Otherwise, the authentication pod of the OpenShift Container Platform cluster might never start successfully.

Procedure

1. Connect to the node where the Ignition configuration failed:

   Failed Units: 1
     machine-config-daemon-firstboot.service

2. Restart the machine-config-daemon-firstboot service:

   [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service

Procedure

1. Check for differences in the AGE of the cluster nodes. For example:

   $ oc get nodes

   NAME                         STATUS   ROLES    AGE   VERSION
   master-0.cloud.example.com   Ready    master   145m   v1.16.2
   master-1.cloud.example.com   Ready    master   135m   v1.16.2
   master-2.cloud.example.com   Ready    master   145m   v1.16.2
   worker-2.cloud.example.com   Ready    worker   100m   v1.16.2

2. Check for inconsistent timing delays due to clock drift. For example:

   $ oc get bmh -n openshift-machine-api

   master-1   error registering master-1  ipmi://<out-of-band-ip>

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 18:20:02 UTC
              Universal time: Tue 2020-03-10 18:20:02 UTC
                    RTC time: Tue 2020-03-10 18:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: no
                 NTP service: active
             RTC in local TZ: no

Addressing clock drift in existing clusters

1. Create a chrony.conf file and encode it as base64 string. For example:

   $ cat << EOF | base 64
   server <NTP-server> iburst1
   stratumweight 0
   driftfile /var/lib/chrony/drift
   rtcsync
   makestep 10 3
   bindcmdaddress 127.0.0.1
   bindcmdaddress ::1
   keyfile /etc/chrony.keys
   commandkey 1
   generatecommandkey
   noclientlog
   logchange 0.5
   logdir /var/log/chrony
   EOF

   1

   Replace <NTP-server> with the IP address of the NTP server. Copy the output.

   [text-in-base-64]

2. Create a MachineConfig object, replacing the base64 string with the [text-in-base-64] string generated in the output of the previous step. The following example adds the file to the control plane (master) nodes. You can modify the file for worker nodes or make an additional machine config for the worker role.

   $ cat << EOF > ./99_masters-chrony-configuration.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     creationTimestamp: null
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-master-etc-chrony-conf
   spec:
     config:
       ignition:
         config: {}
         security:
           tls: {}
         timeouts: {}
         version: 3.1.0
       networkd: {}
       passwd: {}
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,[text-in-base-64]1
           group:
             name: root
           mode: 420
           overwrite: true
           path: /etc/chrony.conf
           user:
             name: root
     osImageURL: ""

   1

   Replace [text-in-base-64] with the base64 string.

3. Make a backup copy of the configuration file. For example:

   $ cp 99_masters-chrony-configuration.yaml 99_masters-chrony-configuration.yaml.backup

4. Apply the configuration file:

   $ oc apply -f ./masters-chrony-configuration.yaml

5. Ensure the System clock synchronized value is yes:

   $ sudo timedatectl

                  Local time: Tue 2020-03-10 19:10:02 UTC
              Universal time: Tue 2020-03-10 19:10:02 UTC
                    RTC time: Tue 2020-03-10 19:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: yes
                 NTP service: active
             RTC in local TZ: no

   To setup clock synchronization prior to deployment, generate the manifest files and add this file to the openshift directory. For example:

   $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml

   Then, continue to create the cluster.