Chapter 8. Deploying installer-provisioned clusters on bare metal

1. Boot the node and enter the BIOS menu.
2. Set the node’s boot mode to UEFI Enabled.

3. Enable Secure Boot.

   Important

   Red Hat does not support Secure Boot with self-generated keys.

Procedure

1. Log in to the provisioner node via ssh.

2. Create a non-root user (kni) and provide that user with sudo privileges:

   # useradd kni
   # passwd kni
   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni
   # chmod 0440 /etc/sudoers.d/kni

3. Create an ssh key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

4. Log in as the new user on the provisioner node:

   # su - kni
   $

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach
   $ sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhel-8-for-x86_64-baseos-rpms

   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

7. Modify the user to add the libvirt group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

8. Restart firewalld and enable the http service:

   $ sudo systemctl start firewalld
   $ sudo firewall-cmd --zone=public --add-service=http --permanent
   $ sudo firewall-cmd --reload

9. Start and enable the libvirtd service:

   $ sudo systemctl enable libvirtd --now

10. Create the default storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images
    $ sudo virsh pool-start default
    $ sudo virsh pool-autostart default

11. Configure networking.

    Note

    You can also configure networking from the web console.

    Export the baremetal network NIC name:

    $ export PUB_CONN=<baremetal_nic_name>

    Configure the baremetal network:

    $ sudo nohup bash -c "
        nmcli con down \"$PUB_CONN\"
        nmcli con delete \"$PUB_CONN\"
        # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
        nmcli con down \"System $PUB_CONN\"
        nmcli con delete \"System $PUB_CONN\"
        nmcli connection add ifname baremetal type bridge con-name baremetal
        nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
        pkill dhclient;dhclient baremetal
    "

    If you are deploying with a provisioning network, export the provisioning network NIC name:

    $ export PROV_CONN=<prov_nic_name>

    If you are deploying with a provisioning network, configure the provisioning network:

    $ sudo nohup bash -c "
        nmcli con down \"$PROV_CONN\"
        nmcli con delete \"$PROV_CONN\"
        nmcli connection add ifname provisioning type bridge con-name provisioning
        nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
        nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
        nmcli con down provisioning
        nmcli con up provisioning
    "

    Note

    The ssh connection might disconnect after executing these steps.

    The IPv6 address can be any address as long as it is not routable via the baremetal network.

    Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

12. Configure the IPv4 address on the provisioning network connection.

    $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

13. ssh back into the provisioner node (if required).

    # ssh kni@provisioner.<cluster-name>.<domain>

14. Verify the connection bridges have been properly created.

    $ sudo nmcli con show

    NAME               UUID                                  TYPE      DEVICE
    baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
    provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
    virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
    bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
    bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

15. Create a pull-secret.txt file.

    $ vim pull-secret.txt

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure, and scroll down to the Downloads section. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install
   $ export pullsecret_file=~/pull-secret.txt
   $ export extract_dir=$(pwd)

2. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

3. Extract the installer:

   $ sudo cp oc /usr/local/bin
   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
   $ sudo cp openshift-baremetal-install /usr/local/bin

1. Install podman.

   $ sudo dnf install -y podman

2. Open firewall port 8080 to be used for RHCOS image caching.

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   $ sudo firewall-cmd --reload

3. Create a directory to store the bootstraposimage and clusterosimage.

   $ mkdir /home/kni/rhcos_image_cache

4. Set the appropriate SELinux context for the newly created directory.

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"
   $ sudo restorecon -Rv rhcos_image_cache/

5. Get the commit ID from the installer. The ID determines which images the installer needs to download.

   $ export COMMIT_ID=$(/usr/local/bin/openshift-baremetal-install version | grep '^built from commit' | awk '{print $4}')

6. Get the URI for the RHCOS image that the installer will deploy on the nodes.

   $ export RHCOS_OPENSTACK_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.openstack.path | sed 's/"//g')

7. Get the URI for the RHCOS image that the installer will deploy on the bootstrap VM.

   $ export RHCOS_QEMU_URI=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq .images.qemu.path | sed 's/"//g')

8. Get the path where the images are published.

   $ export RHCOS_PATH=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json | jq .baseURI | sed 's/"//g')

9. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM.

   $ export RHCOS_QEMU_SHA_UNCOMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.qemu["uncompressed-sha256"]')

10. Get the SHA hash for the RHCOS image that will be deployed on the nodes.

    $ export RHCOS_OPENSTACK_SHA_COMPRESSED=$(curl -s -S https://raw.githubusercontent.com/openshift/installer/$COMMIT_ID/data/data/rhcos.json  | jq -r '.images.openstack.sha256')

11. Download the images and place them in the /home/kni/rhcos_image_cache directory.

    $ curl -L ${RHCOS_PATH}${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_URI}
    $ curl -L ${RHCOS_PATH}${RHCOS_OPENSTACK_URI} -o /home/kni/rhcos_image_cache/${RHCOS_OPENSTACK_URI}

12. Confirm SELinux type is of httpd_sys_content_t for the newly created files.

    $ ls -Z /home/kni/rhcos_image_cache

13. Create the pod.

    $ podman run -d --name rhcos_image_cache \
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    quay.io/centos7/httpd-24-centos7:latest

Procedure

1. Create a router-replicas.yaml file.

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     replicas: <num-of-router-pods>
     endpointPublishingStrategy:
       type: HostNetwork
     nodePlacement:
       nodeSelector:
         matchLabels:
           node-role.kubernetes.io/worker: ""

   Note

   Replace <num-of-router-pods> with an appropriate value. If working with just one worker node, set replicas: to 1. If working with more than 3 worker nodes, you can increase replicas: from the default value 2 as appropriate.

2. Save and copy the router-replicas.yaml file to the clusterconfigs/openshift directory.

   cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml

Procedure

1. Remove or reformat the disks for the bootstrap, control plane (also known as master) node, and worker nodes. If you are working in a hypervisor environment, you must add any disks you removed.

2. Delete the artifacts that the previous installation generated:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

3. Generate new manifests and Ignition config files. See “Creating the Kubernetes manifest and Ignition config files" for more information.
4. Upload the new bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. This will overwrite the previous Ignition files.

Procedure

1. Get the oc binary, if needed. It should already exist on the provisioner node.

   [kni@provisioner ~]$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc

   [kni@provisioner ~]$ sudo cp oc /usr/local/bin

2. Power off the bare metal node via the baseboard management controller and ensure it is off.

3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the user name and password. In the following example, the user name is root and the password is calvin.

   [kni@provisioner ~]$ echo -ne "root" | base64

   [kni@provisioner ~]$ echo -ne "calvin" | base64

4. Create a configuration file for the bare metal node.

   [kni@provisioner ~]$ vim bmh.yaml

   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: openshift-worker-<num>-bmc-secret
   type: Opaque
   data:
     username: <base64-of-uid>
     password: <base64-of-pwd>
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: openshift-worker-<num>
   spec:
     online: true
     bootMACAddress: <NIC1-mac-address>
     bmc:
       address: <protocol>://<bmc-ip>
       credentialsName: openshift-worker-<num>-bmc-secret

   Replace <num> for the worker number of the bare metal node in the two name fields and the credentialsName field. Replace <base64-of-uid> with the base64 string of the user name. Replace <base64-of-pwd> with the base64 string of the password. Replace <NIC1-mac-address> with the MAC address of the bare metal node’s first NIC.

   Refer to the BMC addressing section for additional BMC configuration options. Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc-ip> with the IP address of the bare metal node’s baseboard management controller.

   Note

   If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See Diagnosing a host duplicate MAC address for more information.

5. Create the bare metal node.

   [kni@provisioner ~]$ oc -n openshift-machine-api create -f bmh.yaml

   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

   Where <num> will be the worker number.

6. Power up and inspect the bare metal node.

   [kni@provisioner ~]$ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

Procedure

1. Ensure the PROVISIONING STATUS is ready before provisioning the bare metal node.

   $  oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER   BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       ready                            ipmi://<out-of-band-ip>   unknown            true

2. Get a count of the number of worker nodes.

   $ oc get nodes

   NAME                                                STATUS   ROLES           AGE     VERSION
   provisioner.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-1.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-2.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-master-3.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-worker-0.openshift.example.com            Ready    master          30h     v1.16.2
   openshift-worker-1.openshift.example.com            Ready    master          30h     v1.16.2

3. Get the machine set.

   $ oc get machinesets -n openshift-machine-api

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   ...
   openshift-worker-0.example.com      1         1         1       1           55m
   openshift-worker-1.example.com      1         1         1       1           55m

4. Increase the number of worker nodes by one.

   $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

   Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the machine set from the previous step.

5. Check the status of the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number. The status changes from ready to provisioning.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       provisioning          openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

   The provisioning status remains until the OpenShift Container Platform cluster provisions the node. This can take 30 minutes or more. Once complete, the status will change to provisioned.

   NAME                 STATUS   PROVISIONING STATUS   CONSUMER                  BMC                 HARDWARE PROFILE   ONLINE   ERROR
   openshift-worker-<num>   OK       provisioned           openshift-worker-<num>-65tjz   ipmi://<out-of-band-ip>   unknown            true

6. Once provisioned, ensure the bare metal node is ready.

   $ oc get nodes

   NAME                                          STATUS   ROLES   AGE     VERSION
   provisioner.openshift.example.com             Ready    master  30h     v1.16.2
   openshift-master-1.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-master-2.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-master-3.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-0.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-1.openshift.example.com      Ready    master  30h     v1.16.2
   openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.16.2

   You can also check the kubelet.

   $ ssh openshift-worker-<num>

   [kni@openshift-worker-<num>]$ journalctl -fu kubelet

Procedure

1. Use the guidelines in YAML-tips.
2. Verify the YAML syntax is correct using syntax-check.

3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the install-config.yaml. For example:

   $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.x86_64.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7

   If the output is 200, there is a valid response from the webserver storing the bootstrap VM image.

Procedure

1. About 10 to 15 minutes after triggering the installer, check to ensure the bootstrap VM is operational using the virsh command:

   $ sudo virsh list

    Id    Name                           State
    --------------------------------------------
    12    openshift-xf6fq-bootstrap      running

   Note

   The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap."

   If the bootstrap VM is not running after 10-15 minutes, troubleshoot why it is not running. Possible issues include:

2. Verify libvirtd is running on the system:

   $ systemctl status libvirtd

   ● libvirtd.service - Virtualization daemon
      Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
        Docs: man:libvirtd(8)
              https://libvirt.org
    Main PID: 9850 (libvirtd)
       Tasks: 20 (limit: 32768)
      Memory: 74.8M
      CGroup: /system.slice/libvirtd.service
              ├─ 9850 /usr/sbin/libvirtd

   If the bootstrap VM is operational, log in to it.

3. Use the virsh console command to find the IP address of the bootstrap VM:

   $ sudo virsh console example.com

   Connected to domain example.com
   Escape character is ^]
   
   Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
   SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
   SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
   SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
   ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
   ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
   localhost login:

   Important

   When deploying a OpenShift Container Platform cluster without the provisioning network, you must use a public IP address and not a private IP address like 172.22.0.2.

4. Once you obtain the IP address, log in to the bootstrap VM using the ssh command:

   Note

   In the console output of the previous step, you can use the IPv6 IP address provided by ens3 or the IPv4 IP provided by ens4.

   $ ssh core@172.22.0.2

Procedure

1. Check the network connectivity to the provisioning network.
2. Ensure PXE is enabled on the NIC for the provisioning network and PXE is disabled for all other NICs.

3. Verify that the install-config.yaml configuration file has the proper hardware profile and boot MAC address for the NIC connected to the provisioning network. For example:

   control plane node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
   hardwareProfile: default          #control plane node settings

   Worker node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
   hardwareProfile: unknown          #worker node settings

Procedure

1. Hostname Resolution: Check the cluster nodes to ensure they have a fully qualified domain name, and not just localhost.localdomain. For example:

   $ hostname

   If a hostname is not set, set the correct hostname. For example:

   $ hostnamectl set-hostname <hostname>

2. Incorrect Name Resolution: Ensure that each node has the correct name resolution in the DNS server using dig and nslookup. For example:

   $ dig api.<cluster-name>.example.com

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster-name>.example.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
   ;; QUESTION SECTION:
   ;api.<cluster-name>.example.com. IN A
   
   ;; ANSWER SECTION:
   api.<cluster-name>.example.com. 10800 IN	A 10.19.13.86
   
   ;; AUTHORITY SECTION:
   <cluster-name>.example.com. 10800 IN NS	<cluster-name>.example.com.
   
   ;; ADDITIONAL SECTION:
   <cluster-name>.example.com. 10800 IN A	10.19.14.247
   
   ;; Query time: 0 msec
   ;; SERVER: 10.19.14.247#53(10.19.14.247)
   ;; WHEN: Tue May 19 20:30:59 UTC 2020
   ;; MSG SIZE  rcvd: 140

   The output in the foregoing example indicates that the appropriate IP address for the api.<cluster-name>.example.com VIP is 10.19.13.86. This IP address should reside on the baremetal network.

Procedure

1. Power off all bare metal nodes prior to installing the OpenShift Container Platform cluster:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off

2. Remove all old bootstrap resources if any are left over from a previous deployment attempt:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Remove the following from the clusterconfigs directory to prevent Terraform from failing:

   $ rm -rf ~/clusterconfigs/auth ~/clusterconfigs/terraform* ~/clusterconfigs/tls ~/clusterconfigs/metadata.json

Procedure

1. Check to ensure authentication is successful:

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.json
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Note

   Example output of the variables used to mirror the install images:

   UPSTREAM_REPO=${RELEASE_IMAGE}
   LOCAL_REG=<registry_FQDN>:<registry_port>
   LOCAL_REPO='ocp4/openshift4'

   The values of RELEASE_IMAGE and VERSION were set during the Retrieving OpenShift Installer step of the Setting up the environment for an OpenShift installation section.

2. After mirroring the registry, confirm that you can access it in your disconnected environment:

   $ curl -k -u <user>:<password> https://registry.example.com:<registry-port>/v2/_catalog
   {"repositories":["<Repo-Name>"]}

Procedure

1. When the OpenShift Container Platform cluster nodes are installed appropriately, the following Ready state is seen within the STATUS column:

   $ oc get nodes

   NAME                   STATUS   ROLES           AGE  VERSION
   master-0.example.com   Ready    master,worker   4h   v1.16.2
   master-1.example.com   Ready    master,worker   4h   v1.16.2
   master-2.example.com   Ready    master,worker   4h   v1.16.2

2. Confirm the installer deployed all pods successfully. The following command removes any pods that are still running or have completed as part of the output.

   $ oc get pods --all-namespaces | grep -iv running | grep -iv complete