Procedure

1. Log in to the RedHat Hybrid Cloud Console.
2. In the menu, click OpenShift.
3. Click Create cluster.
4. Click the Datacenter tab.
5. Under the Assisted Installer section, select Create cluster.
6. Enter a name for the cluster in the Cluster name field.

7. Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.

   Note

   The base domain must be a valid DNS name. You must not have a wild card domain set up for the base domain.

8. Select the version of OpenShift Container Platform to install.
9. Optional: Select Install single node Openshift (SNO) if you want to install OpenShift Container Platform on a single node.
10. Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.
11. Optional: Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on arm64 CPUs, select Use arm64 CPU architecture. Keep in mind, some features are not available with arm64 CPU architecture.
12. Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds.
13. Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select Control plane node, worker for single-node OpenShift. For multi-node clusters, you can select Control plane nodes to encrypt the control plane node installation disks and select Workers to encrypt worker node installation disks.

Procedure

1. Select the internet protocol version. Valid options are IPv4 and Dual stack.
2. If the cluster hosts are on a shared VLAN, enter the VLAN ID.

3. Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.

   1. Enter the cluster network’s IP address range in CIDR notation.
   2. Enter the default gateway IP address.
   3. Enter the DNS server IP addresss.

4. Enter the host-specific configuration.

   1. If you are only setting a static IP address that uses a single network interface, use the form view to enter the IP address and the MAC address for each host.
   2. If you are using multiple interfaces, bonding, or other advanced networking features, use the YAML view and enter the desired network state for each host using NMState syntax. Then, add the MAC address and interface name for each host interface used in your network configuration.

Procedure

1. To install OpenShift Virtualization, select Install OpenShift Virtualization.
2. To install OpenShift Data Foundation Logical Volume Manager, select Install OpenShift Data Foundation Logical Volume Manager.

Procedure

1. Click the Add hosts button and select the installation media.

   1. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method.
   2. Select Full image file: Provision with physical media to download the larger full image.
   3. Select iPXE: Provision from your network server to boot the hosts using iPXE.
2. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.
3. Add an SSH public key so that you can connect to the cluster nodes as the core user. Having a login to the cluster nodes can provide you with debugging information during the installation.
4. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
5. Configure the discovery image if needed.
6. Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform.
7. Click Generate Discovery ISO or Generate Script File.
8. Download the discovery ISO or iPXE script.
9. Boot the host(s) with the discovery image or iPXE script.

Procedure

1. From the Options (⋮) menu for a host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.

   Alternatively, from the Actions list, select Change hostname to rename multiple selected hosts. In the Change Hostname dialog, type the new name and include {{n}} to make each hostname unique. Then click Change.

   Note

   You can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.

2. From the Options (⋮) menu, you can select Delete host to delete a host. Click Delete to confirm the deletion.

   Alternatively, from the Actions list, select Delete to delete multiple selected hosts at the same time. Then click Delete hosts.

   Note

   In a regular deployment, a cluster can have three or more hosts, and three of these must be control plane hosts. If you delete a host that is also a control plane, or if you are left with only two hosts, you will get a message saying that the system is not ready. To restore a host, you will need to reboot it from the discovery ISO.

3. From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.

4. For multi-host clusters, in the Role column next to the host name, you can click on the menu to change the role of the host.

   If you do not select a role, the Assisted Installer will assign the role automatically. The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.

5. Click the Status link to view hardware, network and operator validations for the host.
6. Click the arrow to the left of a host name to expand the host details.

Procedure

1. To the left of the checkbox next to a host name, click to display the storage disks for that host.
2. If there are multiple storage disks for a host, you can select a different disk to act as the installation disk. Click the Role dropdown list for the disk, and then select Installation disk. The role of the previous installation disk changes to None.
3. All bootable disks are marked for reformatting during the installation by default, with the exception of read-only disks such as CDROMs. Deselect the Format checkbox to prevent a disk from being reformatted. The installation disk must be reformatted. Back up any sensitive data before proceeding.

Procedure

1. In the Networking page, select one of the following if it is not already selected for you:

   • Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology, including keepalived and Virtual Router Redundancy Protocol (VRRP) for managing the API and Ingress VIP addresses.
   • User-Managed Networking: Selecting user-managed networking allows you to deploy OpenShift Container Platform with a non-standard network topology. For example, if you want to deploy with an external load balancer instead of keepalived and VRRP, or if you intend to deploy the cluster nodes across many distinct L2 network segments.

2. For cluster-managed networking, configure the following settings:

   1. Define the Machine network. You can use the default network or select a subnet.
   2. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   3. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.

3. For user-managed networking, configure the following settings:

   1. Select your Networking stack type:

      • IPv4: Select this type when your hosts are only using IPv4.
      • Dual-stack: You can select dual-stack when your hosts are using IPv4 together with IPv6.
   2. Define the Machine network. You can use the default network or select a subnet.
   3. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   4. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.
   5. Optional: You can select Allocate IPs via DHCP server to automatically allocate the API IP and Ingress IP using the DHCP server.

4. Optional: Select Use advanced networking to configure the following advanced networking properties:

   • Cluster network CIDR: Define an IP address block from which Pod IP addresses are allocated.
   • Cluster network host prefix: Define a subnet prefix length to assign to each node.
   • Service network CIDR: Define an IP address to use for service IP addresses.
   • Network type: Select either Software-Defined Networking (SDN) for standard networking or Open Virtual Networking (OVN) for IPv6, dual-stack networking, and telco features. In OpenShift Container Platform 4.12 and later releases, OVN is the default Container Network Interface (CNI).

Procedure

1. Press Begin installation.
2. Click on the link in the Status column of the Host Inventory list to see the installation status of a particular host.

Procedure

1. Make a copy of the kubeadmin username and password.

2. Download the kubeconfig file and copy it to the auth directory under your working directory:

   $ mkdir -p <working_directory>/auth

   $ cp kubeadmin <working_directory>/auth

   Note

   The kubeconfig file is available for download for 24 hours after completing the installation.

3. Add the kubeconfig file to your environment:

   $ export KUBECONFIG=<your working directory>/auth/kubeconfig

4. Login with the oc CLI tool:

   $ oc login -u kubeadmin -p <password>

   Replace <password> with the password of the kubeadmin user.

5. Click on the web console URL or click Launch OpenShift Console to open the console.
6. Enter the kubeadmin username and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers.
7. Add a bookmark of the OpenShift Container Platform console.
8. Complete any post-installation platform integration steps.

Procedure

1. In the menu, click OpenShift.
2. In the submenu, click Downloads.
3. In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.

4. Click Load Token.

   Important

   Disable pop-up blockers.

5. In the Your API token section, copy the offline token.

6. In your terminal, set the offline token to the OFFLINE_TOKEN variable:

   $ export OFFLINE_TOKEN=<copied_api_token>

   Tip

   To make the offline token permanent, add it to your profile.

7. Click Download ocm CLI.
8. Copy the downloaded file to your path. For example, copy the file to /usr/bin or ~/.local/bin and create an ocm symbolic link.

9. Copy and paste the authentication command to your terminal and press Enter to login:

   $ ocm login --token="${OFFLINE_TOKEN}"

Procedure

1. Set the API_TOKEN variable using the OFFLINE_TOKEN to validate the user.

   1. (Optional) On the command line terminal, execute the following command:

      $ export API_TOKEN=$( \
        curl \
        --silent \
        --header "Accept: application/json" \
        --header "Content-Type: application/x-www-form-urlencoded" \
        --data-urlencode "grant_type=refresh_token" \
        --data-urlencode "client_id=cloud-services" \
        --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
        "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
        | jq --raw-output ".access_token" \
      )

   2. (Optional) On the command line terminal, login to the ocm client:

      $ ocm login --token="${OFFLINE_TOKEN}"

      Then, generate an API token:

      $ export API_TOKEN=$(ocm token)

2. Create a script in your path for one of the token generating methods. For example:

   $ vim ~/.local/bin/refresh-token

   export API_TOKEN=$( \
     curl \
     --silent \
     --header "Accept: application/json" \
     --header "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "grant_type=refresh_token" \
     --data-urlencode "client_id=cloud-services" \
     --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
     "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
     | jq --raw-output ".access_token" \
   )

   Then, save the file.

3. Change the file mode to make it executable:

   $ chmod +x ~/.local/bin/refresh-token

4. Refresh the API token:

   $ source refresh-token

5. Verify that you can access the API by running the following command:

   $ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jq

   Example output

   {
     "release_tag": "v2.11.3",
     "versions": {
       "assisted-installer": "registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:v1.0.0-211",
       "assisted-installer-controller": "registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:v1.0.0-266",
       "assisted-installer-service": "quay.io/app-sre/assisted-service:78d113a",
       "discovery-agent": "registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:v1.0.0-195"
     }
   }

Procedure

1. In the menu, click OpenShift.
2. In the submenu, click Downloads.
3. In the Tokens section under Pull secret, click Download.

4. To use the pull secret from a shell variable, execute the following command:

   $ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)

5. To slurp the pull secret file using jq, reference it in the pull_secret variable, piping the value to tojson to ensure that it is properly formatted as escaped JSON. For example:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
           --slurpfile pull_secret ~/Downloads/pull-secret.txt ' 1
       {
           "name": "testcluster",
           "high_availability_mode": "None",
           "openshift_version": "4.11",
           "pull_secret": $pull_secret[0] | tojson, 2
           "base_dns_domain": "example.com"
       }
   ')"

   1

   Slurp the pull secret file.

   2

   Format the pull secret to escaped JSON format.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new cluster.

   1. Optional: You can register a new cluster by slurping the pull secret file in the request:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
          --slurpfile pull_secret ~/Downloads/pull-secret.txt '
      {
          "name": "testcluster",
          "openshift_version": "4.11",
          "high_availability_mode": "None",
          "base_dns_domain": "example.com",
          "pull_secret": $pull_secret[0] | tojson
      }
      ')" | jq '.id'

   2. Optional: You can register a new cluster by writing the configuration to a JSON file and then referencing it in the request:

      cat << EOF > cluster.json
      {
        "name": "testcluster",
        "openshift_version": "4.11",
        "high_availability_mode": "None",
        "base_dns_domain": "example.com",
        "pull_secret": $PULL_SECRET
      }
      EOF

      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \
        -d @./cluster.json \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
        | jq '.id'

3. Assign the returned cluster_id to the CLUSTER_ID variable and export it:

   $ export CLUSTER_ID=<cluster_id>

   Note

   If you close your terminal session, you need to export the CLUSTER_ID variable again in a new terminal session.

4. Check the status of the new cluster:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
     | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the cluster. For example:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
   {
       "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZrD4LMkAEeoU2vShhF8VM+cCZtVRgB7tqtsMxms2q3TOJZAgfuqReKYWm+OLOZTD+DO3Hn1pah/mU3u7uJfTUg4wEX0Le8zBu9xJVym0BVmSFkzHfIJVTn6SfZ81NqcalisGWkpmkKXVCdnVAX6RsbHfpGKk9YPQarmRCn5KzkelJK4hrSWpBPjdzkFXaIpf64JBZtew9XVYA3QeXkIcFuq7NBuUH9BonroPEmIXNOa41PUP1IWq3mERNgzHZiuU8Ks/pFuU5HCMvv4qbTOIhiig7vidImHPpqYT/TCkuVi5w0ZZgkkBeLnxWxH0ldrfzgFBYAxnpTU8Ih/4VhG538Ix1hxPaM6cXds2ic71mBbtbSrk+zjtNPaeYk1O7UpcCw4jjHspU/rVV/DY51D5gSiiuaFPBMucnYPgUxy4FMBFfGrmGLIzTKiLzcz0DiSz1jBeTQOX++1nz+KDLBD8CPdi5k4dq7lLkapRk85qdEvgaG5RlHMSPSS3wDrQ51fD8= user@hostname"
   }
   ' | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the image_type. You can specify either full-iso or minimal-iso. The default value is minimal-iso.

   1. Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:

      $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
        --slurpfile pull_secret ~/Downloads/pull-secret.txt \
        --arg cluster_id ${CLUSTER_ID} '
          {
            "name": "testcluster-infra-env",
            "image_type":"full-iso",
            "cluster_id": $cluster_id,
            "pull_secret": $pull_secret[0] | tojson
          }
      ')" | jq '.id'

   2. Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:

      $ cat << EOF > infra-envs.json
      {
       "name": "testcluster-infra-env",
       "image_type": "full-iso",
       "cluster_id": "$CLUSTER_ID",
       "pull_secret": $PULL_SECRET
      }
      EOF

      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs" \
       -d @./infra-envs.json \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $API_TOKEN" \
       | jq '.id'

3. Assign the returned id to the INFRA_ENV_ID variable and export it:

   $ export INFRA_ENV_ID=<id>

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the infrastructure environment:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
     --slurpfile pull_secret ~/Downloads/pull-secret.txt '
       {
         "image_type":"minimal-iso",
         "pull_secret": $pull_secret[0] | tojson
       }
   ')" | jq

Procedure

1. Configure the discovery image if needed.

2. Refresh the API token:

   $ source refresh-token

3. Get the download URL:

   $ curl -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

4. Download the discovery image:

   $ wget -O discovery.iso '<url>'

   Replace <url> with the download URL from the previous step.

5. Boot the host(s) with the discovery image. If you are installing on a platform and want to integrate with the platform, see the additional resources below for details.
6. Assign a role to host(s).

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

   Example output

   [
     "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
   ]

3. Modify the host:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ 1
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
         "host_name" : "worker-1"
       }
   ' | jq

   1

   Replace <host_id> with the ID of the host.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Install the cluster:

   $ curl -H "Authorization: Bearer $API_TOKEN" \
   -X POST \
   https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq

3. Complete any post-installation platform integration steps.

Procedure

1. Optional: Using the UI, in the Cluster details step of the user interface wizard, choose to enable TPM v2 encryption on either the control plane nodes, workers, or both.

2. Optional: Using the API, follow the "Modifying hosts" procedure. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tpmv2.

   1. Refresh the API token:

      $ source refresh-token

   2. Enable TPM v2 encryption:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "none",
          "mode": "tpmv2"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none.

Procedure

1. Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.
2. Optional: In the Cluster details step of the user interface wizard, choose to enable Tang encryption on either the control plane nodes, workers, or both. You will be required to enter URLs and thumbprints for the Tang servers.

3. Optional: Using the API, follow the "Modifying hosts" procedure.

   1. Refresh the API token:

      $ source refresh-token

   2. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tang. Set disk_encyrption.tang_servers to provide the URL and thumbprint details about one or more Tang servers:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "all",
          "mode": "tang",
          "tang_servers": "[{\"url\":\"http://tang.example.com:7500\",\"thumbprint\":\"PLjNyRdGw03zlRoGjQYMahSZGu9\"},{\"url\":\"http://tang2.example.com:7500\",\"thumbprint\":\"XYjNyRdGw03zlRoGjQYMahSZGu3\"}]"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none. Within the tang_servers value, comment out the quotes within the object(s).

1. Validates that your environment meets the prerequisites outlined below.

2. Configures virtual machine storage as follows:

   1. For single-node OpenShift clusters version 4.10 and newer, the Assisted Installer configures the hostpath provisioner.
   2. For single-node OpenShift clusters on earlier versions, the Assisted Installer configures the Local Storage Operator.
   3. For multi-node clusters, the Assisted Installer configures OpenShift Data Foundation.

Procedure

1. Create an Ignition file and specify the configuration specification version:

   $ vim ~/ignition.conf

   {
     "ignition": { "version": "3.1.0" }
   }

2. Add configuration data to the Ignition file. For example, add a password to the core user.

   1. Generate a password hash:

      $ openssl passwd -6

   2. Add the generated password hash to the core user:

      {
        "ignition": { "version": "3.1.0" },
        "passwd": {
          "users": [
            {
              "name": "core",
              "passwordHash": "$6$spam$M5LGSMGyVD.9XOboxcwrsnwNdF4irpJdAWy.1Ry55syyUiUssIzIAHaOrUHr2zg6ruD8YNBPW9kW0H8EnKXyc1"
            }
          ]
        }
      }

3. Save the Ignition file and export it to the IGNITION_FILE variable:

   $ export IGNITION_FILE=~/ignition.conf

Procedure

1. Create an ignition_config_override JSON object and redirect it to a file:

   $ jq -n \
     --arg IGNITION "$(jq -c . $IGNITION_FILE)" \
     '{ignition_config_override: $IGNITION}' \
     > discovery_ignition.json

2. Refresh the API token:

   $ source refresh-token

3. Patch the infrastructure environment:

   $ curl \
     --header "Authorization: Bearer $API_TOKEN" \
     --header "Content-Type: application/json" \
     -XPATCH \
     -d @discovery_ignition.json \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID | jq

   The ignition_config_override object references the Ignition file.

4. Download the updated discovery image.

Procedure

1. On the administration host, insert a USB drive into a USB port.

2. Copy the ISO image to the USB drive, for example:

   # dd if=<path_to_iso> of=<path_to_usb> status=progress

   where:

   <path_to_iso>

   is the relative path to the downloaded discovery ISO file, for example, discovery.iso.

   <path_to_usb>

   is the location of the connected USB drive, for example, /dev/sdb.

   After the ISO is copied to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.

Procedure

1. Insert the RHCOS discovery ISO USB drive into the target host.
2. Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.

3. Wait for the host to boot up.

   1. For UI installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.

   2. For API installations, refresh the token, check the enabled host count, and gather the host IDs:

      $ source refresh-token

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.enabled_host_count'

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.host_networks[].host_ids'

      Example output

      [
        "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
      ]

Procedure

1. Copy the ISO file to an HTTP server accessible in your network.

2. Boot the host from the hosted ISO file, for example:

   1. Call the redfish API to set the hosted ISO as the VirtualMedia boot media by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \
      -H "Content-Type: application/json" \
      -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia

      Where:

      <bmc_username>:<bmc_password>

      Is the username and password for the target host BMC.

      <hosted_iso_file>

      Is the URL for the hosted installation ISO, for example: http://webserver.example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine.

      <host_bmc_address>

      Is the BMC IP address of the target host machine.

   2. Set the host to boot from the VirtualMedia device by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -X PATCH -H 'Content-Type: application/json' \
      -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \
      <host_bmc_address>/redfish/v1/Systems/System.Embedded.1

   3. Reboot the host:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "ForceRestart"}' \
      -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

   4. Optional: If the host is powered off, you can boot it using the {"ResetType": "On"} switch. Run the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "On"}' -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

Procedure

1. Download the iPXE script directly from the UI, or get the iPXE script from the Assisted Installer:

   $ curl \
     --silent \
     --header "Authorization: Bearer $API_TOKEN" \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-script

   Example

   #!ipxe
   initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10
   kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

2. Download the required artifacts by extracting URLs from the ipxe-script.

   1. Download the initial RAM disk:

      $ awk '/^initrd /{print $NF}' ipxe-script | curl -o initrd.img

   2. Download the linux kernel:

      $ awk '/^kernel /{print $2}' ipxe-script | curl -o kernel

   3. Download the root filesystem:

      $ grep ^kernel ipxe-script | xargs -n1| grep ^coreos.live.rootfs_url | cut -d = -f 2- | curl -o rootfs.img

3. Change the URLs to the different artifacts in the ipxe-script` to match your local HTTP server. For example:

   #!ipxe
   set webserver http://192.168.0.1
   initrd --name initrd $webserver/initrd.img
   kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

Procedure

1. Go to the Host Discovery tab and scroll down to the Host Inventory table.

2. Select the Auto-assign drop-down for the required host.

   1. Select Control plane node to assign this host a control plane role.
   2. Select Worker to assign this host a worker role.
3. Check the validation status.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

   Example output

   [
     "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
   ]

3. Modify the host_role setting:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
       }
   ' | jq

   Replace <host_id> with the ID of the host.

Prerequisites

1. You have created an infrastructure environment using the API or have created a cluster using the UI.
2. You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
3. You have credentials to use when accessing the API and have exported a token as $API_TOKEN in your shell.
4. You have YAML files with a static network configuration available as server-a.yaml and server-b.yaml.

Procedure

1. Create a temporary file /tmp/request-body.txt with the API request:

   ---
   jq -n --arg NMSTATE_YAML1 "$(cat server-a.yaml)" --arg NMSTATE_YAML2 "$(cat server-b.yaml)" \
   '{
     "static_network_config": [
       {
         "network_yaml": $NMSTATE_YAML1,
         "mac_interface_map": [{"mac_address": "02:00:00:2c:23:a5", "logical_nic_name": "eth0"}, {"mac_address": "02:00:00:68:73:dc", "logical_nic_name": "eth1"}]
       },
       {
         "network_yaml": $NMSTATE_YAML2,
         "mac_interface_map": [{"mac_address": "02:00:00:9f:85:eb", "logical_nic_name": "eth1"}, {"mac_address": "02:00:00:c8:be:9b", "logical_nic_name": "eth0"}]
        }
     ]
   }' >> /tmp/request-body.txt
   ---

2. Refresh the API token:

   $ source refresh-token

3. Send the request to the Assisted Service API endpoint:

   ---
   curl -H "Content-Type: application/json" \
   -X PATCH -d @/tmp/request-body.txt \
   -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID
   ---

Procedure

1. Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
2. Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
3. Optional: Modify ignition files as needed.
4. Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
5. Select the host role. It can be either a worker or a control plane host.
6. Start the installation.

7. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.

   When the host is sucessfully installed, it is listed as a host in the cluster web console.

Procedure

1. Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.

2. Set the $API_URL variable by running the following command:

   $ export API_URL=<api_url> 1

   1

   Replace <api_url> with the Assisted Installer API URL, for example, https://api.openshift.com

3. Import the cluster by running the following commands:

   1. Set the $CLUSTER_ID variable. Log in to the cluster and run the following command:

      $ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')

   2. Set the $CLUSTER_REQUEST variable that is used to import the cluster:

      $ export CLUSTER_REQUEST=$(jq --null-input --arg openshift_cluster_id "$CLUSTER_ID" '{
        "api_vip_dnsname": "<api_vip>", 1
        "openshift_cluster_id": $CLUSTER_ID,
        "name": "<openshift_cluster_name>" 2
      }')

      1

      Replace <api_vip> with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example, api.compute-1.example.com.

      2

      Replace <openshift_cluster_name> with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.

   3. Import the cluster and set the $CLUSTER_ID variable. Run the following command:

      $ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \
        -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')

4. Generate the InfraEnv resource for the cluster and set the $INFRA_ENV_ID variable by running the following commands:

   1. Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.

   2. Set the $INFRA_ENV_REQUEST variable:

      export INFRA_ENV_REQUEST=$(jq --null-input \
          --slurpfile pull_secret <path_to_pull_secret_file> \1
          --arg ssh_pub_key "$(cat <path_to_ssh_pub_key>)" \2
          --arg cluster_id "$CLUSTER_ID" '{
        "name": "<infraenv_name>", 3
        "pull_secret": $pull_secret[0] | tojson,
        "cluster_id": $cluster_id,
        "ssh_authorized_key": $ssh_pub_key,
        "image_type": "<iso_image_type>" 4
      }')

      1

      Replace <path_to_pull_secret_file> with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com.

      2

      Replace <path_to_ssh_pub_key> with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode.

      3

      Replace <infraenv_name> with the plain text name for the InfraEnv resource.

      4

      Replace <iso_image_type> with the ISO image type, either full-iso or minimal-iso.

   3. Post the $INFRA_ENV_REQUEST to the /v2/infra-envs API and set the $INFRA_ENV_ID variable:

      $ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')

5. Get the URL of the discovery ISO for the cluster host by running the following command:

   $ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'

   Example output

   https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12

6. Download the ISO:

   $ curl -L -s '<iso_url>' --output rhcos-live-minimal.iso 1

   1

   Replace <iso_url> with the URL for the ISO from the previous step.

7. Boot the new worker host from the downloaded rhcos-live-minimal.iso.

8. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:

   $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'

   Example output

   2294ba03-c264-4f11-ac08-2f1bb2f8c296

9. Set the $HOST_ID variable for the new host, for example:

   $ HOST_ID=<host_id> 1

   1

   Replace <host_id> with the host ID from the previous step.

10. Check that the host is ready to install by running the following command:

    Note

    Ensure that you copy the entire command including the complete jq expression.

    $ curl -s $API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID -H "Authorization: Bearer ${API_TOKEN}" | jq '
    def host_name($host):
        if (.suggested_hostname // "") == "" then
            if (.inventory // "") == "" then
                "Unknown hostname, please wait"
            else
                .inventory | fromjson | .hostname
            end
        else
            .suggested_hostname
        end;
    
    def is_notable($validation):
        ["failure", "pending", "error"] | any(. == $validation.status);
    
    def notable_validations($validations_info):
        [
            $validations_info // "{}"
            | fromjson
            | to_entries[].value[]
            | select(is_notable(.))
        ];
    
    {
        "Hosts validations": {
            "Hosts": [
                .hosts[]
                | select(.status != "installed")
                | {
                    "id": .id,
                    "name": host_name(.),
                    "status": .status,
                    "notable_validations": notable_validations(.validations_info)
                }
            ]
        },
        "Cluster validations info": {
            "notable_validations": notable_validations(.validations_info)
        }
    }
    ' -r

    Example output

    {
      "Hosts validations": {
        "Hosts": [
          {
            "id": "97ec378c-3568-460c-bc22-df54534ff08f",
            "name": "localhost.localdomain",
            "status": "insufficient",
            "notable_validations": [
              {
                "id": "ntp-synced",
                "status": "failure",
                "message": "Host couldn't synchronize with any NTP server"
              },
              {
                "id": "api-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "api-int-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "apps-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              }
            ]
          }
        ]
      },
      "Cluster validations info": {
        "notable_validations": []
      }
    }

11. When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:

    $ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install"  -H "Authorization: Bearer ${API_TOKEN}"

12. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.

    Important

    You must approve the CSRs to complete the installation.

    Keep running the following API call to monitor the cluster installation:

    $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq '{
        "Cluster day-2 hosts":
            [
                .hosts[]
                | select(.status != "installed")
                | {id, requested_hostname, status, status_info, progress, status_updated_at, updated_at, infra_env_id, cluster_id, created_at}
            ]
    }'

    Example output

    {
      "Cluster day-2 hosts": [
        {
          "id": "a1c52dde-3432-4f59-b2ae-0a530c851480",
          "requested_hostname": "control-plane-1",
          "status": "added-to-existing-cluster",
          "status_info": "Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs",
          "progress": {
            "current_stage": "Done",
            "installation_percentage": 100,
            "stage_started_at": "2022-07-08T10:56:20.476Z",
            "stage_updated_at": "2022-07-08T10:56:20.476Z"
          },
          "status_updated_at": "2022-07-08T10:56:20.476Z",
          "updated_at": "2022-07-08T10:57:15.306369Z",
          "infra_env_id": "b74ec0c3-d5b5-4717-a866-5b6854791bd3",
          "cluster_id": "8f721322-419d-4eed-aa5b-61b50ea586ae",
          "created_at": "2022-07-06T22:54:57.161614Z"
        }
      ]
    }

13. Optional: Run the following command to see all the events for the cluster:

    $ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'

    Example output

    {"severity":"info","message":"Host compute-0: updated status from insufficient to known (Host is ready to be installed)","event_time":"2022-07-08T11:21:46.346Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from known to installing (Installation is in progress)","event_time":"2022-07-08T11:28:28.647Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing to installing-in-progress (Starting installation)","event_time":"2022-07-08T11:28:52.068Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Uploaded logs for host compute-0 cluster 8f721322-419d-4eed-aa5b-61b50ea586ae","event_time":"2022-07-08T11:29:47.802Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing-in-progress to added-to-existing-cluster (Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs)","event_time":"2022-07-08T11:29:48.259Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host: compute-0, reached installation stage Rebooting","event_time":"2022-07-08T11:29:48.261Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}

14. Log in to the cluster and approve the pending CSRs to complete the installation.

Procedure

1. In Host discovery, click the Add hosts button and select the installation media.
2. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot.
3. Add an SSH public key so that you can connect to the Nutanix VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.
4. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.
5. Optional: Configure the discovery image if you want to boot it with an ignition file. See Configuring the discovery image for additional details.
6. Click Generate Discovery ISO.
7. Copy the Discovery ISO URL.
8. In the Nutanix Prism UI, follow the directions to upload the discovery image from the Assisted Installer.

9. In the Nutanix Prism UI, create the control plane (master) VMs through Prism Central.

   1. Enter the Name. For example, control-plane or master.
   2. Enter the Number of VMs. This should be 3 for the control plane.
   3. Ensure the remaining settings meet the minimum requirements for control plane hosts.

10. In the Nutanix Prism UI, create the worker VMs through Prism Central.

    1. Enter the Name. For example, worker.
    2. Enter the Number of VMs. You should create at least 2 worker nodes.
    3. Ensure the remaining settings meet the minimum requirements for worker hosts.
11. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
12. Move the Integrate with your virtualization platform slider to enable integration with Nutanix.
13. Continue with the installation procedure.

Procedure

1. Configure the discovery image if you want it to boot with an ignition file.

2. Create a Nutanix cluster configuration file to hold the environment variables:

   $ touch ~/nutanix-cluster-env.sh

   $ chmod +x ~/nutanix-cluster-env.sh

   If you have to start a new terminal session, you can reload the environment variables easily. For example:

   $ source ~/nutanix-cluster-env.sh

3. Assign the Nutanix cluster’s name to the NTX_CLUSTER_NAME environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_CLUSTER_NAME=<cluster_name>
   EOF

   Replace <cluster_name> with the name of the Nutanix cluster.

4. Assign the Nutanix cluster’s subnet name to the NTX_SUBNET_NAME environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_SUBNET_NAME=<subnet_name>
   EOF

   Replace <subnet_name> with the name of the Nutanix cluster’s subnet.

5. Refresh the API token:

   $ source refresh-token

6. Get the download URL:

   $ curl -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

7. Create the Nutanix image configuration file:

   $ cat << EOF > create-image.json
   {
     "spec": {
       "name": "ocp_ai_discovery_image.iso",
       "description": "ocp_ai_discovery_image.iso",
       "resources": {
         "architecture": "X86_64",
         "image_type": "ISO_IMAGE",
         "source_uri": "<image_url>",
         "source_options": {
           "allow_insecure_connection": true
         }
       }
     },
     "metadata": {
       "spec_version": 3,
       "kind": "image"
     }
   }
   EOF

   Replace <image_url> with the image URL downloaded from the previous step.

8. Create the Nutanix image:

   $ curl  -k -u <user>:'<password>' -X 'POST' \
   'https://<domain-or-ip>:<port>/api/nutanix/v3/images \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d @./create-image.json | jq '.metadata.uuid'

   Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440.

9. Assign the returned UUID to the NTX_IMAGE_UUID environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_IMAGE_UUID=<uuid>
   EOF

10. Get the Nutanix cluster UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/clusters/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "cluster"
    }'  | jq '.entities[] | select(.spec.name=="<nutanix_cluster_name>") | .metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <nutanix_cluster_name> with the name of the Nutanix cluster.

11. Assign the returned Nutanix cluster UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_CLUSTER_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the Nutanix cluster.

12. Get the Nutanix cluster’s subnet UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/subnets/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "subnet",
      "filter": "name==<subnet_name>"
    }' | jq '.entities[].metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <subnet_name> with the name of the cluster’s subnet.

13. Assign the returned Nutanix subnet UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_SUBNET_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the cluster subnet.

14. Ensure the Nutanix environment variables are set:

    $ source ~/nutanix-cluster-env.sh

15. Create a VM configuration file for each Nutanix host. Create three control plane (master) VMs and at least two worker VMs. For example:

    $ touch create-master-0.json

    $ cat << EOF > create-master-0.json
    {
       "spec": {
          "name": "<host_name>",
          "resources": {
             "power_state": "ON",
             "num_vcpus_per_socket": 1,
             "num_sockets": 16,
             "memory_size_mib": 32768,
             "disk_list": [
                {
                   "disk_size_mib": 122880,
                   "device_properties": {
                      "device_type": "DISK"
                   }
                },
                {
                   "device_properties": {
                      "device_type": "CDROM"
                   },
                   "data_source_reference": {
                     "kind": "image",
                     "uuid": "$NTX_IMAGE_UUID"
                  }
                }
             ],
             "nic_list": [
                {
                   "nic_type": "NORMAL_NIC",
                   "is_connected": true,
                   "ip_endpoint_list": [
                      {
                         "ip_type": "DHCP"
                      }
                   ],
                   "subnet_reference": {
                      "kind": "subnet",
                      "name": "$NTX_SUBNET_NAME",
                      "uuid": "$NTX_SUBNET_UUID"
                   }
                }
             ],
             "guest_tools": {
                "nutanix_guest_tools": {
                   "state": "ENABLED",
                   "iso_mount_state": "MOUNTED"
                }
             }
          },
          "cluster_reference": {
             "kind": "cluster",
             "name": "$NTX_CLUSTER_NAME",
             "uuid": "$NTX_CLUSTER_UUID"
          }
       },
       "api_version": "3.1.0",
       "metadata": {
          "kind": "vm"
       }
    }
    EOF

    Replace <host_name> with the name of the host.

16. Boot each Nutanix virtual machine:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d @./<vm_config_file_name> | jq '.metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <vm_config_file_name> with the name of the VM configuration file.

17. Assign the returned VM UUID to a unique environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_MASTER_0_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the VM.

    Note

    The environment variable must have a unique name for each VM.

18. Wait until the Assisted Installer has discovered each VM and they have passed validation.

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID"
    --header "Content-Type: application/json"
    -H "Authorization: Bearer $API_TOKEN"
    | jq '.enabled_host_count'

19. Modify the cluster definition to enable integration with Nutanix:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "platform_type":"nutanix"
    }
    ' | jq

20. Continue with the installation procedure.

Procedure

1. Update the Nutanix configuration:

   $ oc patch infrastructure/cluster --type=merge --patch-file=/dev/stdin <<-EOF
   {
     "spec": {
       "platformSpec": {
         "nutanix": {
           "prismCentral": {
             "address": "<prismcentral_address>", 1
             "port": <prismcentral_port> 2
           },
           "prismElements": [
             {
               "endpoint": {
                 "address": "<prismelement_address>", 3
                 "port": <prismelement_port> 4
               },
               "name": "<prismelement_clustername>" 5
             }
           ]
         },
         "type": "Nutanix"
       }
     }
   }
   EOF

   1

   Replace <prismcentral_address> with the Nutanix Prism Central IP address.

   2

   Replace <prismcentral_port> with the Nutanix Prism Central port.

   3

   Replace <prismelement_address> with Nutanix Prism Element IP address.

   4

   Replace <prismelement_port> with the Nutanix Prism Element port.

   5

   Replace <prismelement_clustername> with the Nutanix Prism Element cluster name.

   For additional details, see Creating a machine set on Nutanix.

2. Update the secret:

   $ cat <<EOF | oc create -f -
   apiVersion: v1
   kind: Secret
   metadata:
      name: nutanix-credentials
      namespace: openshift-machine-api
   type: Opaque
   stringData:
     credentials: |
       [{"type":"basic_auth","data":{"prismCentral":{"username":"<nutanix_username>","password":"<nutanix_password>"},"prismElements":null}}]
   EOF

   Replace <nutanix_username> with the Nutanix Prism Element login. Replace <nutanix_password> with the Nutanix Prism Element password.

   For additional details, see Configuring the default storage container.

Procedure

1. Configure the discovery image if you want it to boot with an ignition file.

2. Generate and download the ISO from the Assisted Installer UI.

   1. In Host discovery, click the Add hosts button and select the installation media.
   2. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot.
   3. Add an SSH public key so that you can connect to the vSphere VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.
   4. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.
   5. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy or the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates and add the additional certificates.
   6. Optional: Configure the discovery image if you want to boot it with an ignition file. See Configuring the discovery image for additional details.
   7. Click Generate Discovery ISO.
   8. Copy the Discovery ISO URL.

   9. Download the discovery ISO:

      $ wget - O vsphere-discovery-image.iso <discovery_url>

      Replace <discovery_url> with the URL from the preceding step.

3. On the command line, power down and destroy any pre-existing virtual machines:

   $ for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>)
   do
    	/usr/local/bin/govc vm.power -off $VM
    	/usr/local/bin/govc vm.destroy $VM
   done

   Replace <datacenter> with the name of the datacenter. Replace <folder_name> with the name of the VM inventory folder.

4. Remove pre-existing ISO images from the data store, if there are any:

   $ govc datastore.rm -ds <iso_datastore> <image>

   Replace <iso_datastore> with the name of the data store. Replace image with the name of the ISO image.

5. Upload the Assisted Installer discovery ISO:

   $ govc datastore.upload -ds <iso_datastore>  vsphere-discovery-image.iso

   Replace <iso_datastore> with the name of the data store.

   Note

   All nodes in the cluster must boot from the discovery image.

6. Boot three control plane (master) nodes:

   $ govc vm.create -net.adapter <network_adapter_type> \
                    -disk.controller <disk_controller_type> \
                    -pool=<resource_pool> \
                    -c=16 \
                    -m=32768 \
                    -disk=120GB \
                    -disk-datastore=<datastore_file> \
                    -net.address="<nic_mac_address>" \
                    -iso-datastore=<iso_datastore> \
                    -iso="vsphere-discovery-image.iso" \
                    -folder="<inventory_folder>" \
                    <hostname>.<cluster_name>.example.com

   See vm.create for details.

   Note

   The foregoing example illustrates the minimum required resources for control plane nodes.

7. Boot at least two worker nodes:

   $ govc vm.create -net.adapter <network_adapter_type> \
                    -disk.controller <disk_controller_type> \
                    -pool=<resource_pool> \
                    -c=4 \
                    -m=8192 \
                    -disk=120GB \
                    -disk-datastore=<datastore_file> \
                    -net.address="<nic_mac_address>" \
                    -iso-datastore=<iso_datastore> \
                    -iso="vsphere-discovery-image.iso" \
                    -folder="<inventory_folder>" \
                    <hostname>.<cluster_name>.example.com

   See vm.create for details.

   Note

   The foregoing example illustrates the minimum required resources for worker nodes.

8. Ensure the VMs are running:

   $  govc ls /<datacenter>/vm/<folder_name>

   Replace <datacenter> with the name of the datacenter. Replace <folder_name> with the name of the VM inventory folder.

9. After 2 minutes, shut down the VMs:

   $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
   do
        govc vm.power -s=true $VM
   done

   Replace <datacenter> with the name of the datacenter. Replace <folder_name> with the name of the VM inventory folder.

10. Set the disk.enableUUID setting to TRUE:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.change -vm $VM -e disk.enableUUID=TRUE
    done

    Replace <datacenter> with the name of the datacenter. Replace <folder_name> with the name of the VM inventory folder.

    Note

    You must set disk.enableUUID to TRUE on all of the nodes to enable autoscaling with vSphere.

11. Restart the VMs:

    $  for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.power -on=true $VM
    done

    Replace <datacenter> with the name of the datacenter. Replace <folder_name> with the name of the VM inventory folder.

12. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
13. Move the Integrate with your virtualization platform slider to enable integration with vSphere.
14. Select roles if needed.
15. In Networking, uncheck Allocate IPs via DHCP server.
16. Set the API VIP address.
17. Set the Ingress VIP address.
18. Continue with the installation procedure.

Procedure

1. Backup the vSphere credentials:

   $ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml

   $ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config_backup.yaml

2. Generate a base64-encoded username and password for vCenter:

   $ echo -n "<vcenter_username>" | base64 -w0

   Replace <vcenter_username> with your vCenter username.

   $ echo -n "<vcenter_password>" | base64 -w0

   Replace <vcenter_password> with your vCenter password.

3. Edit the vSphere credentials:

   $ cp creds_backup.yaml vsphere-creds.yaml

   $ vi vsphere-creds.yaml

   apiVersion: v1
   data:
     <vcenter_address>.username: <vcenter_username_encoded>
     <vcenter_address>.password: <vcenter_password_encoded>
   kind: Secret
   metadata:
     annotations:
       cloudcredential.openshift.io/mode: passthrough
     creationTimestamp: "2022-01-25T17:39:50Z"
     name: vsphere-creds
     namespace: kube-system
     resourceVersion: "2437"
     uid: 06971978-e3a5-4741-87f9-2ca3602f2658
   type: Opaque

   Replace <vcenter_address> with the vCenter address. Replace <vcenter_username_encoded> with the base64-encoded version of your vSphere username. Replace <vcenter_password_encoded> with the base64-encoded version of your vSphere password.

4. Replace the vSphere credentials:

   $ oc replace -f vsphere-creds.yaml

5. Redeploy the kube-controller-manager pods:

   $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

6. Edit the cloud provider configuration:

   $ cloud-provider-config_backup.yaml cloud-provider-config.yaml

   $ vi cloud-provider-config.yaml

   apiVersion: v1
   data:
     config: |
       [Global]
       secret-name = "vsphere-creds"
       secret-namespace = "kube-system"
       insecure-flag = "1"
   
       [Workspace]
       server = "<vcenter_address>"
       datacenter = "<datacenter>"
       default-datastore = "<datastore>"
       folder = "/<datacenter>/vm/<folder>"
   
       [VirtualCenter "<vcenter_address>"]
       datacenters = "<datacenter>"
   kind: ConfigMap
   metadata:
     creationTimestamp: "2022-01-25T17:40:49Z"
     name: cloud-provider-config
     namespace: openshift-config
     resourceVersion: "2070"
     uid: 80bb8618-bf25-442b-b023-b31311918507

   Replace <vcenter_address> with the vCenter address. Replace <datacenter> with the name of the data center. Replace <datastore> with the name of the data store. Replace <folder> with the folder containing the cluster VMs.

7. Apply the cloud provider configuration:

   $ oc apply -f cloud-provider-config.yaml

Procedure

1. In the Administrator perspective, navigate to Home → Overview.
2. Under Status, click vSphere connection to open the vSphere connection configuration wizard.
3. In the vCenter field, enter the network address of the vSphere vCenter server. This can be either a domain name or an IP address. It appears in the vSphere web client URL; for example https://[your_vCenter_address]/ui.
4. In the Username field, enter your vSphere vCenter username.

5. In the Password field, enter your vSphere vCenter password.

   Warning

   The system stores the username and password in the vsphere-creds secret in the kube-system namespace of the cluster. An incorrect vCenter username or password makes the cluster nodes unschedulable.

6. In the Datacenter field, enter the name of the vSphere data center that contains the virtual machines used to host the cluster; for example, SDDC-Datacenter.

7. In the Default data store field, enter the vSphere data store that stores the persistent data volumes; for example, /SDDC-Datacenter/datastore/datastorename.

   Warning

   Updating the vSphere data center or default data store after the configuration has been saved detaches any active vSphere PersistentVolumes.

8. In the Virtual Machine Folder field, enter the data center folder that contains the virtual machine of the cluster; for example, /SDDC-Datacenter/vm/ci-ln-hjg4vg2-c61657-t2gzr. For the OpenShift Container Platform installation to succeed, all virtual machines comprising the cluster must be located in a single data center folder.
9. Click Save Configuration. This updates the cloud-provider-config file in the openshift-config namespace, and starts the configuration process.
10. Reopen the vSphere connection configuration wizard and expand the Monitored operators panel. Check that the status of the operators is either Progressing or Healthy.

1. Check that the configuration process completed successfully:

   1. In the OpenShift Container Platform Administrator perspective, navigate to Home → Overview.
   2. Under Status click Operators. Wait for all operator statuses to change from Progressing to All succeeded. A Failed status indicates that the configuration failed.
   3. Under Status, click Control Plane. Wait for the response rate of all Control Pane components to return to 100%. A Failed control plane component indicates that the configuration failed.

   A failure indicates that at least one of the connection settings is incorrect. Change the settings in the vSphere connection configuration wizard and save the configuration again.

2. Check that you are able to bind PersistentVolumeClaims objects by performing the following steps:

   1. Create a StorageClass object using the following YAML:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
       name: vsphere-sc
      provisioner: kubernetes.io/vsphere-volume
      parameters:
       datastore: YOURVCENTERDATASTORE
       diskformat: thin
      reclaimPolicy: Delete
      volumeBindingMode: Immediate

   2. Create a PersistentVolumeClaims object using the following YAML:

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: test-pvc
       namespace: openshift-config
       annotations:
         volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
       finalizers:
         - kubernetes.io/pvc-protection
      spec:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
          storage: 10Gi
       storageClassName: vsphere-sc
       volumeMode: Filesystem

   For instructions, see Dynamic provisioning in the OpenShift Container Platform documentation. To troubleshoot a PersistentVolumeClaims object, navigate to Storage → PersistentVolumeClaims in the Administrator perspective of the OpenShift Container Platform UI.

Procedure

1. Verify that your host machine is powered on.
2. If you selected DHCP networking, check that the DHCP server is enabled.
3. If you selected Static IP, bridges and bonds networking, check that your configurations are correct.

4. Verify that you can access your host machine using SSH, a console such as the BMC, or a virtual machine console:

   $ ssh core@<host_ip_address>

   You can specify private key file using the -i parameter if it isn’t stored in the default directory.

   $ ssh -i <ssh_private_key_file> core@<host_ip_address>

   If you fail to ssh to the host, the host failed during boot or it failed to configure the network.

   Upon login you should see this message:

   Example login

   If you are not seeing this message it means that the host didn’t boot with the assisted-installer ISO. Make sure you configured the boot order properly (The host should boot once from the live-ISO).

5. Check the agent service logs:

   $ sudo journalctl -u agent.service

   In the following example, the errors indicate there is a network issue:

   Example agent service log screenshot of agent service log

   

   If there is an error pulling the agent image, check the proxy settings. Verify that the host is connected to the network. You can use nmcli to get additional information about your network configuration.