Chapter 14. Installing an on-premise cluster with the Agent-based Installer

1. You must set value of the fips field to True in the install-config.yaml file:

   Sample install-config.yaml.file

   apiVersion: v1
   baseDomain: test.example.com
   metadata:
     name: sno-cluster
   fips: True

2. Optional: If you are using the ZTP manifests, you must set the value of fips as True in the Agent-install.openshift.io/install-config-overrides field in the agent-cluster-install.yaml file:

   Sample agent-cluster-install.yaml file

   apiVersion: extensions.hive.openshift.io/v1beta1
   kind: AgentClusterInstall
   metadata:
     annotations:
       agent-install.openshift.io/install-config-overrides: '{"fips": True}'
     name: sno-cluster
     namespace: sno-cluster-test

Procedure

1. Log in to the OpenShift Container Platform web console using your login credentials.

2. Navigate to Datacenter.

   
3. Click Run Agent-based Installer locally. You are directed to the Install OpenShift Container Platform on Bare Metal locally with Agent page.

4. Optional: Alternatively, you can also click Bare Metal (x86_64) on the Select an OpenShift Container Platform cluster type to create page. You are directed to the Create an OpenShift Container Platform Cluster: Bare Metal page. Then, select Local Agent-based to go to the Install OpenShift Container Platform on Bare Metal locally with Agent page.

   
5. Select the operating system and architecture.
6. Click Download Installer to download and extract the install program.
7. You can either download or copy the pull secret by clicking on Download pull secret or Copy pull secret.
8. Click Download command-line tools and place the openshift-install binary in a directory that is on your PATH.

9. Install nmstate dependency by running the following command:

   $ sudo dnf install /usr/bin/nmstatectl -y

10. Place the openshift-install binary in a directory that is on your PATH.

11. Create a directory to store the install configuration by running the following command:

    $ mkdir ~/<directory_name>

    Note

    This is the preferred method for the Agent-based installation. Using ZTP manifests is optional.

12. Create the install-config.yaml file:

    $ cat << EOF > ./my-cluster/install-config.yaml
    apiVersion: v1
    baseDomain: test.example.com
    compute:
    - architecture: amd64
      hyperthreading: Enabled
      name: worker
      replicas: 0
    controlPlane:
      architecture: amd64
      hyperthreading: Enabled
      name: master
      replicas: 1
    metadata:
      name: sno-cluster 1
    networking:
      clusterNetwork:
      - cidr: 10.128.0.0/14
        hostPrefix: 23
      machineNetwork:
      - cidr: 192.168.111.0/16
      networkType: OVNKubernetes 2
      serviceNetwork:
      - 172.30.0.0/16
    platform:
      none: {}
    pullSecret: '<pull_secret>' 3
    sshKey: |
      <ssh_pub_key> 4
      EOF

    1

    Required.

    2

    The cluster network plugin to install. The supported values are OVNKubernetes and OpenShiftSDN. The default value is OVNKubernetes.

    3

    Enter your pull secret.

    4

    Enter your ssh public key.

    Note

    If you set the platform to vSphere or baremetal, you can configure IP address endpoints for cluster nodes in three ways:

    • IPv4
    • IPv6
    • IPv4 and IPv6 in parallel (dual-stack)

    Example of dual-stack networking

    networking:
      clusterNetwork:
        - cidr: 172.21.0.0/16
          hostPrefix: 23
        - cidr: fd02::/48
          hostPrefix: 64
      machineNetwork:
        - cidr: 192.168.11.0/16
        - cidr: 2001:DB8::/32
      serviceNetwork:
        - 172.22.0.0/16
        - fd03::/112
      networkType: OVNKubernetes
    platform:
      baremetal:
        apiVIPs:
        - 192.168.11.3
        - 2001:DB8::4
        ingressVIPs:
        - 192.168.11.4
        - 2001:DB8::5

    IPv6 is supported only on bare metal platforms.

13. Create the agent-config.yaml file:

    $ cat > agent-config.yaml << EOF
    apiVersion: v1alpha1
    kind: AgentConfig
    metadata:
      name: sno-cluster
    rendezvousIP: 192.168.111.80 1
    hosts: 2
      - hostname: master-0 3
        interfaces:
          - name: eno1
            macAddress: 00:ef:44:21:e6:a5
        rootDeviceHints: 4
          deviceName: /dev/sdb
        networkConfig: 5
          interfaces:
            - name: eno1
              type: ethernet
              state: up
              mac-address: 00:ef:44:21:e6:a5
              ipv4:
                enabled: true
                address:
                  - ip: 192.168.111.80
                    prefix-length: 23
                dhcp: false
          dns-resolver:
            config:
              server:
                - 192.168.111.1
          routes:
            config:
              - destination: 0.0.0.0/0
                next-hop-address: 192.168.111.2
                next-hop-interface: eno1
                table-id: 254
      EOF

    1

    This IP address is used to determine which node performs the bootstrapping process as well as running the assisted-service component. You must provide the rendezvous IP address when you do not specify at least one host’s IP address in the networkConfig parameter. If this address is not provided, one IP address is selected from the provided hosts' networkConfig.

    2

    Host configuration is optional. The number of hosts defined must not exceed the total number of hosts defined in the install-config.yaml file, which is the sum of the values of the compute.replicas and controlPlane.replicas parameters.

    3

    The optional hostname parameter overrides the hostname obtained from either the Dynamic Host Configuration Protocol (DHCP) or a reverse DNS lookup. Each host must have a unique hostname supplied by one of these methods.

    4

    The rootDeviceHints parameter enables provisioning of the Red Hat Enterprise Linux CoreOS (RHCOS) image to a particular device. It examines the devices in the order it discovers them, and compares the discovered values with the hint values. It uses the first discovered device that matches the hint value.

    5

    Set this optional parameter to configure the network interface of a host in NMState format.

14. Create the agent image by running the following command:

    $ openshift-install --dir <install_directory> agent create image

    Note

    Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing on the primary disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with a default /etc/multipath.conf configuration.

15. Boot the agent.x86_64.iso image on the bare metal machines.

16. Optional: To know when the bootstrap host (which is the rendezvous host) reboots, run the following command:

    $ ./openshift-install --dir <install_directory> agent wait-for bootstrap-complete \ 1
        --log-level=info 2

    1

    For <install_directory>, specify the path to the directory where the agent ISO was generated.

    2

    To view different installation details, specify warn, debug, or error instead of info.

    Example output

    ...................................................................
    ...................................................................
    INFO Bootstrap configMap status is complete
    INFO cluster bootstrap is complete

    The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

17. To track the progress and verify sucessful installation, run the following command:

    $ openshift-install --dir <install_directory> agent wait-for install-complete 1

    1

    For <install_directory> directory, specify the path to the directory where the agent ISO was generated.

    Example output

    ...................................................................
    ...................................................................
    INFO Cluster is installed
    INFO Install complete!
    INFO To access the cluster as the system:admin user when using 'oc', run
    INFO     export KUBECONFIG=/home/core/installer/auth/kubeconfig
    INFO Access the OpenShift web-console here: https://console-openshift-console.apps.sno-cluster.test.example.com

Procedure

1. Run the following command and collect the output:

   $ ./openshift-install --dir <install_directory> agent wait-for bootstrap-complete --log-level=debug

   Example error message

   ...
   ERROR Bootstrap failed to complete: : bootstrap process timed out: context deadline exceeded

2. If the output from the previous command indicates a failure, or if the bootstrap is not progressing, run the following command on node 0 and collect the output:

   $ ssh core@<node-ip> sudo /usr/local/bin/agent-gather -O > <local_tmp_path>/agent-gather.tar.xz

   Note

   You only need to gather data from node 0, but gathering this data from every node can be helpful.

3. If the bootstrap completes and the cluster nodes reboot, run the following command and collect the output:

   $ ./openshift-install --dir <install_directory> agent wait-for install-complete --log-level=debug

4. If the output from the previous command indicates a failure, perform the following steps:

   1. Export the kubeconfig file to your environment by running the following command:

      $ export KUBECONFIG=<install_directory>/auth/kubeconfig

   2. To gather information for debugging, run the following command:

      $ oc adm must-gather

   3. Create a compressed file from the must-gather directory that was just created in your working directory by running the following command:

      $ tar cvaf must-gather.tar.gz <must_gather_directory>

5. Excluding the /auth subdirectory, attach the installation directory used during the deployment to your support case on the Red Hat Customer Portal.
6. Attach all other data gathered from this procedure to your support case.

Procedure

1. Create an <assets_directory> folder to contain valid install-config.yaml and agent-config.yaml files. This directory is used to store all the assets.

2. To mirror an OpenShift Container Platform image repository, the multicluster engine, and the LSO, create a ImageSetConfiguration.yaml file with the following settings:

   Example ImageSetConfiguration.yaml

     kind: ImageSetConfiguration
     apiVersion: mirror.openshift.io/v1alpha2
     archiveSize: 4 1
     storageConfig: 2
       imageURL: <your-local-registry-dns-name>:<your-local-registry-port>/mirror/oc-mirror-metadata 3
       skipTLS: true
     mirror:
       platform:
         architectures:
           - "amd64"
         channels:
           - name: stable-4.12 4
             type: ocp
       additionalImages:
         - name: registry.redhat.io/ubi8/ubi:latest
       operators:
         - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12 5
           packages: 6
             - name: multicluster-engine 7
             - name: local-storage-operator 8

   1

   Specify the maximum size, in GiB, of each file within the image set.

   2

   Set the back-end location to receive the image set metadata. This location can be a registry or local directory. You must specify storageConfig values unless you are using the Technology Preview OCI feature.

   3

   Set the registry URL for the storage backend.

   4

   Set the channel that contains the OpenShift Container Platform images for the version you are installing.

   5

   Set the Operator catalog that contains the OpenShift Container Platform images that you are installing.

   6

   Specify only certain Operator packages and channels to include in the image set. Remove this field to retrieve all packages in the catalog.

   7

   The multicluster engine packages and channels.

   8

   The LSO packages and channels.

   Note

   This file is required by the oc mirror command when mirroring content.

3. To mirror a specific OpenShift Container Platform image repository, the multicluster engine, and the LSO, run the following command:

   $ oc mirror --dest-skip-tls --config ocp-mce-imageset.yaml docker://<your-local-registry-dns-name>:<your-local-registry-port>

4. Update the registry and certificate in the install-config.yaml file:

   Example imageContentSources.yaml

     imageContentSources:
       - source: "quay.io/openshift-release-dev/ocp-release"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/openshift/release-images"
       - source: "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/openshift/release"
       - source: "registry.redhat.io/ubi8"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/ubi8"
       - source: "registry.redhat.io/multicluster-engine"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/multicluster-engine"
       - source: "registry.redhat.io/rhel8"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/rhel8"
       - source: "registry.redhat.io/redhat"
         mirrors:
           - "<your-local-registry-dns-name>:<your-local-registry-port>/redhat"

   Additionally, ensure your certificate is present in the additionalTrustBundle field of the install-config.yaml.

   Example install-config.yaml

   additionalTrustBundle: |
     -----BEGIN CERTIFICATE-----
     zzzzzzzzzzz
     -----END CERTIFICATE-------

   Important

   The oc mirror command creates a folder called oc-mirror-workspace with several outputs. This includes the imageContentSourcePolicy.yaml file that identifies all the mirrors you need for OpenShift Container Platform and your selected Operators.

5. Generate the cluster manifests by running the following command:

   $ openshift-install agent create cluster-manifests

   This command updates the cluster manifests folder to include a mirror folder that contains your mirror configuration.

Procedure

1. Create a sub-folder named openshift in the <assets_directory> folder. This sub-folder is used to store the extra manifests that will be applied during the installation to further customize the deployed cluster. The <assets_directory> folder contains all the assets including the install-config.yaml and agent-config.yaml files.

   Note

   The installer does not validate extra manifests.

2. For the multicluster engine, create the following manifests and save them in the <assets_directory>/openshift folder:

   Example mce_namespace.yaml

     apiVersion: v1
     kind: Namespace
     metadata:
       labels:
         openshift.io/cluster-monitoring: "true"
       name: multicluster-engine

   Example mce_operatorgroup.yaml

     apiVersion: operators.coreos.com/v1
     kind: OperatorGroup
     metadata:
       name: multicluster-engine-operatorgroup
       namespace: multicluster-engine
     spec:
       targetNamespaces:
       - multicluster-engine

   Example mce_subscription.yaml

     apiVersion: operators.coreos.com/v1alpha1
     kind: Subscription
     metadata:
       name: multicluster-engine
       namespace: multicluster-engine
     spec:
       channel: "stable-2.1"
       name: multicluster-engine
       source: redhat-operators
       sourceNamespace: openshift-marketplace

   Note

   You can install a distributed unit (DU) at scale with the Red Hat Advanced Cluster Management (RHACM) using the assisted installer (AI). These distributed units must be enabled in the hub cluster. The AI service requires persistent volumes (PVs), which are manually created.

3. For the AI service, create the following manifests and save them in the <assets_directory>/openshift folder:

   Example lso_namespace.yaml

     apiVersion: v1
     kind: Namespace
     metadata:
       annotations:
         openshift.io/cluster-monitoring: "true"
       name: openshift-local-storage

   Example lso_operatorgroup.yaml

     apiVersion: operators.coreos.com/v1
     kind: OperatorGroup
     metadata:
       name: local-operator-group
       namespace: openshift-local-storage
     spec:
       targetNamespaces:
         - openshift-local-storage

   Example lso_subscription.yaml

     apiVersion: operators.coreos.com/v1alpha1
     kind: Subscription
     metadata:
       name: local-storage-operator
       namespace: openshift-local-storage
     spec:
       installPlanApproval: Automatic
       name: local-storage-operator
       source: redhat-operators
       sourceNamespace: openshift-marketplace

   Note

   After creating all the manifests, your filesystem must display as follows:

   Example Filesystem

   <assets_directory>
       ├─ install-config.yaml
       ├─ agent-config.yaml
       └─ /openshift
           ├─ mce_namespace.yaml
           ├─ mce_operatorgroup.yaml
           ├─ mce_subscription.yaml
           ├─ lso_namespace.yaml
           ├─ lso_operatorgroup.yaml
           └─ lso_subscription.yaml

4. Create the agent ISO image by running the following command:

   $ openshift-install agent create image --dir <assets_directory>

5. When the image is ready, boot the target machine and wait for the installation to complete.

6. To monitor the installation, run the following command:

   $ openshift-install agent wait-for install-complete --dir <assets_directory>

   Note

   To configure a fully functional hub cluster, you must create the following manifests and manually apply them by running the command $ oc apply -f <manifest-name>. The order of the manifest creation is important and where required, the waiting condition is displayed.

7. For the PVs that are required by the AI service, create the following manifests:

     apiVersion: local.storage.openshift.io/v1
     kind: LocalVolume
     metadata:
      name: assisted-service
      namespace: openshift-local-storage
     spec:
      logLevel: Normal
      managementState: Managed
      storageClassDevices:
        - devicePaths:
            - /dev/vda
            - /dev/vdb
          storageClassName: assisted-service
          volumeMode: Filesystem

8. Use the following command to wait for the availability of the PVs, before applying the subsequent manifests:

   $ oc wait localvolume -n openshift-local-storage assisted-service --for condition=Available --timeout 10m

   Note

   The `devicePath` is an example and may vary depending on the actual hardware configuration used.

9. Create a manifest for a multicluster engine instance.

   Example MultiClusterEngine.yaml

     apiVersion: multicluster.openshift.io/v1
     kind: MultiClusterEngine
     metadata:
       name: multiclusterengine
     spec: {}

10. Create a manifest to enable the AI service.

    Example agentserviceconfig.yaml

      apiVersion: agent-install.openshift.io/v1beta1
      kind: AgentServiceConfig
      metadata:
        name: agent
        namespace: assisted-installer
      spec:
       databaseStorage:
        storageClassName: assisted-service
        accessModes:
        - ReadWriteOnce
        resources:
         requests:
          storage: 10Gi
       filesystemStorage:
        storageClassName: assisted-service
        accessModes:
        - ReadWriteOnce
        resources:
         requests:
          storage: 10Gi

11. Create a manifest to deploy subsequently spoke clusters.

    Example clusterimageset.yaml

      apiVersion: hive.openshift.io/v1
      kind: ClusterImageSet
      metadata:
        name: "4.12"
      spec:
        releaseImage: quay.io/openshift-release-dev/ocp-release:4.12.0-x86_64

12. Create a manifest to import the agent installed cluster (that hosts the multicluster engine and the Assisted Service) as the hub cluster.

    Example autoimport.yaml

      apiVersion: cluster.open-cluster-management.io/v1
      kind: ManagedCluster
      metadata:
       labels:
         local-cluster: "true"
         cloud: auto-detect
         vendor: auto-detect
       name: local-cluster
      spec:
       hubAcceptsClient: true

13. Wait for the managed cluster to be created.

    $ oc wait -n multicluster-engine managedclusters local-cluster --for condition=ManagedClusterJoined=True --timeout 10m