Chapter 3. Deploying OpenShift sandboxed containers workloads using peer pods

You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI (oc). Before installing the OpenShift sandboxed containers Operator, you must prepare your OpenShift Container Platform cluster.

3.1. Prerequisites

Before you install OpenShift sandboxed containers and enable peer pods, you must meet the following requirements:

  • You have OpenShift Container Platform 4.15 installed on AWS or Azure.
  • For installations on IBM Z or IBM® LinuxONE, you must have OpenShift Container Platform 4.14 or later installed.

    Important

    Deploying OpenShift sandboxed containers workloads on IBM Z using peer pods is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

    For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

3.1.1. About peer pod resource requirements in OpenShift sandboxed containers

A peer pod uses resources in two locations:

  • The worker node. The worker node stores metadata, Kata shim resources (containerd-shim-kata-v2), remote-hypervisor resources (cloud-api-adaptor), and the tunnel setup between the worker nodes and the peer-pod virtual machine (VM).
  • The cloud instance. This is the actual peer-pod VM running in the cloud.

The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote) definition used for creating peer pods.

The total number of peer-pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit attribute in the peerpodConfig custom resource (CR). The peerpodConfig CR, named peerpodconfig-openshift, is created when you create the kataConfig CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator namespace.

The following peerpodConfig CR example displays the default spec values:

apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
  name: peerpodconfig-openshift
  namespace: openshift-sandboxed-containers-operator
spec:
  cloudSecretName: peer-pods-secret
  configMapName: peer-pods-cm
  limit: "10" 1
  nodeSelector:
    node-role.kubernetes.io/kata-oc: ""
1
The default limit is 10 VMs per node.

The extended resource is named kata.peerpods.io/vm, and enables the Kubernetes scheduler to handle capacity tracking and accounting.

You can edit the limit per node based on the requirements for your environment. See Modifying the VM limit per node in peer pods for more information.

A mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer-pod is only scheduled when resources are available.

The mutating webhook modifies a Kubernetes pod as follows:

  • The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIME_CLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIME_CLASS, the webhook exits without modifying the pod.
  • If the RuntimeClassName values match, the webhook makes the following changes to the pod spec:

    1. The webhook removes every resource specification from the resources field of all containers and init containers in the pod.
    2. The webhook adds the extended resource (kata.peerpods.io/vm) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes.
Note

The mutating webhook excludes specific system namespaces in OpenShift Container Platform from mutation. If a peer-pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.

As a best practice, define a cluster-wide policy to only allow peer-pod creation in specific namespaces.

3.1.1.1. Modifying the peer pod VM limit per node

You can change the limit of peer pod VMs per node by editing the peerpodConfig custom resource (CR).

Procedure

  1. Check the current limit by running the following command:

    $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
    -o jsonpath='{.spec.limit}{"\n"}'
  2. Modify the limit attribute of the peerpodConfig CR by running the following command:

    $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
    --type merge --patch '{"spec":{"limit":"<value>"}}' 1
    1
    Replace <value> with the limit you want to define.

3.1.2. Prerequisites for peer-pods using AWS

If you are using AWS to create peer pods, you must ensure the following requirements:

  • Your OpenShift Container Platform cluster must be installed on AWS, with at least one worker node.
  • You have access to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY credentials. These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster.
  • You must have the AWS CLI tool installed and configured.
  • You must enable internal cluster communication on ports 15150 and 9000.

    You can enable these ports in either the AWS web console or using the CLI.

3.1.2.1. Enable ports 15150 and 9000 for AWS

Procedure

  1. Retrieve the Instance ID:

    $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
  2. Retrieve the AWS region:

    $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
  3. Retrieve the security groups:

    $ SG=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region ${AWS_REGION})
  4. Authorize the peer-pods shim and to access kata-agent communication. Run the following command:

    $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 15150 --source-group ${SG} --region ${AWS_REGION}
  5. Set up the peer-pods tunnel. Run the following command:

    $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 9000 --source-group ${SG} --region ${AWS_REGION}

The ports are now enabled.

3.1.3. Prerequisites for peer-pods using Azure

If you are using Microsoft Azure to create peer pods, you must ensure the following requirements:

  • Your OpenShift Container Platform cluster must be installed on Azure, with at least one worker node.
  • You have access to the following credentials and subscription details:

    • AZURE_SUBSCRIPTION_ID
    • AZURE_CLIENT_ID
    • AZURE_CLIENT_SECRET
    • AZURE_TENANT_ID

    These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster.

  • You must have the Azure CLI tool installed and configured.

3.1.4. Prerequisites for peer pods using IBM Z or IBM(R) LinuxONE with RHEL KVM

You can install the OpenShift sandboxed containers Operator on IBM Z using the OpenShift CLI (oc).

Note

While this document refers only to IBM Z, all information in it also applies to IBM® LinuxONE.

If you are using IBM Z to create peer pods, you must ensure the following requirements:

  • Your OpenShift Container Platform cluster must be installed on IBM Z, with at least one compute node.
  • You have installed the following tools on the IBM Z KVM host

    • libvirt
    • podman
    • git
    • tar
    • virt-customize
  • You must have the oc CLI tool installed and configured.

If you want to test peer pods in a non-production environment, the following requirements must be met:

  • Your OpenShift Container Platform cluster must be version 4.14 or later.
  • You have a multi-node OpenShift Container Platform cluster with three control nodes and two compute nodes set up.
  • Cluster nodes and peer pods must run on a single IBM Z KVM host logical partition (LPAR).
  • Your cluster setup ensures that peer pod virtual machines (VMs) and cluster nodes are part of the same subnet, allowing connection between node and peer pod VM.

3.2. Deploying OpenShift sandboxed containers workloads using peer pods with the web console

You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create a secret object, the VM image, and a peer-pod ConfigMap. The secret object and ConfigMap are unique, depending on your cloud provider. Finally, you must create the KataConfig custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata-remote as the runtimeClassName to the workload YAML file.

3.2.1. Installing the OpenShift sandboxed containers Operator using the web console

You can install the OpenShift sandboxed containers Operator from the OpenShift Container Platform web console.

Prerequisites

  • You have OpenShift Container Platform 4.15 installed.
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsOperatorHub.
  2. In the Filter by keyword field, type OpenShift sandboxed containers.
  3. Select the OpenShift sandboxed containers tile.
  4. Read the information about the Operator and click Install.
  5. On the Install Operator page:

    1. Select stable from the list of available Update Channel options.
    2. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

      Note

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

    3. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
  6. Click Install.

The OpenShift sandboxed containers Operator is now installed on your cluster.

Verification

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.

3.2.2. Configuring peer-pod parameters for AWS using the web console

You must create a secret object and a ConfigMap to deploy OpenShift sandboxed containers using peer pods on AWS.

After creating the secret object, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.2.2.1. Creating a secret object for AWS using the web console

Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for AWS using the CLI.

In addition, in the AWS web console, you must generate and save the following values:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AWS_ACCESS_KEY_ID: "<enter value>" 1
      AWS_SECRET_ACCESS_KEY: "<enter value>" 2
      AWS_REGION: "<enter value>" 3
      AWS_SUBNET_ID: "<enter value>" 4
      AWS_VPC_ID: "<enter value>" 5
      AWS_SG_IDS: "<enter value>" 6
    1
    Enter the AWS_ACCESS_KEY_ID value you prepared before you began.
    2
    Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.
    3
    Enter the AWS_REGION value you retrieved.
    4
    Enter the AWS_SUBNET_ID value you retrieved.
    5
    Enter the AWS_VPC_ID value you retrieved.
    6
    Enter the AWS_SG_IDS value you retrieved.
  5. Click Create.

The secret object is created. You can see it listed under WorkloadsSecrets.

3.2.2.2. Creating a peer-pod ConfigMap for AWS using the web console

You can create a peer-pod ConfigMap for AWS using the web console.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "aws"
      VXLAN_PORT: "9000"
      PODVM_INSTANCE_TYPE: "t3.medium" 1
      PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
      PROXY_TIMEOUT: "5m"
    1
    Defines the default instance type that is used when a type is not defined in the workload.
    2
    Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
  5. Click Create.

The ConfigMap object is created. You can see it listed under WorkloadsConfigMaps.

Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on AWS.

3.2.3. Configuring peer-pod parameters for Azure using the web console

You must create a secret object and a ConfigMap in order to deploy OpenShift sandboxed containers using peer pods on Microsoft Azure.

After creating the secret object, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.2.3.1. Creating a secret object for Azure using the web console

Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for Azure, you must set specific environment values. You can retrieve these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for Azure using the CLI.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AZURE_CLIENT_ID: "<enter value>" 1
      AZURE_CLIENT_SECRET: "<enter value>" 2
      AZURE_TENANT_ID: "<enter value>" 3
      AZURE_SUBSCRIPTION_ID: "<enter value>" 4
      AZURE_REGION: "<enter value>" 5
      AZURE_RESOURCE_GROUP: "<enter value>" 6
    1
    Enter the AZURE_CLIENT_ID value you generated in the RBAC file.
    2
    Enter the AZURE_CLIENT_SECRET value you generated in the RBAC file.
    3
    Enter the AZURE_TENANT_ID value you generated in the RBAC file.
    4
    Enter the AZURE_SUBSCRIPTION_ID value you retrieved.
    5
    Enter the AZURE_REGION value you retrieved.
    6
    Enter the AZURE_RESOURCE_GROUP value you retrieved.
  5. Click Create.

The secret object is created. You can see it listed under WorkloadsSecrets.

3.2.3.2. Creating a peer-pod ConfigMap for Azure using the web console

You can create a peer-pod ConfigMap for Azure using the web console.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "azure"
      VXLAN_PORT: "9000"
      AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
      AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5" 2
      AZURE_SUBNET_ID: "<enter value>" 3
      AZURE_NSG_ID: "<enter value>" 4
      PROXY_TIMEOUT: "5m"
      DISABLECVM: "true"
    1
    Defines the default instance size that is used when an instance is not defined in the workload.
    2
    List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
    3
    Enter the AZURE_SUBNET_ID value that you retrieved.
    4
    Enter the AZURE_NSG_ID value that you retrieved.
  5. Click Create.

The ConfigMap` object is created. You can see it listed under WorkloadsConfigMaps.

3.2.3.3. Creating an SSH key secret object for Azure using the web console

You must create an SSH key secret object to use peer pods with Azure. If you do not already have an SSH key to create the object, you must generate one using the CLI. Fo more information, see

Procedure

  1. From the Administrator perspective in the web console, navigate to WorkloadsSecrets.
  2. In the Secrets window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
  3. Click Create and select Key/value secret from the list.
  4. In the Secret name field, enter ssh-key-secret.
  5. In the Key field, enter id_rsa.pub.
  6. In the Value field, paste your public SSH key.
  7. Click Create.

The SSH key secret object is created. You can see it listed under WorkloadsSecrets.

Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on Azure.

3.2.4. Creating the KataConfig custom resource in the web console

You must create one KataConfig custom resource (CR) to enable installing kata-remote as a RuntimeClass on your cluster nodes.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Note

Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote as a RuntimeClass only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig CR when you create it.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. In the KataConfig tab, click Create KataConfig.
  4. In the Create KataConfig page, enter the following details:

    • Name: Enter a name for the KataConfig resource. By default, the name is defined as example-kataconfig.
    • Labels (Optional): Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
    • checkNodeEligibility (Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
    • enablePeerPods (For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment.
    • kataConfigPoolSelector: By default, kata-remote is installed as a RuntimeClass on all nodes. If you want to install kata-remote as a RuntimeClass only on selected nodes, you must add a matchExpression:

      1. Expand the kataConfigPoolSelector area.
      2. In the kataConfigPoolSelector, expand matchExpressions. This is a list of label selector requirements.
      3. Click Add matchExpressions.
      4. In the key field, add the label key the selector applies to.
      5. In the operator field, add the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
      6. Expand the values area, and then click Add value.
      7. In the Value field, enter true or false for key label value.
    • logLevel: Define the level of log data retrieved for nodes running kata-remote as a RuntimeClass. For more information, see "Collecting OpenShift sandboxed containers data".
  5. Click Create.

The new KataConfig CR is created and begins to install kata-remote as a RuntimeClass on the worker nodes. Wait for the installation to complete and the worker nodes to reboot before continuing to the next step.

Note

When you run the CR, the VM image is created. The image creation is done through the cloud provider and it may use additional resources.

Important

OpenShift sandboxed containers installs kata-remote only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  1. In the KataConfig tab, select the new KataConfig CR.
  2. In the KataConfig page, select the YAML tab.
  3. Monitor the status field.

    A message appears each time there is an update. Click Reload to view the updated KataConfig CR.

    The status field has conditions and kataNodes subfields. The kataNodes subfield is a collection of node lists, each of which lists nodes in a particular state of kata installation.

    When all workers under kataNodes are listed as installed, and the condition InProgress is False without specifying a reason, this indicates that kata is installed on the cluster. For more information, see "Installation and uninstall transitions".

3.2.5. Deploying a workload with peer pods in a sandboxed container using the web console

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload using peer pods in a sandboxed container, you must manually add kata-remote as the runtimeClassName to the workload YAML file.

You must also define whether the workload should be deployed using a default instance size or type which you previously defined in the ConfigMap by adding an annotation to the YAML file. The use of an instance size or an instance type depends on your cloud provider. If you do not want to define the instance size or type manually, you must add an annotation to define the use of an automatic instance size or type, based on the memory available.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a secret object and peer-pod ConfigMap unique to your cloud provider.
  • You have created a KataConfig custom resource (CR).

Procedure

  1. From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
  2. In the workload page, click to create the workload.
  3. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata-remote.
  4. In the YAML file for the workload, add an annotation to define whether to use a default instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.

    • For a specific instance size type, add the following annotation:

      io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>

      Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the ConfigMap for peer pods. Choose from one of those.

    • For an automatic instance size or type, add the following annotations:

      io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
      io.katacontainers.config.hypervisor.default_memory: <memory>

      Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.

    Example for Pod object

    apiVersion: v1
    kind: Pod
    metadata:
      name: hello-openshift
      labels:
        app: hello-openshift
      annotations:
        io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1
    spec:
      runtimeClassName: kata-remote
      containers:
        - name: hello-openshift
          image: quay.io/openshift/origin-hello-openshift
          ports:
            - containerPort: 8888
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

    1
    This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.
  5. Click Save.

OpenShift Container Platform creates the workload and begins scheduling it.

3.3. Deploying OpenShift sandboxed containers workloads using peer pods with the CLI

You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata-remote as the runtimeClassName to the workload YAML file.

3.3.1. Installing the OpenShift sandboxed containers Operator using the CLI

You can install the OpenShift sandboxed containers Operator using the OpenShift Container Platform CLI.

Prerequisites

  • You have OpenShift Container Platform 4.15 installed on your cluster.
  • For installations on IBM Z or IBM® LinuxONE, you must have OpenShift Container Platform 4.14 or later installed.

    Important

    Deploying OpenShift sandboxed containers workloads on IBM Z using peer pods is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

    For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have subscribed to the OpenShift sandboxed containers catalog.

    Note

    Subscribing to the OpenShift sandboxed containers catalog provides openshift-sandboxed-containers-operator namespace access to the OpenShift sandboxed containers Operator.

Procedure

  1. Create the Namespace object for the OpenShift sandboxed containers Operator.

    1. Create a Namespace object YAML file that contains the following manifest:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-sandboxed-containers-operator
    2. Create the Namespace object:

      $ oc create -f Namespace.yaml
  2. Create the OperatorGroup object for the OpenShift sandboxed containers Operator.

    1. Create an OperatorGroup object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        targetNamespaces:
        - openshift-sandboxed-containers-operator
    2. Create the OperatorGroup object:

      $ oc create -f OperatorGroup.yaml
  3. Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator.

    1. Create a Subscription object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: sandboxed-containers-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: sandboxed-containers-operator.v1.5.2
    2. Create the Subscription object:

      $ oc create -f Subscription.yaml

The OpenShift sandboxed containers Operator is now installed on your cluster.

Note

All the object file names listed above are suggestions. You can create the object YAML files using other names.

Verification

  • Ensure that the Operator is correctly installed:

    $ oc get csv -n openshift-sandboxed-containers-operator

    Example output

    NAME                             DISPLAY                                  VERSION  REPLACES     PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.5.2    1.5.1        Succeeded

3.3.2. Setting up peer pods for AWS using the CLI

To set up peer pods for use on AWS, you must create a secret object, an AWS image VM (AMI), and a peer-pod ConfigMap.

After setting up peer pods for AWS, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.3.2.1. Creating a secret object for AWS using the CLI

Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

You can generate these values in the AWS console.

Procedure

  1. Collect the necessary parameter values for the secret object. Make sure to write down each value.

    1. Retrieve the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This value is not needed for the secret object itself, but is used to retrieve other values for the secret object.

    2. Retrieve the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""
    3. Retrieve the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""
    4. Retrieve the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""
    5. Retrieve the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region ${AWS_REGION} --output text)
      && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AWS_ACCESS_KEY_ID: "<enter value>" 1
      AWS_SECRET_ACCESS_KEY: "<enter value>" 2
      AWS_REGION: "<enter value>" 3
      AWS_SUBNET_ID: "<enter value>" 4
      AWS_VPC_ID: "<enter value>" 5
      AWS_SG_IDS: "<enter value>" 6
    1
    Enter the AWS_ACCESS_KEY_ID value you prepared before you began.
    2
    Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.
    3
    Enter the AWS_REGION value you retrieved.
    4
    Enter the AWS_SUBNET_ID value you retrieved.
    5
    Enter the AWS_VPC_ID value you retrieved.
    6
    Enter the AWS_SG_IDS value you retrieved.
  3. Apply the secret object:

    $ oc apply -f peer-pods-secret.yaml

The secret object is applied.

3.3.2.2. Creating a peer-pod ConfigMap for AWS using the CLI

When creating a ConfigMap for AWS, you must set the AMI ID. You can retrieve this value before you create the ConfigMap.

Procedure

  1. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "aws"
      VXLAN_PORT: "9000"
      PODVM_INSTANCE_TYPE: "t3.medium" 1
      PODVM_INSTANCE_TYPES: "t2.small,t2.medium,t3.large" 2
      PROXY_TIMEOUT: "5m"
    1
    Defines the default instance type that is used when a type is not defined in the workload.
    2
    Lists all of the instance types you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
  2. Apply the ConfigMap:

    $ oc apply -f peer-pods-cm.yaml

The ConfigMap is applied. Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on AWS.

3.3.3. Setting up peer pods for Azure using the CLI

To set up peer pods for use on Microsoft Azure, you must create a secret object, an Azure image VM, a peer-pod ConfigMap, and an SSH key secret object.

After setting up peer pods for Azure, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You must have the Azure CLI tool installed and configured.

3.3.3.1. Creating a secret object for Azure using the CLI

Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for Azure, you must set specific environment values. You can retrieve these values before creating the secret object. In addition, you must create a role-based access control (RBAC) file. This file generates the following values:

  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET
  • AZURE_TENANT_ID

Procedure

  1. Retrieve the Azure subscription ID:

    $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
  2. Generate the RBAC content. This generates the client ID, client secret, and the tenant ID:

    $ az ad sp create-for-rbac --role Contributor --scopes /subscriptions/$AZURE_SUBSCRIPTION_ID --query "{ client_id: appId, client_secret: password, tenant_id: tenant }

    The following output appears:

    {
      "client_id": `AZURE_CLIENT_ID`,
      "client_secret": `AZURE_CLIENT_SECRET`,
      "tenant_id": `AZURE_TENANT_ID`
    }
  3. Save the values in the RBAC output to use in the secret object.
  4. Collect the additional parameter values for the secret object. Make sure to write down each value.

    1. Retrieve the resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
    2. Retrieve the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
  5. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AZURE_CLIENT_ID: "<enter value>" 1
      AZURE_CLIENT_SECRET: "<enter value>" 2
      AZURE_TENANT_ID: "<enter value>" 3
      AZURE_SUBSCRIPTION_ID: "<enter value>" 4
      AZURE_REGION: "<enter value>" 5
      AZURE_RESOURCE_GROUP: "<enter value>" 6
    1
    Enter the AZURE_CLIENT_ID value you generated in the RBAC file.
    2
    Enter the AZURE_CLIENT_SECRET value you generated in the RBAC file.
    3
    Enter the AZURE_TENANT_ID value you generated in the RBAC file.
    4
    Enter the AZURE_SUBSCRIPTION_ID value you retrieved.
    5
    Enter the AZURE_REGION value you retrieved.
    6
    Enter the AZURE_RESOURCE_GROUP value you retrieved.
  6. Apply the secret object:

    $ oc apply -f peer-pods-secret.yaml

The secret object is applied.

3.3.3.2. Creating a peer-pod ConfigMap for Azure using the CLI

When creating a ConfigMap for Azure, you must set specific configuration values. You can retrieve these values before you create the ConfigMap.

Procedure

  1. Collect the configuration values for the Azure peer-pod ConfigMap. Make sure to write down each value.

    1. Retrieve the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is not needed for the ConfigMap, but is used to retrieve the Azure subnet ID.

    2. Retrieve the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
    3. Retrieve the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "azure"
      VXLAN_PORT: "9000"
      AZURE_INSTANCE_SIZE: "Standard_B2als_v2" 1
      AZURE_INSTANCE_SIZES: "Standard_DC2as_v5,Standard_DC4as_v5,Standard_DC8as_v5" 2
      AZURE_SUBNET_ID: "<enter value>" 3
      AZURE_NSG_ID: "<enter value>" 4
      PROXY_TIMEOUT: "5m"
      DISABLECVM: "true"
    1
    Defines the default instance size that is used when an instance is not defined in the workload.
    2
    List all of the instance sizes you can specify when creating the pod. This allows you to define smaller instances for workloads that need less memory and CPU, or larger instances for larger workloads.
    3
    Enter the AZURE_SUBNET_ID value that you retrieved.
    4
    Enter the AZURE_NSG_ID value that you retrieved.
  3. Apply the ConfigMap:

    $ oc apply -f peer-pods-cm.yaml

The ConfigMap is deployed.

3.3.3.3. Creating an SSH key secret object for Azure using the CLI

You must generate an SSH key and create an SSH key secret object to use peer pods with Azure.

Procedure

  1. Generate the SSH key:

    $ ssh-keygen -f ./id_rsa -N ""
  2. Create the SSH secret object:

    $ oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa

The SSH key is created and the SSH key secret object is created. Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on Azure.

3.3.4. Setting up peer pods for IBM Z using the CLI

To set up peer pods for use on an OpenShift Container Platform cluster running on IBM Z, you must create a secret object, a RHEL KVM image VM, and a peer-pod ConfigMap, which hold the credentials that are required for the communication between libvirt and the KVM host.

After setting up peer pods for IBM Z, you must create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed OpenShift Container Platform 4.14 or later on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have libvirt installed on your KVM host and you have administrator privileges.

3.3.4.1. Setting up libvirt on the KVM host

You must set up libvirt on the KVM host. Peer pods on IBM Z use the libvirt provider of the cloud API Adaptor to create and manage virtual machines.

Procedure

  1. Log in to IBM Z KVM host and set the shell variables to be used by libvirt for storage management.

    1. Set the name of the libvirt pool by running the following command:

      $ export LIBVIRT_POOL=<name_of_libvirt_pool_to_create>
    2. Set the name of the libvirt pool by running the following command:

      $ export LIBVIRT_VOL_NAME=<name_of_libvirt_volume_to_create>
    3. Set the path of the default storage pool location, by running the following command:

      $ export LIBVIRT_POOL_DIRECTORY=<name_of_target_directory> 1
      1
      To ensure libvirt has read and write access permissions, use a subdirectory of libvirt’s storage directory. The default is /var/lib/libvirt/images/.
  2. Create a libvirt pool by running the following command:

    $ virsh pool-define-as $LIBVIRT_POOL --type dir --target "$LIBVIRT_POOL_DIRECTORY"
  3. Start the libvirt pool by running the following command:

    $ virsh pool-start $LIBVIRT_POOL
  4. Create a libvirt volume for the pool by running the following command:

    $ virsh -c qemu:///system \
      vol-create-as --pool $LIBVIRT_POOL \
      --name $LIBVIRT_VOL_NAME \
      --capacity 20G \
      --allocation 2G \
      --prealloc-metadata \
      --format qcow2

3.3.4.2. Creating a peer-pod VM image for IBM Z

To run OpenShift sandboxed containers using peer pods on IBM Z, you must first create a KVM guest from which the libvirt provider starts the peer-pod VM.

Once the image is created, upload the image to the volume that you created in the previous step.

Prerequisites

  • IBM z15 or later, or IBM® LinuxONE III or later.
  • At least one LPAR running on RHEL 9 or later with KVM.

Procedure

  1. Set the RHEL ORG_ID and ACTIVATION_KEY shell variables for your system.

    1. When using a subscribed RHEL system, set the shell environment variables to the files holding the organization ID and the activation key for Red Hat Subscription Management (RHSM) by running the following commands:

      $ export ORG_ID=$(cat ~/.rh_subscription/orgid)
      
      $ export ACTIVATION_KEY=$(cat ~/.rh_subscription/activation_key)
    2. When using an unsubscribed RHEL system, set the appropriate subscription values by running the following commands:

      $ export ORG_ID=<RHEL_ORGID_VALUE>
      
      $ export ACTIVATION_KEY=<RHEL_ACVTIVATION_KEY>
  2. Log in to your IBM Z system and perform the following steps:

    1. Download an s390x RHEL KVM guest image from the Red Hat Customer Portal to your libvirt storage directory to grant libvirt correct access. The default directory is /var/lib/libvirt/images. The image is used to generate the peer pod VM image, which will include the relevant binaries.
    2. Set the IMAGE_URL shell environment variable for the downloaded image by running the following command:

      $ export IMAGE_URL=<location_of_downloaded_KVM_guest_image> 1
      1
      Enter the path of the KVM guest image that you downloaded in the previous step.
    3. Register the guest KVM image by running the following command:

      $ export REGISTER_CMD="subscription-manager register --org=${ORG_ID} \
        --activationkey=${ACTIVATION_KEY}"
    4. Customize the guest KVM image by running the following command:

      $ virt-customize -v -x -a ${IMAGE_URL} --run-command "${REGISTER_CMD}"
    5. Set the checksum of the image by running the following command:

      $ export IMAGE_CHECKSUM=$(sha256sum ${IMAGE_URL} | awk '{ print $1 }')
3.3.4.2.1. Building the peer-pod VM QCOW2 image

To run OpenShift sandboxed containers using peer pods on IBM Z, you must build a peer-pod VM QCOW2 image.

Procedure

  1. Clone the cloud-api-adaptor repository onto your build workstation by running the following command:

    $ git clone --single-branch https://github.com/confidential-containers/cloud-api-adaptor.git
  2. Change into the podvm directory by running the following command:

    $ cd cloud-api-adaptor && git checkout 8577093
  3. Create a builder image from which the final QCOW2 image is generated.

    1. When using a subscribed RHEL system, run the following command:

      $ podman build -t podvm_builder_rhel_s390x \
        --build-arg ARCH="s390x" \
        --build-arg GO_VERSION="1.21.3" \
        --build-arg PROTOC_VERSION="25.1" \
        --build-arg PACKER_VERSION="v1.9.4" \
        --build-arg RUST_VERSION="1.72.0" \
        --build-arg YQ_VERSION="v4.35.1" \
        --build-arg YQ_CHECKSUM="sha256:4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
        -f podvm/Dockerfile.podvm_builder.rhel .
    2. When using an unsubscribed RHEL system, run the following command:

      $ podman build -t podvm_builder_rhel_s390x \
        --build-arg ORG_ID=$ORG_ID \
        --build-arg ACTIVATION_KEY=$ACTIVATION_KEY \
        --build-arg ARCH="s390x" \
        --build-arg GO_VERSION="1.21.3" \
        --build-arg PROTOC_VERSION="25.1" \
        --build-arg PACKER_VERSION="v1.9.4" \
        --build-arg RUST_VERSION="1.72.0" \
        --build-arg YQ_VERSION="v4.35.1" \
        --build-arg YQ_CHECKSUM="sha256:4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
        -f podvm/Dockerfile.podvm_builder.rhel .
  4. Generate an intermediate image package with the required binaries for running peer pods by running the following command:

    $ podman build -t podvm_binaries_rhel_s390x \
      --build-arg BUILDER_IMG="podvm_builder_rhel_s390x:latest" \
      --build-arg ARCH=s390x \
      -f podvm/Dockerfile.podvm_binaries.rhel .
    Note

    This process is expected to take significant time.

  5. Extract the binaries and build the peer-pod QCOW2 image by running the following command:

    $ podman build -t podvm_rhel_s390x \
      --build-arg ARCH=s390x \
      --build-arg CLOUD_PROVIDER=libvirt \
      --build-arg BUILDER_IMG="localhost/podvm_builder_rhel_s390x:latest" \
      --build-arg BINARIES_IMG="localhost/podvm_binaries_rhel_s390x:latest" \
      -v ${IMAGE_URL}:/tmp/rhel.qcow2:Z \
      --build-arg IMAGE_URL="/tmp/rhel.qcow2" \
      --build-arg IMAGE_CHECKSUM=${IMAGE_CHECKSUM} \
      -f podvm/Dockerfile.podvm.rhel .
  6. Extract the peer pod QCOW2 image to a directory of your choice by running the following commands:

    $ export IMAGE_OUTPUT_DIR=<image_output_directory> 1
    
    $ mkdir -p $IMAGE_OUTPUT_DIR
    
    $ podman save podvm_rhel_s390x | tar -xO --no-wildcards-match-slash '*.tar' | tar -x -C ${IMAGE_OUTPUT_DIR}
    1
    Enter the image_output_directory where to extract the final QCOW image.
  7. Upload the peer pod QCOW2 image to your libvirt volume:

    $ virsh -c qemu:///system vol-upload \
      --vol $ LIBVIRT_VOL_NAME \
      $IMAGE_OUTPUT_DIR/podvm-*.qcow2 \
      --pool $LIBVIRT_POOL --sparse

3.3.4.3. Creating a RHEL secret for peer-pod credentials

When creating a secret object for IBM Z, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:

  • LIBVIRT_POOL
  • LIBVIRT_VOL_NAME
  • LIBVIRT_URI

LIBVIRT_URI is the default gateway IP address of your libvirt network. Check your libvirt network setup to obtain this value.

Note

If your libvirt installation is using the default bridge virtual network, you can obtain the LIBVIRT_URI by running the following commands:

$ virtint=$(bridge_line=$(virsh net-info default | grep Bridge);  echo "${bridge_line//Bridge:/}" | tr -d [:blank:])

$ LIBVIRT_URI=$( ip -4 addr show $virtint | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

Procedure

  1. Create a YAML file peer-pods-secret.yaml with the following manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      CLOUD_PROVIDER: "libvirt" 1
      LIBVIRT_URI: "<libvirt_gateway_uri>" 2
      LIBVIRT_POOL: "<libvirt_pool>" 3
      LIBVIRT_VOL_NAME: "<libvirt_volume>" 4
    1
    Enter libvirt as the cloud provider.
    2
    Enter the libvirt_gateway_uri value you retrieved.
    3
    Enter the libvirt_pool value you retrieved.
    4
    Enter the libvirt_volume value you retrieved.
  2. Create the secret object:

    $ oc apply -f peer-pods-secret.yaml

The secret object is applied.

3.3.4.4. Creating a peer-pod ConfigMap for IBM Z using the CLI

When creating a ConfigMap for IBM Z, you must use the libvirt provider.

Procedure

  1. Create a YAML file peer-pods-cm.yaml with the following manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "libvirt"
      PROXY_TIMEOUT: "15m"
  2. Apply the ConfigMap:

    $ oc apply -f peer-pods-cm.yaml

The ConfigMap is applied.

3.3.4.5. Creating an SSH key secret object for IBM Z using the CLI

You must generate an SSH key pair and create an SSH key secret object to use peer pods with IBM Z.

Procedure

  1. Generate the SSH key:

    $ ssh-keygen -f ./id_rsa -N ""
  2. Copy the SSH public key to your KVM host:

    $ ssh-copy-id -i ./id_rsa.pub <KVM_HOST_ADDRESS> 1
    1
    Enter the IP address of your KVM host.
  3. Create the secret object:

    $ oc create secret generic ssh-key-secret \
        -n openshift-sandboxed-containers-operator \
        --from-file=id_rsa.pub=./id_rsa.pub \
        --from-file=id_rsa=./id_rsa
  4. Delete the SSH keys:

    $ shred –remove id_rsa.pub id_rsa

The secret object is created. Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on IBM Z.

3.3.5. Creating the KataConfig custom resource using the CLI

You must create one KataConfig custom resource (CR) to install kata-remote as a RuntimeClass on your nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following:

  • Install the needed RHCOS extensions, such as QEMU and kata-containers, on your RHCOS node.
  • Ensure that the CRI-O runtime is configured with the correct runtime handlers.
  • Create a RuntimeClass CR named kata-remote with a default configuration. This enables users to configure workloads to use kata-remote as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime.
Note

Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote as a RuntimeClass only on specific nodes, you can add labels to those nodes, and then define the label in the KataConfig CR when you create it.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger OpenShift Container Platform deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Procedure

  1. Create a YAML file with the following manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      enablePeerPods: true
      logLevel: info
  2. (Optional) If you want to install kata-remote as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      enablePeerPods: true
      logLevel: info
      kataConfigPoolSelector:
        matchLabels:
          <label_key>: '<label_value>' 1
    1
    Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported.
  3. Create the KataConfig resource:

    $ oc create -f cluster-kataconfig.yaml

The new KataConfig CR is created and begins to install kata-remote as a RuntimeClass on the worker nodes. Wait for the kata-remote installation to complete and the worker nodes to reboot before continuing to the next step.

Note

When you run the CR, the VM image is created. The image creation is done through the cloud provider and it may use additional resources.

Important

OpenShift sandboxed containers installs kata-remote only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  • Monitor the installation progress:

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

    When all workers under kataNodes are listed as installed, and the condition InProgress is False without specifying a reason, this indicates that kata is installed on the cluster. For more information, see "Installation and uninstall transitions".

3.3.6. Deploying a workload with peer pods in a sandboxed container using the CLI

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload using peer pods in a sandboxed container, you must add kata-remote as the runtimeClassName to the workload YAML file.

You must also define whether the workload should be deployed using a default instance size or type which you previously defined in the ConfigMap by adding an annotation to the YAML file. The use of an instance size or an instance type depends on your cloud provider. If you do not want to define the instance size or type manually, you must add an annotation to define the use of an automatic instance size or type, based on the memory available.

Prerequisites

  • You have installed OpenShift Container Platform 4.15 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a secret object and peer-pod ConfigMap unique to your cloud provider.
  • You have created a KataConfig custom resource (CR).

Procedure

  1. Add runtimeClassName: kata-remote to any pod-templated object:

    • Pod objects
    • ReplicaSet objects
    • ReplicationController objects
    • StatefulSet objects
    • Deployment objects
    • DeploymentConfig objects
  2. Add an annotation to the pod-templated object, defining whether to use a specific instance size or type, or an automatic instance size or type. Instance size is used for specific cloud providers, while instance type is used for other cloud providers.

    • For a specific instance size or type, add the following annotation:

      io.katacontainers.config.hypervisor.machine_type: <instance type/instance size>

      Define which instance size or type the workload should use. You pre-defined these default sizes or types previously when creating the ConfigMap for peer pods. Choose from one of those.

    • For an automatic instance size or type, add the following annotations:

      io.katacontainers.config.hypervisor.default_vcpus: <vcpus>
      io.katacontainers.config.hypervisor.default_memory: <memory>

      Define the amount of memory available for the workload to use. The workload will run on an automatic instance size or type based on the amount of memory available.

    Example for Pod object

    apiVersion: v1
    kind: Pod
    metadata:
      name: hello-openshift
      labels:
        app: hello-openshift
      annotations:
        io.katacontainers.config.hypervisor.machine_type: Standard_DC4as_v5 1
    spec:
      runtimeClassName: kata-remote
      containers:
        - name: hello-openshift
          image: quay.io/openshift/origin-hello-openshift
          ports:
            - containerPort: 8888
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

    1
    This example uses a previously defined instance size for peer pods using Azure. Peer pods using AWS use instance types.

OpenShift Container Platform creates the workload and begins scheduling it.

Verification

  • Inspect the runtimeClassName field on a pod-templated object. If the runtimeClassName is kata-remote, then the workload is running on OpenShift sandboxed containers, using peer pods.

3.4. Additional resources