Chapter 3. Deploying OpenShift sandboxed containers workloads using peer pods

You can install the OpenShift sandboxed containers Operator using either the web console or OpenShift CLI (oc). Before installing the OpenShift sandboxed containers Operator, you must prepare your Red Hat OpenShift cluster.

Important

Deploying OpenShift sandboxed containers workloads using peer pods is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

3.1. Prerequisites

Before you install OpenShift sandboxed containers and enable peer pods, you must meet the following requirements:

  • You have Red Hat OpenShift 4.13 installed on AWS or Azure.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.

3.1.1. About peer pod resource requirements in OpenShift sandboxed containers

A peer pod uses resources in two locations:

  • The worker node. The worker node stores metadata, Kata shim resources (containerd-shim-kata-v2), remote-hypervisor resources (cloud-api-adaptor), and the tunnel setup between the worker nodes and the peer-pod virtual machine (VM).
  • The cloud instance. This is the actual peer-pod VM running in the cloud.

The CPU and memory resources used in the Kubernetes worker node are handled by the pod overhead included in the RuntimeClass (kata-remote) definition used for creating peer pods.

The total number of peer-pod VMs running in the cloud is defined as Kubernetes Node extended resources. This limit is per node and is set by the limit attribute in the peerpodConfig custom resource (CR). The peerpodConfig CR, named peerpodconfig-openshift, is created when you create the kataConfig CR and enable peer pods, and is located in the openshift-sandboxed-containers-operator namespace.

The following peerpodConfig CR example displays the default spec values:

apiVersion: confidentialcontainers.org/v1alpha1
kind: PeerPodConfig
metadata:
  name: peerpodconfig-openshift
  namespace: openshift-sandboxed-containers-operator
spec:
  cloudSecretName: peer-pods-secret
  configMapName: peer-pods-cm
  limit: "10" 1
  nodeSelector:
    node-role.kubernetes.io/kata-oc: ""
1
The default limit is 10 VMs per node.

The extended resource is named kata.peerpods.io/vm, and enables the Kubernetes scheduler to handle capacity tracking and accounting.

You can edit the limit per node based on the requirements for your environment. See Modifying the VM limit per node in peer pods for more information.

A mutating webhook adds the extended resource kata.peerpods.io/vm to the pod specification. It also removes any resource-specific entries from the pod specification, if present. This enables the Kubernetes scheduler to account for these extended resources, ensuring the peer-pod is only scheduled when resources are available.

The mutating webhook modifies a Kubernetes pod as follows:

  • The mutating webhook checks the pod for the expected RuntimeClassName value, specified in the TARGET_RUNTIME_CLASS environment variable. If the value in the pod specification does not match the value in the TARGET_RUNTIME_CLASS, the webhook exits without modifying the pod.
  • If the RuntimeClassName values match, the webhook makes the following changes to the pod spec:

    1. The webhook removes every resource specification from the resources field of all containers and init containers in the pod.
    2. The webhook adds the extended resource (kata.peerpods.io/vm) to the spec by modifying the resources field of the first container in the pod. The extended resource kata.peerpods.io/vm is used by the Kubernetes scheduler for accounting purposes.
Note

The mutating webhook excludes specific system namespaces in Red Hat OpenShift from mutation. If a peer-pod is created in those system namespaces, then resource accounting using Kubernetes extended resources does not work unless the pod spec includes the extended resource.

As a best practice, define a cluster-wide policy to only allow peer-pod creation in specific namespaces.

3.1.1.1. Modifying the peer pod VM limit per node

You can change the limit of peer pod VMs per node by editing the peerpodConfig custom resource (CR).

Procedure

  1. Check the current limit by running the following command:

    $ oc get peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
    -o jsonpath='{.spec.limit}{"\n"}'
  2. Modify the limit attribute of the peerpodConfig CR by running the following command:

    $ oc patch peerpodconfig peerpodconfig-openshift -n openshift-sandboxed-containers-operator \
    --type merge --patch '{"spec":{"limit":"<value>"}}' 1
    1
    Replace <value> with the limit you want to define.

3.1.2. Prerequisites for peer-pods using AWS

If you are using AWS to create peer pods, you must ensure the following requirements:

  • Your Red Hat OpenShift cluster must be installed on AWS, with at least one worker node.
  • You have access to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY credentials. These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster.
  • You must have the AWS CLI tool installed and configured.
  • You must enable internal cluster communication on ports 15150 and 9000.

    You can enable these ports in either the AWS web console or using the CLI.

3.1.2.1. Enable ports 15150 and 9000 for AWS

Procedure

  1. Retrieve the Instance ID:

    $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
  2. Retrieve the AWS region:

    $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}')
  3. Retrieve the security groups:

    $ SG=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text --region ${AWS_REGION})
  4. Authorize the peer-pods shim and to access kata-agent communication. Run the following command:

    $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 15150 --source-group ${SG} --region ${AWS_REGION}
  5. Set up the peer-pods tunnel. Run the following command:

    $ aws ec2 authorize-security-group-ingress --group-id ${SG} --protocol tcp --port 9000 --source-group ${SG} --region ${AWS_REGION}

The ports are now enabled.

3.1.3. Prerequisites for peer-pods using Azure

If you are using Microsoft Azure to create peer pods, you must ensure the following requirements:

  • Your Red Hat OpenShift cluster must be installed on Azure, with at least one worker node.
  • You have access to the following credentials and subscription details:

    • AZURE_SUBSCRIPTION_ID
    • AZURE_CLIENT_ID
    • AZURE_CLIENT_SECRET
    • AZURE_TENANT_ID

    These are used to create additional cloud instances in the same virtual private cloud (VPC) of the cluster.

  • You must have the Azure CLI tool installed and configured.
  • You must enable cluster communication on ports 15150 and 9000.

    In Azure, internal communication is allowed on these ports. However, if communication is blocked, you can enable the ports in either the Azure web console or using the CLI.

3.1.3.1. Enable ports 15150 and 9000 for Azure

Procedure

  1. Retrieve the Instance ID:

    $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')
  2. Retrieve the Azure resource group:

    $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}')
  3. Retrieve the Azure network security group (NSG) name:

    $ AZURE_NSG_NAME=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
  4. Retrieve the Azure VNet name:

    $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)
  5. Retrieve the Azure subnet name:

    $ AZURE_SUBNET_NAME=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name ${AZURE_VNET_NAME}  --query "[].{Name:name} | [? contains(Name, 'worker')]" --output tsv)
  6. Retrieve the Azure subnet prefix:

    $ AZURE_SUBNET_PREFIX=$(az network vnet subnet show --name ${AZURE_SUBNET_NAME} --vnet-name ${AZURE_VNET_NAME} --resource-group ${AZURE_RESOURCE_GROUP} --query "addressPrefix" --output tsv)
  7. Authorize the peer-pods shim to access kata-agent communication. Run the following command:

    $ az network nsg rule create \
      --resource-group $AZURE_RESOURCE_GROUP \
      --nsg-name $AZURE_NSG_NAME \
      --name Allow-Kata-Agent-Internal \
      --access Allow \
      --protocol Tcp \
      --direction Inbound \
      --priority 112 \
      --source-address-prefixes $AZURE_SUBNET_PREFIX \
      --source-port-range "*" \
      --destination-address-prefixes $AZURE_SUBNET_PREFIX  \
      --destination-port-range 15150
  8. Set up the peer-pods tunnel. Run the following command:

    $ az network nsg rule create \
      --resource-group $AZURE_RESOURCE_GROUP \
      --nsg-name $AZURE_NSG_NAME \
      --name Allow-VXLAN-Internal \
      --access Allow \
      --protocol Tcp \
      --direction Inbound \
      --priority 111 \
      --source-address-prefixes $AZURE_SUBNET_PREFIX \
      --source-port-range "*" \
      --destination-address-prefixes $AZURE_SUBNET_PREFIX  \
      --destination-port-range 9000

The ports are now enabled.

3.2. Deploying OpenShift sandboxed containers workloads using peer pods with the web console

You can deploy OpenShift sandboxed containers workloads from the web console. First, you must install the OpenShift sandboxed containers Operator, then create a secret object, the VM image, and a peer-pod ConfigMap. The secret object and ConfigMap are unique, depending on your cloud provider. Finally, you must create the KataConfig custom resource (CR). Once you are ready to deploy a workload in a sandboxed container, you must manually add kata-remote-cc as the runtimeClassName to the workload YAML file.

3.2.1. Installing the OpenShift sandboxed containers Operator using the web console

You can install the OpenShift sandboxed containers Operator from the Red Hat OpenShift web console.

Prerequisites

  • You have Red Hat OpenShift 4.13 installed.
  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsOperatorHub.
  2. In the Filter by keyword field, type OpenShift sandboxed containers.
  3. Select the OpenShift sandboxed containers tile.
  4. Read the information about the Operator and click Install.
  5. On the Install Operator page:

    1. Select stable from the list of available Update Channel options.
    2. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory openshift-sandboxed-containers-operator namespace. If this namespace does not yet exist, it is automatically created.

      Note

      Attempting to install the OpenShift sandboxed containers Operator in a namespace other than openshift-sandboxed-containers-operator causes the installation to fail.

    3. Verify that Automatic is selected for Approval Strategy. Automatic is the default value, and enables automatic updates to OpenShift sandboxed containers when a new z-stream release is available.
  6. Click Install.

The OpenShift sandboxed containers Operator is now installed on your cluster.

Verification

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Verify that the OpenShift sandboxed containers Operator is listed in the in operators list.

3.2.2. Configuring peer-pod parameters for AWS using the web console

You must create a secret object and a ConfigMap to deploy OpenShift sandboxed containers using peer pods on AWS.

After creating the secret object, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.2.2.1. Creating a secret object for AWS using the web console

Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for AWS using the CLI.

In addition, in the AWS web console, you must locate and prepare the following values:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AWS_ACCESS_KEY_ID: "<enter value>" 1
      AWS_SECRET_ACCESS_KEY: "<enter value>" 2
      AWS_REGION: "<enter value>" 3
      AWS_SUBNET_ID: "<enter value>" 4
      AWS_VPC_ID: "<enter value>" 5
      AWS_SG_IDS: "<enter value>" 6
    1
    Enter the AWS_ACCESS_KEY_ID value you prepared before you began.
    2
    Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.
    3
    Enter the AWS_REGION value you retrieved.
    4
    Enter the AWS_SUBNET_ID value you retrieved.
    5
    Enter the AWS_VPC_ID value you retrieved.
    6
    Enter the AWS_SG_IDS value you retrieved.
  5. Click Create.

The secret object is created. You can see it listed under WorkloadsSecrets.

3.2.2.2. Creating an AWS VM image (AMI) using the web console

To run OpenShift sandboxed containers using peer pods on AWS, you must first create a RHEL AMI using your AWS account and resources.

Procedure

  1. From the Administrator perspective in the web console, navigate to WorkloadsJobs.
  2. In the Jobs window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
  3. Click Create Job.
  4. In the Create Job window, paste this complete YAML manifest.
  5. Click Create.

The image is created.

Note

This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the AWS web console or AWS CLI tool.

Once the image is created, you must set the image using a peer-pod ConfigMap.

3.2.2.3. Creating a peer-pod ConfigMap for AWS using the web console

When creating a ConfigMap for AWS, you must set the AMI ID. You can retrieve this value before you create the ConfigMap. Retrieving this value must be done using the CLI. For more information, see Creating a peer-pod ConfigMap for AWS using the CLI.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "aws"
      VXLAN_PORT: "9000"
      PODVM_INSTANCE_TYPE: "t3.medium"
      PROXY_TIMEOUT: "5m"
      PODVM_AMI_ID: "<enter value>" 1
    1
    Enter the PODVM_AMI_ID value you retrieved.
  5. Click Create.

The ConfigMap object is created. You can see it listed under WorkloadsConfigMaps.

Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on AWS.

3.2.3. Configuring peer-pod parameters for Azure using the web console

You must create a secret object and a ConfigMap in order to deploy OpenShift sandboxed containers using peer pods on Microsoft Azure.

After creating the secret object, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.2.3.1. Creating a secret object for Azure using the web console

Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for Azure, you must set specific environment values. You can retrieve some of these values before creating the secret object. Retrieving these values must be done using the CLI. For more information, see Creating a secret object for Azure using the CLI.

In addition, in the Azure web console, you must locate and prepare the following values:

  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET
  • AZURE_TENANT_ID

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AZURE_CLIENT_ID: "<enter value>" 1
      AZURE_CLIENT_SECRET: "<enter value>" 2
      AZURE_TENANT_ID: "<enter value>" 3
      AZURE_SUBSCRIPTION_ID: "<enter value>" 4
      AZURE_REGION: "<enter value>" 5
      AZURE_RESOURCE_GROUP: "<enter value>" 6
    1
    Enter the AZURE_CLIENT_ID value you prepared before you began.
    2
    Enter the AZURE_CLIENT_SECRET value you prepared before you began.
    3
    Enter the AZURE_TENANT_ID value you prepared before you began.
    4
    Enter the AZURE_SUBSCRIPTION_ID value you retrieved.
    5
    Enter the AZURE_REGION value you retrieved.
    6
    Enter the AZURE_RESOURCE_GROUP value you retrieved.
  5. Click Create.

The secret object is created. You can see it listed under WorkloadsSecrets.

3.2.3.2. Creating an Azure VM image using the web console

To run OpenShift sandboxed containers using peer pods on Azure, you must first create a RHEL image for Azure using your Azure account and resources.

Procedure

  1. From the Administrator perspective in the web console, navigate to WorkloadsJobs.
  2. In the Jobs window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
  3. Click Create Job.
  4. In the Create Job window, paste this complete YAML manifest.
  5. Click Create.

The image is created.

Note

This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the Azure web console or Azure CLI tool.

Once the image is created, you must set the image using a peer-pod ConfigMap.

3.2.3.3. Creating a peer-pod ConfigMap for Azure using the web console

When creating a ConfigMap for Azure, you must set specific configuration values. Retrieving these values must be done using the CLI. For more information, see Creating a peer-pod ConfigMap for Azure using the CLI.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. Click the Import icon (+) in the top right corner.
  4. In the Import YAML window, paste the following YAML manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "azure"
      VXLAN_PORT: "9000"
      AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
      AZURE_SUBNET_ID: "<enter value>" 1
      AZURE_NSG_ID: "<enter value>" 2
      AZURE_IMAGE_ID: "<enter value>" 3
      PROXY_TIMEOUT: "5m"
      DISABLECVM: "true"
    1
    Enter the AZURE_SUBNET_ID value you retrieved.
    2
    Enter the AZURE_NSG_ID value you retrieved.
    3
    Enter the AZURE_IMAGE_ID value you retrieved.
  5. Click Create.

The ConfigMap object is created. You can see it listed under WorkloadsConfigMaps.

3.2.3.4. Creating an SSH key secret object for Azure using the web console

You must create an SSH key secret object to use peer pods with Azure. If you do not already have an SSH key to create the object, you must generate one using the CLI. Fo more information, see

Procedure

  1. From the Administrator perspective in the web console, navigate to WorkloadsSecrets.
  2. In the Secrets window, in the top left corner, verify that you are in the openshift-sandboxed-containers-operator project.
  3. Click Create and select Key/value secret from the list.
  4. In the Secret name field, enter ssh-key-secret.
  5. In the Key field, enter id_rsa.pub.
  6. In the Value field, paste your public SSH key.
  7. Click Create.

The SSH key secret object is created. You can see it listed under WorkloadsSecrets.

Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on Azure.

3.2.4. Creating the KataConfig custom resource in the web console

You must create one KataConfig custom resource (CR) to enable installing kata-remote-cc as a RuntimeClass on your cluster nodes.

Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger Red Hat OpenShift deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Note

Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote-cc as a RuntimeClass only on specific nodes, you can add labels to those nodes, then define the label in the KataConfig CR when you create it.

Procedure

  1. From the Administrator perspective in the web console, navigate to OperatorsInstalled Operators.
  2. Select the OpenShift sandboxed containers Operator from the list of operators.
  3. In the KataConfig tab, click Create KataConfig.
  4. In the Create KataConfig page, enter the following details:

    • Name: Enter a name for the KataConfig resource. By default, the name is defined as example-kataconfig.
    • Labels (Optional): Enter any relevant, identifying attributes to the KataConfig resource. Each label represents a key-value pair.
    • checkNodeEligibility (Optional, not relevant for peer pods): Select this checkbox to use the Node Feature Discovery Operator (NFD) to detect node eligibility to run kata as a RuntimeClass. For more information, see "Checking whether cluster nodes are eligible to run OpenShift sandboxed containers".
    • enablePeerPods (For peer pods): Select this checkbox to enable peer pods and use OpenShift sandboxed containers in a public cloud environment.
    • kataConfigPoolSelector: By default, kata-remote-cc is installed as a RuntimeClass on all nodes. If you want to install kata-remote-cc as a RuntimeClass only on selected nodes, you must add a matchExpression:

      1. Expand the kataConfigPoolSelector area.
      2. In the kataConfigPoolSelector, expand matchExpressions. This is a list of label selector requirements.
      3. Click Add matchExpressions.
      4. In the key field, add the label key the selector applies to.
      5. In the operator field, add the key’s relationship to the label values. Valid operators are In, NotIn, Exists, and DoesNotExist.
      6. Expand the values area, and then click Add value.
      7. In the Value field, enter true or false for key label value.
    • logLevel: Define the level of log data retrieved for nodes running kata-remote-cc as a RuntimeClass. For more information, see "Collecting OpenShift sandboxed containers data".
  5. Click Create.

The new KataConfig CR is created and begins to install kata-remote-cc as a RuntimeClass on the worker nodes. Wait for the installation to complete and the worker nodes to reboot before continuing to the next step.

Important

OpenShift sandboxed containers installs kata-remote-cc only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  1. In the KataConfig tab, select the new KataConfig CR.
  2. In the KataConfig page, select the YAML tab.
  3. Monitor the installationStatus field in the status.

    A message appears each time there is an update. Click Reload to view the updated KataConfig CR.

    Once the value of Completed nodes equals the number of worker or labeled nodes, the installation is complete. The status also contains a list of nodes where the installation is completed.

3.2.5. Deploying a workload with peer pods in a sandboxed container using the web console

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload using peer pods in a sandboxed container, you must manually add kata-remote-cc as the runtimeClassName to the workload YAML file.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a secret object and peer-pod config map unique to your cloud provider.
  • You have created a KataConfig custom resource (CR).

Procedure

  1. From the Administrator perspective in the web console, expand Workloads and select the type of workload you want to create.
  2. In the workload page, click to create the workload.
  3. In the YAML file for the workload, in the spec field where the container is listed, add runtimeClassName: kata-remote-cc.

    Example for Pod object

    apiVersion: v1
    kind: Pod
    metadata:
      name: hello-openshift
      labels:
        app: hello-openshift
    spec:
      runtimeClassName: kata-remote-cc
      containers:
        - name: hello-openshift
          image: quay.io/openshift/origin-hello-openshift
          ports:
            - containerPort: 8888
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            runAsUser: 1001
            capabilities:
              drop:
                - ALL
            seccompProfile:
              type: RuntimeDefault

  4. Click Save.

Red Hat OpenShift creates the workload and begins scheduling it.

3.3. Deploying OpenShift sandboxed containers workloads using peer pods with the CLI

You can deploy OpenShift sandboxed containers workloads using the CLI. First, you must install the OpenShift sandboxed containers Operator, then create the KataConfig custom resource. Once you are ready to deploy a workload in a sandboxed container, you must add kata-remote-cc as the runtimeClassName to the workload YAML file.

3.3.1. Installing the OpenShift sandboxed containers Operator using the CLI

You can install the OpenShift sandboxed containers Operator using the Red Hat OpenShift CLI.

Prerequisites

  • You have Red Hat OpenShift 4.13 installed on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have subscribed to the OpenShift sandboxed containers catalog.

    Note

    Subscribing to the OpenShift sandboxed containers catalog provides openshift-sandboxed-containers-operator namespace access to the OpenShift sandboxed containers Operator.

Procedure

  1. Create the Namespace object for the OpenShift sandboxed containers Operator.

    1. Create a Namespace object YAML file that contains the following manifest:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-sandboxed-containers-operator
    2. Create the Namespace object:

      $ oc create -f Namespace.yaml
  2. Create the OperatorGroup object for the OpenShift sandboxed containers Operator.

    1. Create an OperatorGroup object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        targetNamespaces:
        - openshift-sandboxed-containers-operator
    2. Create the OperatorGroup object:

      $ oc create -f OperatorGroup.yaml
  3. Create the Subscription object to subscribe the Namespace to the OpenShift sandboxed containers Operator.

    1. Create a Subscription object YAML file that contains the following manifest:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-sandboxed-containers-operator
        namespace: openshift-sandboxed-containers-operator
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: sandboxed-containers-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: sandboxed-containers-operator.v1.4.1
    2. Create the Subscription object:

      $ oc create -f Subscription.yaml

The OpenShift sandboxed containers Operator is now installed on your cluster.

Note

All the object file names listed above are suggestions. You can create the object YAML files using other names.

Verification

  • Ensure that the Operator is correctly installed:

    $ oc get csv -n openshift-sandboxed-containers-operator

    Example output

    NAME                             DISPLAY                                  VERSION  REPLACES     PHASE
    openshift-sandboxed-containers   openshift-sandboxed-containers-operator  1.4.1    1.4.0        Succeeded

3.3.2. Setting up peer pods for AWS using the CLI

To set up peer pods for use on AWS, you must create a secret object, an AWS image VM (AMI), and a peer-pod ConfigMap.

After setting up peer pods for AWS, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.3.2.1. Creating a secret object for AWS using the CLI

Set your AWS access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for AWS, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Procedure

  1. Collect the necessary parameter values for the secret object. Make sure to write down each value.

    1. Retrieve the instance ID:

      $ INSTANCE_ID=$(oc get nodes -l 'node-role.kubernetes.io/worker' -o jsonpath='{.items[0].spec.providerID}' | sed 's#[^ ]*/##g')

      This value is not needed for the secret object itself, but is used to retrieve other values for the secret object.

    2. Retrieve the AWS region:

      $ AWS_REGION=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.aws.region}') && echo "AWS_REGION: \"$AWS_REGION\""
    3. Retrieve the AWS subnet ID:

      $ AWS_SUBNET_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SubnetId' --region ${AWS_REGION} --output text) && echo "AWS_SUBNET_ID: \"$AWS_SUBNET_ID\""
    4. Retrieve the AWS VPC ID:

      $ AWS_VPC_ID=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].VpcId' --region ${AWS_REGION} --output text) && echo "AWS_VPC_ID: \"$AWS_VPC_ID\""
    5. Retrieve the AWS security group IDs:

      $ AWS_SG_IDS=$(aws ec2 describe-instances --instance-ids ${INSTANCE_ID} --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --region ${AWS_REGION} --output text)
      && echo "AWS_SG_IDS: \"$AWS_SG_IDS\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AWS_ACCESS_KEY_ID: "<enter value>" 1
      AWS_SECRET_ACCESS_KEY: "<enter value>" 2
      AWS_REGION: "<enter value>" 3
      AWS_SUBNET_ID: "<enter value>" 4
      AWS_VPC_ID: "<enter value>" 5
      AWS_SG_IDS: "<enter value>" 6
    1
    Enter the AWS_ACCESS_KEY_ID value you prepared before you began.
    2
    Enter the AWS_SECRET_ACCESS_KEY value you prepared before you began.
    3
    Enter the AWS_REGION value you retrieved.
    4
    Enter the AWS_SUBNET_ID value you retrieved.
    5
    Enter the AWS_VPC_ID value you retrieved.
    6
    Enter the AWS_SG_IDS value you retrieved.
  3. Apply the secret object:

    $ oc apply -f peer-pods-secret.yaml

The secret object is applied.

3.3.2.2. Creating an AWS VM image (AMI) using the CLI

To run OpenShift sandboxed containers using peer pods on AWS, you must first create a RHEL AMI using your AWS account and resources.

Procedure

  1. Run the following K8s job to create the image:

    $ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/peer-pods-tech-preview/hack/aws-image-job.yaml
    Note

    This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the AWS web console or AWS CLI tool.

  2. Wait for the job to complete:

    $ oc wait --for=condition=complete job.batch/aws-image-creation --timeout=7m -n openshift-sandboxed-containers-operator

Once the image is created, you must set the image using a peer-pod ConfigMap.

3.3.2.3. Creating a peer-pod ConfigMap for AWS using the CLI

When creating a ConfigMap for AWS, you must set the AMI ID. You can retrieve this value before you create the ConfigMap.

Procedure

  1. Retrieve the AMI ID. Make sure to save the value for later.

    $ PODVM_AMI_ID=$(aws ec2 describe-images --query "Images[*].[ImageId]" --filters "Name=name,Values=peer-pod-ami" --region ${AWS_REGION} --output text) && echo "PODVM_AMI_ID: \"$PODVM_AMI_ID\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "aws"
      VXLAN_PORT: "9000"
      PODVM_INSTANCE_TYPE: "t3.medium"
      PROXY_TIMEOUT: "5m"
      PODVM_AMI_ID: "<enter value>" 1
    1
    Enter the AMI ID value you retrieved.
  3. Deploy the ConfigMap:

    $ oc apply -f peer-pods-cm.yaml

The ConfigMap is deployed. Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on AWS.

3.3.3. Setting up peer pods for Azure using the CLI

To set up peer pods for use on Microsoft Azure, you must create a secret object, an Azure image VM, a peer-pod ConfigMap, and an SSH key secret object.

After setting up peer pods for Azure, you must still create the KataConfig custom resource (CR) to deploy OpenShift sandboxed containers using peer pods.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.

3.3.3.1. Creating a secret object for Azure using the CLI

Set your Azure access keys and configure your network in a secret object. The secret object is used to create the pod VM image and used by peer pods.

When creating a secret object for Azure, you must set specific environment values. You can retrieve some of these values before creating the secret object. However, you must have the following values prepared:

  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET
  • AZURE_TENANT_ID

Procedure

  1. Collect additional parameter values for the secret object. Make sure to write down each value.

    1. Retrieve the subscription ID:

      $ AZURE_SUBSCRIPTION_ID=$(az account list --query "[?isDefault].id" -o tsv) && echo "AZURE_SUBSCRIPTION_ID: \"$AZURE_SUBSCRIPTION_ID\""
    2. Retrieve the resource group:

      $ AZURE_RESOURCE_GROUP=$(oc get infrastructure/cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') && echo "AZURE_RESOURCE_GROUP: \"$AZURE_RESOURCE_GROUP\""
    3. Retrieve the Azure region:

      $ AZURE_REGION=$(az group show --resource-group ${AZURE_RESOURCE_GROUP} --query "{Location:location}" --output tsv) && echo "AZURE_REGION: \"$AZURE_REGION\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: Secret
    metadata:
      name: peer-pods-secret
      namespace: openshift-sandboxed-containers-operator
    type: Opaque
    stringData:
      AZURE_CLIENT_ID: "<enter value>" 1
      AZURE_CLIENT_SECRET: "<enter value>" 2
      AZURE_TENANT_ID: "<enter value>" 3
      AZURE_SUBSCRIPTION_ID: "<enter value>" 4
      AZURE_REGION: "<enter value>" 5
      AZURE_RESOURCE_GROUP: "<enter value>" 6
    1
    Enter the AZURE_CLIENT_ID value you prepared before you began.
    2
    Enter the AZURE_CLIENT_SECRET value you prepared before you began.
    3
    Enter the AZURE_TENANT_ID value you prepared before you began.
    4
    Enter the AZURE_SUBSCRIPTION_ID value you retrieved.
    5
    Enter the AZURE_REGION value you retrieved.
    6
    Enter the AZURE_RESOURCE_GROUP value you retrieved.
  3. Apply the secret object:

    $ oc apply -f peer-pods-secret.yaml

The secret object is applied.

3.3.3.2. Creating an Azure VM image using the CLI

To run OpenShift sandboxed containers using peer pods on Azure, you must first create a RHEL image for Azure using your Azure account and resources.

Procedure

  1. Run the following K8s job to create the image:

    $ oc apply -f https://raw.githubusercontent.com/openshift/sandboxed-containers-operator/peer-pods-tech-preview/hack/azure-image-job.yaml
    Note

    This image is not managed by OpenShift sandboxed containers. You can delete it if necessary using the Azure web console or Azure CLI tool.

  2. Wait for the job to complete:

    $ oc wait --for=condition=complete job.batch/azure-image-creation --timeout=7m -n openshift-sandboxed-containers-operator

Once the image is created, you must set the image using a peer-pod ConfigMap.

3.3.3.3. Creating a peer-pod ConfigMap for Azure using the CLI

When creating a ConfigMap for Azure, you must set specific configuration values. You can retrieve these values before you create the ConfigMap.

Procedure

  1. Collect the configuration values for the Azure peer-pod ConfigMap. Make sure to write down each value.

    1. Retrieve the Azure image ID:

      $ AZURE_IMAGE_ID=$(az image list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id: id} | [? contains(Id, 'peer-pod-vmimage')]" --output tsv) && echo "AZURE_IMAGE_ID: \"$AZURE_IMAGE_ID\""
    2. Retrieve the Azure VNet name:

      $ AZURE_VNET_NAME=$(az network vnet list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Name:name}" --output tsv)

      This value is not needed for the ConfigMap, but is used to retrieve the Azure subnet ID.

    3. Retrieve the Azure subnet ID:

      $ AZURE_SUBNET_ID=$(az network vnet subnet list --resource-group ${AZURE_RESOURCE_GROUP} --vnet-name $AZURE_VNET_NAME --query "[].{Id:id} | [? contains(Id, 'worker')]" --output tsv) && echo "AZURE_SUBNET_ID: \"$AZURE_SUBNET_ID\""
    4. Retrieve the Azure network security group (NSG) ID:

      $ AZURE_NSG_ID=$(az network nsg list --resource-group ${AZURE_RESOURCE_GROUP} --query "[].{Id:id}" --output tsv) && echo "AZURE_NSG_ID: \"$AZURE_NSG_ID\""
  2. Create a YAML file with the following manifest:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: peer-pods-cm
      namespace: openshift-sandboxed-containers-operator
    data:
      CLOUD_PROVIDER: "azure"
      VXLAN_PORT: "9000"
      AZURE_INSTANCE_SIZE: "Standard_B2als_v2"
      AZURE_SUBNET_ID: "<enter value>" 1
      AZURE_NSG_ID: "<enter value>" 2
      AZURE_IMAGE_ID: "<enter value>" 3
      PROXY_TIMEOUT: "5m"
      DISABLECVM: "true"
    1
    Enter the AZURE_SUBNET_ID value you retrieved.
    2
    Enter the AZURE_NSG_ID value you retrieved.
    3
    Enter the AZURE_IMAGE_ID value you retrieved.
  3. Deploy the ConfigMap:

    $ oc apply -f peer-pods-cm.yaml

The ConfigMap is deployed.

3.3.3.4. Creating an SSH key secret object for Azure using the CLI

You must generate an SSH key and create an SSH key secret object to use peer pods with Azure.

Procedure

  1. Generate the SSH key:

    $ ssh-keygen -f ./id_rsa -N ""
  2. Create the SHH secret object:

    $ oc create secret generic ssh-key-secret -n openshift-sandboxed-containers-operator --from-file=id_rsa.pub=./id_rsa.pub --from-file=id_rsa=./id_rsa

The SSH key is created and the SSH key secret object is created. Once you create the KataConfig CR, you can run OpenShift sandboxed containers using peer pods on Azure.

3.3.4. Creating the KataConfig custom resource using the CLI

You must create one KataConfig custom resource (CR) to install kata-remote-cc as a RuntimeClass on your nodes. Creating the KataConfig CR triggers the OpenShift sandboxed containers Operator to do the following:

  • Install the needed RHCOS extensions, such as QEMU and kata-containers, on your RHCOS node.
  • Ensure that the CRI-O runtime is configured with the correct runtime handlers.
  • Create a RuntimeClass CR named kata-remote-cc with a default configuration. This enables users to configure workloads to use kata-remote-cc as the runtime by referencing the CR in the RuntimeClassName field. This CR also specifies the resource overhead for the runtime.
Note

Kata for peer pods is installed on all worker nodes by default. If you want to install kata-remote-cc as a RuntimeClass only on specific nodes, you can add labels to those nodes, and then define the label in the KataConfig CR when you create it.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
Important

Creating the KataConfig CR automatically reboots the worker nodes. The reboot can take from 10 to more than 60 minutes. Factors that impede reboot time are as follows:

  • A larger Red Hat OpenShift deployment with a greater number of worker nodes.
  • Activation of the BIOS and Diagnostics utility.
  • Deployment on a hard disk drive rather than an SSD.
  • Deployment on physical nodes such as bare metal, rather than on virtual nodes.
  • A slow CPU and network.

Procedure

  1. Create a YAML file with the following manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      enablePeerPods: true
      logLevel: info
  2. (Optional) If you want to install kata-remote-cc as a RuntimeClass only on selected nodes, create a YAML file that includes the label in the manifest:

    apiVersion: kataconfiguration.openshift.io/v1
    kind: KataConfig
    metadata:
      name: cluster-kataconfig
    spec:
      enablePeerPods: true
      logLevel: info
      kataConfigPoolSelector:
        matchLabels:
          <label_key>: '<label_value>' 1
    1
    Labels in kataConfigPoolSelector only support single values; nodeSelector syntax is not supported.
  3. Create the KataConfig resource:

    $ oc create -f cluster-kataconfig.yaml

The new KataConfig CR is created and begins to install kata-remote-cc as a RuntimeClass on the worker nodes. Wait for the kata-remote-cc installation to complete and the worker nodes to reboot before continuing to the next step.

Important

OpenShift sandboxed containers installs kata-remote-cc only as a secondary, optional runtime on the cluster and not as the primary runtime.

Verification

  • Monitor the installation progress:

    $ watch "oc describe kataconfig | sed -n /^Status:/,/^Events/p"

    Once the value of Is In Progress appears as false, the installation is complete.

3.3.5. Deploying a workload with peer pods in a sandboxed container using the CLI

OpenShift sandboxed containers installs Kata as a secondary, optional runtime on your cluster, and not as the primary runtime.

To deploy a pod-templated workload using peer pods in a sandboxed container, you must add kata-remote-cc as the runtimeClassName to the workload YAML file.

Prerequisites

  • You have installed Red Hat OpenShift 4.13 on your cluster.
  • You have installed the OpenShift CLI (oc).
  • You have access to the cluster as a user with the cluster-admin role.
  • You have installed the OpenShift sandboxed containers Operator.
  • You have created a secret object and peer-pod config map unique to your cloud provider.
  • You have created a KataConfig custom resource (CR).

Procedure

  • Add runtimeClassName: kata-remote-cc to any pod-templated object:

    • Pod objects
    • ReplicaSet objects
    • ReplicationController objects
    • StatefulSet objects
    • Deployment objects
    • DeploymentConfig objects

Example for Pod object

apiVersion: v1
kind: Pod
metadata:
  name: hello-openshift
  labels:
    app: hello-openshift
spec:
  runtimeClassName: kata-remote-cc
  containers:
    - name: hello-openshift
      image: quay.io/openshift/origin-hello-openshift
      ports:
        - containerPort: 8888
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1001
        capabilities:
          drop:
            - ALL
        seccompProfile:
          type: RuntimeDefault

Red Hat OpenShift creates the workload and begins scheduling it.

Verification

  • Inspect the runtimeClassName field on a pod-templated object. If the runtimeClassName is kata-remote-cc, then the workload is running on OpenShift sandboxed containers, using peer pods.

3.4. Additional resources