Setting up confidential model serving with KServe in Red Hat OpenShift AI

Overview

Confidential model serving protects intellectual property of served models by keeping model weights encrypted except inside a hardware-based Trusted Execution Environment (TEE). The model stays encrypted at rest and in transit, and is decrypted only after pod attestation succeeds.

After attestation, the decryption key is released directly into the TEE by Key Broker Service (KBS) and does not appear in etcd, container images, or cluster logs.

KServe implements confidential model serving using the Confidential Containers (CoCo) framework through the following components:

• Red Hat OpenShift Sandboxed Containers: Runs serving pods inside a confidential guest environment on TEE-capable hardware (Intel TDX or AMD SEV-SNP).
• Confidential Data Hub (CDH): Runs as an agent inside the confidential guest to perform hardware attestation for the workload.
• RuntimeClass selection: Varies by deployment environment:
  • Cloud VMs: Uses peer pods with the kata-remote RuntimeClass.
  • Bare metal: Runs the guest directly on the host TEE using kata-cc or variant RuntimeClass values (such as kata-cc-nvidia-gpu).
• Red Hat build of Trustee: Serves as the Key Broker Service (KBS) to verify CDH attestation evidence and release registered decryption keys.
• KServe storage-initializer: Downloads model artifacts and requests decryption keys from CDH to perform in-place decryption before starting the model server container.
• Supported custom resources: Works with both InferenceService and LLMInferenceService custom resources.

Prerequisites

• Access to an OpenShift cluster with nodes capable of running confidential workloads (Intel TDX or AMD SEV-SNP).
• Red Hat OpenShift Sandboxed Containers Operator is installed and configured for confidential containers.
• A confidential RuntimeClass is available on the cluster:
• Red Hat build of Trustee is deployed and configured as a KBS backing cluster attestation policy.
• Red Hat OpenShift AI 3.6 EA2 is installed on the cluster.
• Model storage that is reachable by confidential workload.
• A ServiceAccount bound to a secret containing object storage credentials.
• Model artifacts encrypted using JWE Compact Serialization with A256GCM content encryption, with each file saved as an individual JWE token (typically using a .jwe file extension). Refer to Appendix A for example encryption methods.

Procedure

1. Load the decryption key into Trustee

• Register the model decryption key as a secret resource in Trustee to enable Key Broker Service (KBS) release to attested workloads using the kbs:///<repo>/<type>/<tag> resource ID format.
• Run the following commands in the Trustee cluster namespace:

# Create a Secret holding the raw key bytes (or the full JWK)
  oc create secret generic model-key \
    --from-file=key=model-key.bin \
    -n trustee-operator-system

# Register the Secret with the Trustee KbsConfig
  oc patch kbsconfig trusteeconfig-kbs-config \
    -n trustee-operator-system \
    --type=json \
    -p='[{"op": "add", "path": "/spec/kbsSecretResources/-", "value": "model-key"}]'

# Restart Trustee so it loads the new secret resource
  oc delete pod -l app=trustee -n trustee-operator-system

3. Deploy a confidential InferenceService

• Configure confidential model serving by specifying the RuntimeClass and setting confidential parameters alongside the storageUri field in the predictor specification of the predictor pod:

  apiVersion: serving.kserve.io/v1beta1
  kind: InferenceService
  metadata:
  name: encrypted-model
  namespace: kserve-demo
  spec:
  predictor:
    runtimeClassName: kata-remote
    serviceAccountName: sa-model-storage
    model:
      modelFormat:
        name: onnx
      runtime: kserve-ovms
      storageUri: s3://models/encrypted-model/
      confidential:
        enabled: true
        resourceId: kbs:///default/model-key/key
  

• Where:
  • runtimeClassName: Set to kata-remote or class name matching cluster deployment (such as kata-remote on cloud VMs or kata-cc/kata-cc-nvidia-gpu on bare metal).
  • confidential.enabled: Set to true to activate decryption in storage-initializer after model download completes.
  • confidential.resourceId: KBS resource ID under which decryption key was registered with Trustee.

4. Deploy a confidential LLMInferenceService

• Configure confidential model serving for an LLMInferenceService by setting runtimeClassName under spec.template, and adding confidential.enabled and confidential.resourceId under spec.model:

  apiVersion: serving.kserve.io/v1alpha2
  kind: LLMInferenceService
  metadata:
  name: encrypted-llm
  namespace: kserve-demo
  spec:
  template:
    runtimeClassName: kata-remote
    serviceAccountName: sa-model-storage
  model:
    uri: s3://models/encrypted-llm/
    confidential:
      enabled: true
      resourceId: kbs:///default/model-key/key
  

• Where:

  • runtimeClassName: Specifies confidential RuntimeClass matching cluster deployment (kata-remote for cloud VMs; kata-cc or kata-cc-nvidia-gpu for bare metal).
  • confidential.enabled: Set to true to activate decryption in storage-initializer after model download completes.
  • confidential.resourceId: Specifies the KBS resource ID (formatted as kbs:///default/model-key/key) under which the decryption key was registered in Trustee.

  Note: Ensure serviceAccountName references a ServiceAccount bound to the secret containing object storage credentials.

Limitations

• Supported encryption format: Requires JWE Compact Serialization with A256GCM content encryption.
• Unsupported storage types: Setting confidential.enabled: true is not supported for oci:// or pvc:// storage URIs.
• Supported model size: Up to 1.5 GiB (for example, Qwen3-0.6B).

Appendix A: Example methods for encrypting a model (unsupported)

Note: The following procedures are provided as unsupported reference examples only. Apply approved organizational toolchains to produce JWE-encrypted artifacts and manage keys.

• Expected JWE format demonstration using jose CLI:

1. Generate 256-bit symmetric key:

   jose jwk gen -i '{"alg":"A256GCM", "kty":"oct"}' -o symmetric.jwk
   

2. Encrypt each model file into JWE Compact Serialization:

   jose jwe enc -I model.onnx -k symmetric.jwk -o model.onnx.jwe
   

3. Remove unencrypted plaintext artifact before uploading:

   rm model.onnx
   

4. Repeat encryption step for every file in model directory, then upload resulting .jwe files to storage location.