Select a hardware configuration for llm-d model deployments

Hardware configurations for LLMInferenceService
Deploy a model with llm-d by selecting a serving configuration

Deploy a model with llm-d on IBM Spyre AI accelerators

Hardware configurations for LLMInferenceService

An LLMInferenceServiceConfig is a reusable set of serving settings.
An LLMInferenceService selects one by name to get the runtime image, the environment variables, and the pod placement settings for a particular AI accelerator.

Red Hat OpenShift AI provides configurations for supported AI accelerators.
Configs vary by release and cluster and can set the model image, container variables, resources, node selector, router, or scheduler.
Check the available hardware configurations in the cluster by running the following command:

$ oc get llminferenceserviceconfig -A

Each configuration sets the productized vLLM image for the hardware and the environment variables that the runtime needs on that hardware.

An LLMInferenceService uses a configuration name in its spec.baseRefs list.
For example, a cluster might provide a CPU config named v3-6-0-kserve-config-llm-template-cpu:

spec:
  baseRefs:
    - name: v3-6-0-kserve-config-llm-template-cpu

The name depends on the installed release and cluster.
A listed config describes serving settings.
It does not show which hardware nodes have or whether they have enough capacity for the model.

Deploy a model with llm-d by selecting a serving configuration

Use a serving config to set up an LLMInferenceService. This example uses CPU and the Endpoint Picker (EPP).

Prerequisites

Distributed Inference with llm-d is enabled, and you can create an LLMInferenceService in the cluster.
The cluster has enough resources for the selected model and its validated configuration.
The cluster can access the model source.
You have installed the OpenShift CLI (oc).

About this task

This task describes running a CPU model on a single node with EPP.
Adjust the LLMInferenceService configuration for your cluster and hardware.

Procedure

Review the LLMInferenceServiceConfig resources that are in the cluster and choose the relevant config for your use case.

Choose a configuration for your release, model, hardware, and use case, for example:

$ oc get llminferenceserviceconfig -A

The command returns:

NAMESPACE                 NAME                                                  READY   REASON   AGE
redhat-ods-applications   v3-6-0-kserve-config-llm-template-cpu                 True             13d
redhat-ods-applications   v3-6-0-kserve-config-llm-template-ibm-spyre-ppc64le   True             13d
redhat-ods-applications   v3-6-0-kserve-config-llm-template-ibm-spyre-s390x     True             13d
# ...

Inspect the CPU config:

$ oc get llminferenceserviceconfig v3-6-0-kserve-config-llm-template-cpu -n redhat-ods-applications -o yaml

Add the selected configuration name to the spec.baseRefs list in your LLMInferenceService resource. If you are updating an existing service, save its resource definition to a file:

$ oc get llminferenceservice <service_name> -n <namespace> -o yaml > llminferenceservice.yaml

Edit the file and add the config name:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: facebook-opt-125m-single
spec:
  baseRefs:
    - name: v3-6-0-kserve-config-llm-template-cpu
  model:
    uri: hf://facebook/opt-125m
    name: facebook/opt-125m
  replicas: 1
  router:
    scheduler: { }
    route: { }
    gateway: { }
  template:
    containers:
      - name: main
        resources:
          requests:
            cpu: '4'
            memory: 16Gi
          limits:
            cpu: '4'
            memory: 16Gi

Apply the updated LLMInferenceService resource in the cluster:
```
$ oc apply -f llminferenceservice.yaml -n <namespace>
```

Check the service and its pods for readiness:

$ oc get llminferenceservice <service_name> -n <namespace>

Verification

Confirm that the LLMInferenceService and model-serving pod are ready:
```
$ oc get pods -n <namespace>
```

Send a test inference request to the service's OpenAI-compatible endpoint:

TOKEN=$(oc whoami -t)
curl -X POST <endpoint_url>/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "facebook/opt-125m",
    "prompt": "What is the capital of France?",
    "max_tokens": 50
  }'

Replace <endpoint_url> with the full endpoint URL for the LLMInferenceService. The response should include generated text in the choices field.

Confirm that EPP is running and routes requests to the model server. Check that the EPP pod is ready:
```
$ oc get pods -n <namespace> | grep router-scheduler
```
In the OpenShift web console, select Observe > Metrics. Query this promql metric, replacing the namespace placeholder value with the namespace where the inference service is running.
```
increase(llm_d_epp_scheduler_attempts_total{status="success", namespace="<namespace>"}[5m])
```

Deploy a model with llm-d on IBM Spyre AI accelerators

Use an IBM Z or IBM Power serving config to set up an LLMInferenceService.

Select the IBM Spyre LLMInferenceServiceConfig for your cluster architecture, name it in your LLMInferenceService, and request the IBM Spyre devices that the model needs.

Prerequisites

Distributed Inference with llm-d is enabled, and you can create an LLMInferenceService in the cluster.
The cluster has IBM Power (ppc64le), IBM Z (s390x) nodes with IBM Spyre accelerators, and the device plugin or Operator for Spyre advertises ibm.com/spyre_pf or ibm.com/spyre_vf on those nodes.
The model you intend to serve is verified for vLLM Spyre on that node architecture.
The cluster can access the model source.
You have installed the OpenShift CLI (oc).

About this task

Red Hat OpenShift AI provides one IBM Spyre configuration for each node architecture. The steps are the same for all three. Only the configuration name and the extended resource change:

Node architecture	Configuration name	Extended resource to request
IBM Power (`ppc64le`)	`v3-6-0-kserve-config-llm-template-ibm-spyre-ppc64le`	`ibm.com/spyre_pf`
IBM Z (`s390x`)	`v3-6-0-kserve-config-llm-template-ibm-spyre-s390x`	`ibm.com/spyre_vf`

Each configuration supplies the vLLM Spyre runtime image and the Spyre environment variables for its architecture.
The IBM Spyre configurations serve a model on a single node.

The example in this procedure serves a Granite model on one Spyre device on IBM Power with the Endpoint Picker (EPP).

Procedure

List the accelerator configurations that the cluster provides:

$ oc get llminferenceserviceconfig -A

The command returns one configuration for each supported AI accelerator, including IBM Spyre configurations:

NAMESPACE                 NAME                                                  READY   REASON   AGE
redhat-ods-applications   v3-6-0-kserve-config-llm-template-ibm-spyre-ppc64le   True             13d
redhat-ods-applications   v3-6-0-kserve-config-llm-template-ibm-spyre-s390x     True             13d
# ...

Confirm the extended resource and the topology that the configuration expects:

$ oc get llminferenceserviceconfig -n redhat-ods-applications \
  -l opendatahub.io/config-type=accelerator \
  -o custom-columns='NAME:.metadata.name,ACCELERATOR:.metadata.annotations.opendatahub\.io/recommended-accelerators,TOPOLOGY:.metadata.annotations.opendatahub\.io/supported-topologies,SUPPORT:.metadata.annotations.opendatahub\.io/support-status'

The IBM Spyre rows name the device resource for each architecture:

NAME                                                  ACCELERATOR            TOPOLOGY                   SUPPORT
v3-6-0-kserve-config-llm-template-ibm-spyre-ppc64le   ["ibm.com/spyre_pf"]   ["workload-single-node"]   <none>
v3-6-0-kserve-config-llm-template-ibm-spyre-s390x     ["ibm.com/spyre_vf"]   ["workload-single-node"]   <none>

To see the image, the environment variables, and the volumes that a configuration supplies, inspect it in full:

$ oc get llminferenceserviceconfig <config_name> -n redhat-ods-applications -o yaml

Confirm that the cluster has nodes with Spyre devices and that the devices are not already allocated:

$ oc get nodes -o custom-columns='NAME:.metadata.name,ARCH:.status.nodeInfo.architecture,SPYRE_PF:.status.capacity.ibm\.com/spyre_pf,SPYRE_VF:.status.capacity.ibm\.com/spyre_vf'

Add the configuration name to the spec.baseRefs list in your LLMInferenceService resource, and request the Spyre devices in the main container. If you are updating an existing service, save its resource definition to a file:

$ oc get llminferenceservice <service_name> -n <namespace> -o yaml > llminferenceservice.yaml

Edit the file and add the configuration name for your architecture. The following example serves a Granite model on IBM Power with one Spyre physical function:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: granite-33-8b-instruct-spyre
spec:
  baseRefs:
    - name: v3-6-0-kserve-config-llm-template-ibm-spyre-ppc64le
  model:
    uri: hf://ibm-granite/granite-3.3-8b-instruct
    name: ibm-granite/granite-3.3-8b-instruct
  replicas: 1
  router:
    scheduler: { }
    route: { }
    gateway: { }
  template:
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: --max-model-len 3072 --max-num-seqs 16
        resources:
          requests:
            cpu: '4'
            memory: 16Gi
            ibm.com/spyre_pf: '1'
          limits:
            cpu: '4'
            memory: 16Gi
            ibm.com/spyre_pf: '1'

Request the same device count in the requests and limits fields.

To serve the model on IBM Z, replace the configuration name with v3-6-0-kserve-config-llm-template-ibm-spyre-s390x and the resource name with ibm.com/spyre_vf.

Apply the updated LLMInferenceService resource in the cluster:
```
$ oc apply -f llminferenceservice.yaml -n <namespace>
```
Check the service and its pods for readiness:
```
$ oc get llminferenceservice <service_name> -n <namespace>
```
The model server compiles the model for the Spyre devices on first start, so the pod can take several minutes to become ready.

Verification

Confirm that the LLMInferenceService and model-serving pod are ready:
```
$ oc get pods -n <namespace>
```

Send a test inference request to the service's OpenAI-compatible endpoint:

TOKEN=$(oc whoami -t)
curl -X POST <endpoint_url>/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "ibm-granite/granite-3.3-8b-instruct",
    "prompt": "What is the capital of France?",
    "max_tokens": 50
  }'

Replace <endpoint_url> with the full endpoint URL for the LLMInferenceService. The response should include generated text in the choices field.

Confirm that EPP is running and routes requests to the model server. Check that the EPP pod is ready:
```
$ oc get pods -n <namespace> | grep router-scheduler
```
In the OpenShift web console, select Observe > Metrics. Query this promql metric, replacing the namespace placeholder value with the namespace where the inference service is running.
```
increase(llm_d_epp_scheduler_attempts_total{status="success", namespace="<namespace>"}[5m])
```

Select Your Language

Select a hardware configuration for llm-d model deployments

Table of Contents

Hardware configurations for LLMInferenceService

Deploy a model with llm-d by selecting a serving configuration

Deploy a model with llm-d on IBM Spyre AI accelerators

Comments

Quick Links

Help

Site Info

Related Sites

About

Red Hat legal and privacy links

Red Hat legal and privacy links

Table of Contents

Hardware configurations for LLMInferenceService

Deploy a model with llm-d by selecting a serving configuration

Deploy a model with llm-d on IBM Spyre AI accelerators

Comments

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links