Distributed Inference with llm-d: Release Components Version
Table of Contents
# Released Components Version
| RHOAI Version | RHAII version | llm-d upstream version | vLLM | Dates | |
|---|---|---|---|---|---|
| General Availability (GA) | RHOAI 3.5 | RHAII 3.5 | 0.9 | v0.24 | August 27, 2026 |
| General Availability (GA) | RHOAI 3.4 | RHAII 3.4 | 0.6 | v0.13.0 | May 14, 2026 |
| General Availability (GA) | RHOAI 3.3 | RHAII 3.3 | 0.4 | v0.13.0 | March 5, 2026 |
| Technology Preview (TP) | RHOAI 2.25 | RHAII 3.2.2 | 0.2 | v0.10.1 | October 23, 2025 |
Components level checklist
| Component | Version |
|---|---|
| OpenShift | 4.20+ |
API Compatibility
| API format | Endpoint | Status |
|---|---|---|
| OpenAI-compatible Chat Completions | /v1/chat/completions |
Supported |
| OpenAI-compatible Completions | /v1/completions |
Supported |
| Anthropic-compatible Messages API | /v1/messages |
Supported |
| OpenAI-compatible Responses API | /v1/responses |
Not supported |
| Provider-specific APIs | - | Not supported |
Note: Per-request token usage (prompt_tokens, completion_tokens) is returned in the usage field for text inputs.
GA RHOAI 3.5
Supported configuration(s):
OpenShift: 4.20+
Distributed Inference with llm-d — Supported Well-Lit Paths
A well-lit path is a documented, tested, and benchmarked deployment pattern that reduces adoption risk and maintenance cost.
| Well-lit path | RHOAI 3.5 maturity | Support Boundary |
|---|---|---|
| Intelligent Inference Scheduling | Generally Available | Supported for Valid Configurations |
| P/D Disaggregation | Technology Preview | Technology Preview - Could be Supported through Support Exception |
| Wide Expert Parallelism | Developer Preview | Developer Preview; not recommended for production workloads |
| KV-cache Offload to CPU | Generally Available | Supported |
| KV-cache Offload to a POSIX Filesystem | Developer Preview | Developer Preview; not recommended for production workloads |
NVIDIA: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended NVIDIA Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most appropriate GPU. | H100, H200, B200, B300, A100 | Standard DC Ethernet (25/100 GbE) | Not Applicable |
| P/D Disaggregation | Separate prefill and decode compute stages. | H100, H200, B200, B300 | HPC Fabric with RDMA • InfiniBand • RoCE |
Not Applicable |
| KV-cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | H100, H200, B200, B300, A100 | PCIe 5+ | Not Applicable |
| KV-cache Offload to a POSIX Filesystem | Extend KV-cache capacity beyond GPU and CPU memory. | H100, H200, B200, B300 | Storage-network dependent | ReadWriteMany volume providing standard POSIX file access |
| Wide Expert Parallelism | Distribute mixture-of-experts models across multiple GPUs. | H100, H200, B200, B300 | HPC Fabric with RDMA • InfiniBand • RoCE |
High-speed NVMe SSDs |
AMD: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended AMD Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most appropriate GPU. | MI300X | Standard DC Ethernet (25/100 GbE) | Not Applicable |
| KV-cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | MI300X | PCIe 5+ | Not Applicable |
| KV-cache Offload to a POSIX Filesystem | Extend KV-cache capacity beyond GPU and CPU memory. | MI300X | Storage-network dependent | ReadWriteMany volume providing standard POSIX file access |
Mixed Accelerator Architectures
Mixed accelerator architecture combines different hardware generations from the same vendor within one inference cluster, for example NVIDIA H200 and B200.
| Inference architecture | Mixed Accelerator Support |
|---|---|
| Intelligent Inference Scheduling | Supported |
| P/D Disaggregation | Not supported |
| Wide Expert Parallelism | Not supported |
Approximate and Precise Cache Scoring
| Selection criterion | Approximate scoring (default) | Precise scoring (experimental) |
|---|---|---|
| How it works | Predicts cache locality from request traffic patterns | Uses real-time cache state across the cluster |
| External dependencies | None | Requires access to model tokenizer artifacts |
| Disconnected support | Yes | No, unless required tokenizer artifacts are mirrored internally and the configuration is validated |
| Status | GA and recommended for most deployments | Experimental |
| Best suited for | Standard workloads and benchmarks | Workloads with highly variable prefix patterns |
Approximate scoring delivers strong cache-hit rates across standard benchmarks and production workloads without additional configuration.
GA RHOAI 3.4
Supported configuration(s):
Note:
- P/D Disaggregation : Technology Preview
- Wide Expert-Parallelism multi-node: Developer Preview
- Wide Expert-Parallelism on Blackwell B200: Not available but can be provided as a Developer Preview
- Multi node on GB200 is not supported
Hardware and Accelerator support per "Well-Lit" paths for distributed inference with llm-d
A well-lit path is a documented, tested, and benchmarked deployment pattern that reduces adoption risk and maintenance cost.
NVIDIA: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended NVIDIA Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | H100, H200, B200, A100 | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| P/D Disaggregation | Separate prefill and decode compute stages. | H100, H200, B200 | HPC Fabric with RDMA • InfiniBand • RoCE |
Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | H100, H200, B200, A100 | PCIe 5+ | Not Applicable |
| Wide Expert Parallelism (WEP) | Distribute MoE models across many GPUs. | H100, H200, B200 | HPC Fabric with RDMA • InfiniBand • RoCE |
High-speed NVMe SSDs. |
AMD: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended AMD Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | MI300X | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | MI300X | PCIe 5+ | Not Applicable |
Mixed Accelerator Architectures
Mixed accelerator refers to combining different hardware generations from the same vendor (e.g., NVIDIA H200 and B200) within the same inference cluster.
| Feature | Mixed Accelerator Support |
|---|---|
| Intelligent Inference Scheduling | Supported |
| P/D Disaggregation | Not supported |
| Wide Expert Parallelism | Not supported |
Intelligent Inference Scheduling
The inference scheduler balances requests across load and prefix cache locality to optimize throughput and latency. Two cache scoring methods are available:
| Approximate (default) | Precise (experimental) | |
|---|---|---|
| How it works | Predicts cache locality from request traffic patterns | Uses real-time cache state across the cluster |
| External dependencies | None | Requires HuggingFace connectivity to download the model tokenizer |
| Air-gapped support | Yes | No |
| Status | GA, recommended for most deployments | Experimental, GA expected in a future release |
| Best for | Standard workloads and benchmarks | Workloads with highly variable prefix patterns |
The approximate scorer delivers strong cache hit rates across standard benchmarks and production workloads with no additional configuration.
GA RHOAI 3.3
Supported configuration(s):
Note:
- P/D Disaggregation : Technology Preview
- Wide Expert-Parallelism multi-node: Developer Preview
- Wide Expert-Parallelism on Blackwell B200: Not available but can be provided as a Developer Preview
- Multi node on GB200 is not supported
Hardware and Accelerator support per "Well-Lit" paths for distributed inference with llm-d
A well-lit path is a documented, tested, and benchmarked deployment pattern that reduces adoption risk and maintenance cost.
NVIDIA: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended NVIDIA Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | H100, H200, B200, A100 | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| P/D Disaggregation | Separate prefill and decode compute stages. | H100, H200, B200 | HPC Fabric with RDMA • InfiniBand • RoCE |
Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | H100, H200, B200, A100 | PCIe 5+ | Not Applicable |
| Wide Expert Parallelism (WEP) | Distribute MoE models across many GPUs. | H100, H200, B200 | HPC Fabric with RDMA • InfiniBand • RoCE |
High-speed NVMe SSDs. |
AMD: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended AMD Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | MI300X | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | MI300X | PCIe 5+ | Not Applicable |
Mixed Accelerator Architectures
Mixed accelerator refers to combining different hardware generations from the same vendor (e.g., NVIDIA H200 and B200) within the same inference cluster.
| Feature | Mixed Accelerator Support |
|---|---|
| Intelligent Inference Scheduling | Supported |
| P/D Disaggregation | Not supported |
| Wide Expert Parallelism | Not supported |
Intelligent Inference Scheduling
The inference scheduler balances requests across load and prefix cache locality to optimize throughput and latency. Two cache scoring methods are available:
| Approximate (default) | Precise (experimental) | |
|---|---|---|
| How it works | Predicts cache locality from request traffic patterns | Uses real-time cache state across the cluster |
| External dependencies | None | Requires HuggingFace connectivity to download the model tokenizer |
| Air-gapped support | Yes | No |
| Status | GA, recommended for most deployments | Experimental, GA expected in a future release |
| Best for | Standard workloads and benchmarks | Workloads with highly variable prefix patterns |
The approximate scorer delivers strong cache hit rates across standard benchmarks and production workloads with no additional configuration.
Tech Preview - RHOAI 2.25
Supported configuration:
Note: WIDE EP multi-node support is included in this Tech Preview, but it may not function as expected and is not yet stable.
Hardware and Accelerator support per "Well-Lit" paths for distributed inference with llm-d
A well-lit path is a documented, tested, and benchmarked deployment pattern that reduces adoption risk and maintenance cost.
NVIDIA: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended NVIDIA Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | H100, H200, B200, A100 | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| P/D Disaggregation | Separate prefill and decode compute stages. | H100, H200, B200 | HPC Fabric with RDMA • InfiniBand • RoCE |
Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | H100, H200, B200, A100 | PCIe 5+ | Not Applicable |
| Wide Expert Parallelism (WEP) | Distribute MoE models across many GPUs. | H100, H200, B200, ~~GB200~~ NVL72 | HPC Fabric with RDMA • InfiniBand • RoCE |
High-speed NVMe SSDs. |
AMD: Hardware & Accelerator Matrix
| Well-Lit Path | Primary Goal | Recommended AMD Hardware | Networking/Interconnect Requirement | Storage |
|---|---|---|---|---|
| Intelligent Inference Scheduling | Route requests to the most optimal GPU. | MI300X | Standard DC Ethernet (25/100 GbE) | Local SSD (NVMe Recommended) |
| KV Cache Management (Local CPU Offload) | Increase throughput by offloading KV cache to CPU RAM. | MI300X | PCIe 5+ | Not Applicable |
Mixed Accelerator Architectures
Mixed accelerator refers to combining different hardware generations from the same vendor (e.g., NVIDIA H200 and B200) within the same inference cluster.
| Feature | Mixed Accelerator Support |
|---|---|
| Intelligent Inference Scheduling | Supported |
| P/D Disaggregation | Not supported |
| Wide Expert Parallelism | Not supported |
Intelligent Inference Scheduling
The inference scheduler balances requests across load and prefix cache locality to optimize throughput and latency. Two cache scoring methods are available:
| Approximate (default) | Precise (experimental) | |
|---|---|---|
| How it works | Predicts cache locality from request traffic patterns | Uses real-time cache state across the cluster |
| External dependencies | None | Requires HuggingFace connectivity to download the model tokenizer |
| Air-gapped support | Yes | No |
| Status | GA, recommended for most deployments | Experimental, GA expected in a future release |
| Best for | Standard workloads and benchmarks | Workloads with highly variable prefix patterns |
The approximate scorer delivers strong cache hit rates across standard benchmarks and production workloads with no additional configuration.
Comments