Red Hat AI Inference - Supported Configurations
Red Hat AI Inference - Supported Product and Hardware Configurations
Red Hat AI Inference (RHAII) is a high-performance model inference platform. It includes the following components:
| Component | Description |
|---|---|
| vLLM | High-throughput inference engine with per-accelerator container images |
| Red Hat AI Model Optimization Toolkit | Model quantization and compression (LLM Compressor), NVIDIA GPUs only |
| llm-d | Kubernetes-native distributed inference framework (Technology Preview) |
| Model as a Service (MaaS) | Coming soon |
For product documentation, see Red Hat AI Inference.
Product and version compatibility
| RHAII Version | 3.3 | 3.4 | 3.5 |
|---|---|---|---|
| vLLM CUDA | v0.13.0 | v0.18.0 | v0.24.0 |
| vLLM ROCm | v0.13.0 | v0.18.0 | v0.24.0 |
| vLLM Power/Z | v0.10.1.1.6 | v0.18.0 | v0.24.0 |
| vLLM Spyre ppc64le | v0.11.0 | v0.18.0 | v0.24.0 |
| vLLM Gaudi | v0.13.0 | v0.17.1 | v0.24.0 |
| vLLM Spyre x86 | v0.12.0 | v0.18.0 | v0.24.0 |
| vLLM Spyre s390x | v0.12.0 | v0.18.0 | v0.24.0 |
| OVMS | v2025.4 | v2026.1.0 | v2026.1.0 |
vLLM
vLLM is the core inference engine in Red Hat AI Inference. Each supported accelerator has a dedicated container image.
Supported AI accelerators
The following tables list the supported AI data center grade accelerators for Red Hat AI Inference 3.4.
NVIDIA CUDA (GA)
Container image: registry.redhat.io/rhaii/vllm-cuda-rhel9:3.4.0
| AI accelerators | Requirements | Architecture |
|---|---|---|
| Turing: T4 | CUDA Toolkit 13.0, NVIDIA Container Toolkit 1.14, NVIDIA GPU Operator 24.3, Python 3.12, PyTorch 2.9.1 | x86, AArch64 |
| Ampere: A2, A10, A16, A30, A40, A100 | Same as above | x86, AArch64 |
| Ada Lovelace: L4, L20, L40, L40S | Same as above | x86, AArch64 |
| Hopper: H100, H200, H20, GH200 | Same as above | x86, AArch64 |
| Blackwell: GB200, GB300, B200, B300, RTX PRO 6000/4500 Blackwell SE | Same as above | x86, AArch64 |
NVIDIA T4 and A100 accelerators do not support FP8 (W8A8) quantization. RHAII 3.4.0 is built with CUDA 13.0; container images are compatible with CUDA 12.9 drivers via NVIDIA Forward Compatibility.
AMD ROCm (GA)
Container image: registry.redhat.io/rhaii/vllm-rocm-rhel9:3.4.0
| AI accelerators | Requirements | Architecture |
|---|---|---|
| MI210 (CDNA 2) | ROCm 6.3.4, AMD GPU Operator 6.2, Python 3.12, PyTorch 2.10 | x86 |
| MI300X (CDNA 3) | ROCm 6.3.4, AMD GPU Operator 6.2, Python 3.12, PyTorch 2.10 | x86 |
| MI325X (CDNA 3) | ROCm 6.3.4, AMD GPU Operator 6.2, Python 3.12, PyTorch 2.10 | x86 |
| MI350X (CDNA 4) | ROCm 7.1, AMD GPU Operator 6.2, Python 3.12, PyTorch 2.10 | x86 |
| MI355X (CDNA 4) | ROCm 7.1, AMD GPU Operator 6.2, Python 3.12, PyTorch 2.10 | x86 |
AMD GPUs support FP8 (W8A8) and GGUF quantization schemes only.
Intel Gaudi 3 (Technology Preview)
Container image: registry.redhat.io/rhaii/vllm-gaudi-rhel9:3.4.0
| AI accelerators | Requirements | Architecture |
|---|---|---|
| Intel Gaudi 3 | Gaudi Software Suite 1.23.0, vllm-gaudi plugin 0.16.0, Python 3.12.9 | x86 |
Google TPU (Technology Preview)
Container image: registry.redhat.io/rhaii/vllm-tpu-rhel9:3.4.0
| AI accelerators | Requirements | Architecture |
|---|---|---|
| Google v4, v5e, v5p, v6e (Trillium) | Python 3.12 | x86 |
IBM Spyre
Container image: registry.redhat.io/rhaii/vllm-spyre-rhel9:3.4.0
| AI accelerators | Requirements | Architecture | Status |
|---|---|---|---|
| IBM Spyre for Power (ppc64le) | Python 3.12.9, PyTorch 2.7.1, vllm-tgis-adapter 0.9.2, vllm-spyre 1.6.1, IBM Spyre Enablement Stack 1.1.1 | ppc64le | GA |
| IBM Spyre for Z (s390x) | Python 3.12, PyTorch 2.7.1, vllm-tgis-adapter 0.9.2, vllm-spyre 1.2.0, IBM Spyre Enablement Stack 1.1.1 | s390x | GA |
| IBM AIU (x86) | Python 3.12, PyTorch 2.7.1, vllm-tgis-adapter 0.9.2, vllm-spyre 1.0.2, IBM Spyre Enablement Stack 1.1.1 | x86 | Technology Preview |
Note: For Red Hat AI Inference 3.5, Extended Update Support (EUS) is not available on the IBM Spyre AI Accelerator. EUS support for Spyre is planned for a future release.
Spyre Enablement Stack release compatibility
| IBM Spyre Enablement Stack | RHOAI Release | RHAII Release |
|---|---|---|
| IBM Spyre Enablement Stack 1.0 | RHOAI 3.3 | RHAII 3.2.5 (stable) |
| IBM Spyre Enablement Stack 1.1.1 | RHOAI 3.4 | RHAII 3.4.0 |
AWS Neuron (Technology Preview)
Container image: registry.redhat.io/rhaii/vllm-neuron-rhel9:3.4.0
| AI accelerators | Requirements | Architecture |
|---|---|---|
| AWS Inferentia2 (Inf2), AWS Trainium (Trn1, Trn1n, Trn2) | AWS Neuron SDK 2.x, Python 3.12, vllm-neuron plugin | x86 |
x86 CPU (GA)
Container image: registry.redhat.io/rhaii/vllm-cpu-rhel9:3.4.0
| Supported processors | Requirements | Architecture |
|---|---|---|
| Intel Xeon and AMD EPYC with AVX2 (minimum), AVX512, or AVX512 AMX | Python 3.12, minimum 16 GB RAM (32 GB recommended) | x86 |
CPU inference is best suited for models under ~3 billion parameters. The container automatically detects and uses the best available instruction set.
Supported quantization formats for vLLM
| Format | NVIDIA accelerators | AMD accelerators |
|---|---|---|
| INT8 (W8A8) | Turing through Hopper (not Blackwell) | MI210, MI300X, MI325X, MI350X, MI355X |
| INT4 (W4A16) | Ampere through Blackwell | MI300X, MI325X, MI350X, MI355X |
| FP8 (W8A8) | Ada Lovelace through Blackwell | MI300X, MI325X, MI350X, MI355X |
| NVFP4 | Blackwell only | Not supported |
| GGUF | Not supported | Supported |
Red Hat AI Model Optimization Toolkit (LLM Compressor)
The Red Hat AI Model Optimization Toolkit provides model quantization and compression capabilities using the LLM Compressor library. It is supported on NVIDIA GPUs only and is packaged separately in the model-opt-cuda-rhel9 container image.
Container image: registry.redhat.io/rhaii/model-opt-cuda-rhel9:3.4.0
Version compatibility
| RHAII Version | LLM Compressor version |
|---|---|
| 3.4.0 | v0.10.0.1 |
| 3.3 | v0.9.0.1 |
| 3.2.5 | v0.8.1 |
| 3.2.4 | v0.8.1 |
| 3.2.3 | v0.8.1 |
| 3.2.2 | v0.7.1 |
Supported accelerators
The Model Optimization Toolkit supports NVIDIA CUDA GPUs only:
| AI accelerators | Supported quantization |
|---|---|
| Ampere: A2, A10, A16, A30, A40, A100 | INT8 (W8A8), INT4 (W4A16) |
| Ada Lovelace: L4, L20, L40, L40S | INT8 (W8A8), INT4 (W4A16), FP8 (W8A8) |
| Hopper: H100, H200, H20, GH200 | INT8 (W8A8), INT4 (W4A16), FP8 (W8A8) |
| Blackwell: GB200, GB300, B200, B300 | INT4 (W4A16), FP8 (W8A8), NVFP4 |
NVIDIA Turing (T4) does not support LLM Compressor. AMD, Intel Gaudi, Google TPU, IBM Spyre, AWS Neuron, and x86 CPU are not supported.
Distributed Inference with llm-d (Technology Preview)
llm-d is a Kubernetes-native distributed inference framework for serving LLMs at scale. For detailed component versions, see the llm-d release components page.
Supported platforms
| Platform | Supported versions |
|---|---|
| Red Hat OpenShift Container Platform | 4.19+ |
| Azure Kubernetes Service (AKS) | Kubernetes 1.33+ |
| CoreWeave Kubernetes Service (CKS) | Kubernetes 1.33+ |
Supported AI accelerators for llm-d
| Accelerator | Status |
|---|---|
| NVIDIA CUDA GPUs (Ampere, Ada Lovelace, Hopper, Blackwell) | Supported |
| AMD ROCm GPUs (MI300X, MI325X, MI350X, MI355X) | Supported |
Model as a Service (MaaS)
Coming soon.
Supported deployment environments
| Environment | Supported versions |
|---|---|
| Red Hat OpenShift Container Platform (self-managed) | 4.14+ |
| Red Hat OpenShift Service on AWS (ROSA) | 4.14+ |
| Red Hat Enterprise Linux AI (RHEL AI) | 3.0+ |
| Red Hat Enterprise Linux (RHEL) | 9.2+ |
| Linux (non-RHEL) | Third-party support policy |
| Kubernetes (non-OCP) | Third-party support policy |
OCP software prerequisites for GPU deployments
| Component | Version | Documentation |
|---|---|---|
| NVIDIA GPU Operator | 24.3+ | Documentation |
| NVIDIA Container Toolkit | 1.14+ | Documentation |
| AMD GPU Operator | 6.2+ | Documentation |
| Intel Gaudi Base Operator | See Intel docs | Documentation |
Comments