KEDA Autoscaling for Distributed Inference with llm-d

Updated -

Autoscaling for Distributed inference with llm-d

Red Hat OpenShift AI supports two scaling approaches for LLMInferenceService workloads: static allocation and KEDA autoscaling.

Static allocation is a fixed-capacity configuration and does not use an autoscaler. KEDA autoscaling is the only approach that adjusts capacity in response to demand.

The scaling approach is configured separately for each workload. A standard deployment has one workload. A disaggregated prefill/decode deployment has two workloads: the decode workload, which is configured at the top level of spec, and the prefill workload, which is configured under spec.prefill. Each workload uses one of the following mutually exclusive fields:

Approach Workload field Resources created by the controller
Static allocation replicas No additional resources beyond the workload (Deployment or LeaderWorkerSet)
KEDA autoscaling scaling (with scaling.keda) A KEDA ScaledObject, which creates a Horizontal Pod Autoscaler (HPA)

The replicas and scaling fields are mutually exclusive on the same workload. The API server rejects an LLMInferenceService resource that sets both fields. If neither field is set, the workload runs one replica.

Scaling Approaches and Features

Static allocation

With static allocation, the workload maintains the number of replicas specified in the replicas field regardless of demand. To change capacity, update the value of the field.

Static allocation provides the following benefits:

  • Predictable capacity: No scaling events occur, and resource allocation changes only when you change the configuration.
  • Simplified operations: No autoscaling components require installation, monitoring, or troubleshooting.
  • Reserved resources: GPU resources for every configured replica remain allocated to the workload.

In a disaggregated deployment, spec.replicas sets the number of decode replicas and spec.prefill.replicas sets the number of prefill replicas, so that you can size each phase independently.

Use static allocation when traffic is predictable, when you plan capacity in advance, or when the operational overhead of autoscaling is not justified.

KEDA autoscaling

KEDA autoscaling uses Kubernetes Event-Driven Autoscaling (KEDA), which is provided on OpenShift by the Custom Metrics Autoscaler Operator. You define triggers in spec.scaling.keda.triggers. The KServe controller creates a KEDA ScaledObject resource from this configuration, and KEDA creates and manages an HPA that scales the workload between spec.scaling.minReplicas and spec.scaling.maxReplicas.

KEDA supports many trigger types. The following trigger types are the most relevant for LLM inference:

  • Prometheus: Queries metrics from the OpenShift monitoring stack (Thanos Querier) or from another Prometheus endpoint, for example, the Endpoint Picker (EPP) queue depth.
  • Cron: Scales the workload on a schedule, for example, to match business-hours traffic.
  • Other scalers: Message queues, databases, cloud provider services, and external scalers.

Each trigger defines its own threshold and metadata. For Prometheus triggers, OpenShift AI automatically creates the TriggerAuthentication resource that provides the credentials for querying OpenShift monitoring. When multiple triggers are configured, the HPA uses the highest replica count calculated across all triggers.

To enable scale-to-zero, set spec.scaling.keda.idleReplicaCount to 0. While all triggers are inactive, KEDA keeps the workload at zero replicas. When a trigger becomes active, KEDA scales the workload directly to minReplicas, and the HPA manages scaling from that point.

Use KEDA autoscaling when demand varies, when scaling must respond to custom signals such as queue depth or a schedule, or when GPU resources must be released during idle periods.

Choosing a scaling approach

Use KEDA autoscaling when:

  • Traffic varies and capacity must follow demand automatically.
  • Scaling must respond to Prometheus metrics such as EPP queue depth, to a schedule, or to external signals.
  • The workload must scale to zero during idle periods to release GPU resources.
  • You require control over thresholds, polling, cooldown, and HPA stabilization behavior.
  • KEDA is installed on the cluster, or you can install it.

Use static allocation when:

  • Traffic is predictable and varies little.
  • You prefer planned capacity to reactive scaling.
  • The workload requires fixed, guaranteed GPU capacity without scaling events.
  • You do not want to install or operate KEDA.

Scaling feature availability matrix

The following matrix compares the features of static allocation and KEDA autoscaling for LLMInferenceService workloads.

Feature Static allocation KEDA autoscaling
Automatic scaling based on demand Not supported Supported
Automatic scale-to-zero during idle periods Not supported. You can set replicas to 0 manually. Supported (idleReplicaCount: 0)
Automatic scale-from-zero Not supported Supported when a trigger reports activity while no model server pods are running
Independent prefill and decode sizing Supported (spec.replicas, spec.prefill.replicas) Supported (spec.scaling, spec.prefill.scaling)
GPU release during idle periods Only when you reduce replicas manually Automatic on scale-down and scale-to-zero
Trigger types Not applicable Prometheus, cron, and other KEDA scalers
Stabilization windows Not applicable Supported (advanced.horizontalPodAutoscalerConfig.behavior)
TriggerAuthentication Not applicable Created automatically when autoscaling with a Prometheus trigger is configured. No manual setup is required.
Custom Metrics Autoscaler Operator (KEDA) Not required Required

Implementation notes

Scale-to-zero and scale-from-zero

Scale-to-zero requires idleReplicaCount: 0 and a minReplicas value of at least 1. The minReplicas field does not accept 0.

To scale up from zero, at least one trigger must report activity while no model server pods are running. Metrics that are emitted by model server pods are unavailable at zero replicas and cannot initiate a scale-up. Use one of the following trigger sources instead:

  • A Prometheus query on metrics from a component that continues to run at zero replicas, such as the EPP flow control queue
  • A cron trigger for scheduled activation

TriggerAuthentication

A Prometheus trigger that queries the OpenShift monitoring stack through Thanos Querier must authenticate with a bearer token. When you configure KEDA autoscaling with a Prometheus trigger in an LLMInferenceService resource, OpenShift AI automatically creates a TriggerAuthentication resource named inference-prometheus-auth in the same namespace. No manual setup is required. The trigger references this resource through the authenticationRef field. Static allocation does not require authentication.

KEDA prerequisite

A cluster administrator must install the Custom Metrics Autoscaler Operator and create a KedaController resource before KEDA autoscaling can be configured. Static allocation does not require either.

Disaggregated scaling

Both approaches support disaggregated prefill/decode deployments. The approach is selected separately for each workload:

  • With static allocation, spec.replicas sets the number of decode replicas and spec.prefill.replicas sets the number of prefill replicas.
  • With KEDA autoscaling, spec.scaling and spec.prefill.scaling each produce a separate ScaledObject resource, named <name>-kserve-keda and <name>-kserve-prefill-keda, with independent triggers.

If both workloads use autoscaling, both workloads must use scaling.keda.

Stabilization windows

With KEDA autoscaling, stabilization windows are configured through the HPA behavior settings. A stabilization window prevents rapid fluctuations in the replica count by causing the HPA to use the most conservative recommendation within the window. Stabilization windows apply to scaling between minReplicas and maxReplicas. They do not apply to scaling up from zero or to scale-to-zero, which are controlled by pollingInterval and cooldownPeriod.

Initial stabilization window values

The following values are initial values for tuning and have not been validated. Adjust them for your model, hardware, and traffic pattern.

Workload pattern Scale-up window Scale-down window
Bursty 30 s 300 s
Latency-sensitive 15 s 180 s
Throughput-oriented 60 s 600 s
Steady-state 45 s 450 s

GPU cost optimization

GPU resources are released only when the replica count is reduced:

  • KEDA autoscaling releases GPU resources automatically when it scales down under reduced load and when it scales to zero with idleReplicaCount: 0.
  • Static allocation retains GPU resources for every configured replica until you reduce replicas. Setting replicas to 0 releases all GPU resources, but the workload does not scale up again until you increase the value.

Prerequisites and Installation

KEDA prerequisites for autoscaling

KEDA autoscaling for LLMInferenceService workloads, including scale-to-zero, requires KEDA. On OpenShift, KEDA is provided by the Custom Metrics Autoscaler Operator. Static allocation does not require KEDA.

Red Hat OpenShift AI does not install KEDA. A cluster administrator must install the Custom Metrics Autoscaler Operator and create a KedaController resource before spec.scaling.keda can be configured. This approach prevents conflicts with existing KEDA installations and gives administrators control over the autoscaling infrastructure.

If KEDA is not installed, the ScaledObject custom resource definition (CRD) is not available. The controller cannot create the ScaledObject resource, and the workload is not autoscaled.

KEDA components and resources

ScaledObject CRD

Defines the scaling configuration, including the target workload, replica limits, triggers, polling interval, and cooldown period. The KServe controller creates a ScaledObject resource from the spec.scaling configuration of each workload.

KEDA operator

Watches ScaledObject resources. The operator manages activation, which is scaling between zero (or idleReplicaCount) and minReplicas. It also creates and manages the HPA that scales the workload between minReplicas and maxReplicas.

Metrics API server

Exposes trigger metrics through the Kubernetes external metrics API, from which the HPA reads them.

Admission webhooks

Validate ScaledObject and TriggerAuthentication resources when they are created or updated.

TriggerAuthentication and ClusterTriggerAuthentication CRDs

Store or reference the credentials that triggers use to query authenticated metric sources.

Installed pods

After the operator is installed and the KedaController resource is created, the following pods run in the openshift-keda namespace:

  • custom-metrics-autoscaler-operator: Manages the KEDA installation.
  • keda-operator: Reconciles ScaledObject resources and manages the HPAs.
  • keda-metrics-apiserver: Exposes external metrics to the HPA.
  • keda-admission: Runs the admission webhooks.

The keda-operator, keda-metrics-apiserver, and keda-admission pods are created only after the KedaController resource exists.

Requirements

  • Cluster administrator access to install Operators from OperatorHub (Operator Lifecycle Manager)
  • An OpenShift Container Platform version that is supported by your Red Hat OpenShift AI release
  • No other KEDA installation on the cluster, such as the community KEDA Operator or an upstream Helm installation. A cluster supports only one KEDA installation, because only one external metrics API server can be registered.

RBAC considerations

Installing the Operator requires the cluster-admin role, because the Operator creates cluster-scoped CRDs and registers an aggregated API service.

Users who configure autoscaling require permission to create and update LLMInferenceService resources in their namespace. Users do not create ScaledObject resources directly, because the KServe controller creates them. The odh-model-controller creates the TriggerAuthentication resources for Prometheus triggers. Users who create additional TriggerAuthentication resources for custom authentication require permission to create those resources and any secrets that they reference.

Reasons that OpenShift AI does not install KEDA

Many clusters already run KEDA for other workloads. Automatic installation could result in duplicate Operators, version conflicts, or conflicting external metrics API registrations. Manual installation enables administrators to coordinate KEDA with their existing infrastructure and to select the installation and update method.

Installing the Custom Metrics Autoscaler Operator

Install the Custom Metrics Autoscaler Operator and create a KedaController resource to enable KEDA autoscaling for LLM inference workloads.

Prerequisites

  • You have cluster administrator privileges.
  • No other KEDA installation is present on the cluster.

Procedure

  1. Log in to the OpenShift web console as a cluster administrator.
  2. Navigate to Operators → OperatorHub.
  3. Search for Custom Metrics Autoscaler.
  4. Select Custom Metrics Autoscaler provided by Red Hat, and click Install.
  5. On the Install Operator page, configure the following settings:
    • Update channel: stable
    • Installation mode: A specific namespace on the cluster
    • Installed Namespace: openshift-keda. The namespace is created if it does not exist.
    • Update approval: Automatic or Manual
  6. Click Install, and wait until the Operator status is Succeeded.

  7. Create the KedaController resource, which installs the KEDA components:

    1. Navigate to Operators → Installed Operators → Custom Metrics Autoscaler.
    2. On the KedaController tab, click Create KedaController.
    3. Keep the name keda and the namespace openshift-keda, and click Create.

    Alternatively, apply the following resource by using the CLI:

    apiVersion: keda.sh/v1alpha1
kind: KedaController
metadata:
 name: keda
 namespace: openshift-keda
spec:
 watchNamespace: ""   // an empty value watches all namespaces

    KEDA watches ScaledObject resources only in the namespaces specified by watchNamespace. Leave the value empty so that KEDA can autoscale LLMInferenceService workloads in all namespaces.

Verification

  1. Verify that the KEDA pods are running:

    $ oc get pods -n openshift-keda

    Example output:

    NAME                                                  READY   STATUS    RESTARTS   AGE
custom-metrics-autoscaler-operator-5d7f8c8d8b-x9k2m   1/1     Running   0          5m
keda-admission-6f9c7d8b6d-2kqzv                       1/1     Running   0          90s
keda-metrics-apiserver-7d5c6f8f7b-p4j9n               1/1     Running   0          90s
keda-operator-6c8b5d7f5b-t8w4p                        1/1     Running   0          90s

    If only the custom-metrics-autoscaler-operator pod is running, the KedaController resource has not been created.

  2. Verify that the CRDs are installed:

    $ oc get crd scaledobjects.keda.sh triggerauthentications.keda.sh

KEDA installation reference

This reference provides installation options, version compatibility, namespace conventions, and architectural decision rationale for the Custom Metrics Autoscaler operator (KEDA).

Installation methods

Method Description Recommended for
OperatorHub (Custom Metrics Autoscaler) Installation through the web console or the CLI (OLM Subscription) Most OpenShift clusters (supported)
GitOps Declarative Subscription and KedaController manifests Clusters that are managed through GitOps
Upstream KEDA Helm chart or community Operator Upstream distribution Not recommended on OpenShift. This method is not supported by Red Hat and is not managed by OLM.

For every installation method, you must also create a KedaController resource after the Operator is installed.

Version compatibility

Use a Custom Metrics Autoscaler Operator version that is supported on your OpenShift Container Platform version. The Custom Metrics Autoscaler Operator is the Red Hat distribution of upstream KEDA 2.x, and its version corresponds to the upstream KEDA version that it includes.

Namespace conventions

  • The Custom Metrics Autoscaler Operator must be installed in the openshift-keda namespace.
  • The installation namespace does not restrict the namespaces that KEDA manages. The KedaController.spec.watchNamespace field determines the managed namespaces. An empty value includes all namespaces.

Resource requirements

The resource consumption of the KEDA components is small compared with LLM workloads. Resource consumption increases with the number of ScaledObject resources, the number of triggers per resource, and the polling frequency. You can configure resource requests and limits for the KEDA operator, the metrics API server, and the admission webhook in the KedaController specification.

RBAC

Activity Required permission
Install the Operator and create the KedaController resource cluster-admin
KEDA runtime operations Managed by the Operator. Includes managing HPAs, reading the scale subresource of target workloads, and reading the secrets that TriggerAuthentication resources reference.
Configure autoscaling on an LLMInferenceService resource Permission to create and update LLMInferenceService resources in the namespace. The KServe controller creates the ScaledObject resource.
Create the inference-prometheus-auth authentication resources None. The odh-model-controller creates these resources.
Create an additional TriggerAuthentication resource for custom authentication Permission to create TriggerAuthentication resources and secrets in the namespace

Additional resources

Authentication

TriggerAuthentication for OpenShift Prometheus queries

TriggerAuthentication resources provide the credentials that KEDA Prometheus triggers use to query the OpenShift monitoring stack.

IMPORTANT: When you configure KEDA autoscaling with a Prometheus trigger in an LLMInferenceService resource, Red Hat OpenShift AI automatically creates the inference-prometheus-auth TriggerAuthentication resource and its supporting resources in the same namespace. You do not need to create authentication resources manually. In each Prometheus trigger, set authModes: bearer and set authenticationRef.name to inference-prometheus-auth.

Authentication requirement

The OpenShift monitoring stack exposes Prometheus queries through Thanos Querier. Thanos Querier requires bearer token authentication for all queries to enforce role-based access control.

When KEDA evaluates a Prometheus trigger, it sends a query to Thanos Querier. If the query does not include an authorized bearer token, Thanos Querier rejects the request with 401 Unauthorized or 403 Forbidden. KEDA reports the trigger as failed, and the workload is not scaled.

A TriggerAuthentication resource provides the bearer token and the CA bundle that KEDA includes in each query. The token belongs to a service account that has read access to metrics in the workload namespace.

Automatic creation of authentication resources

Authentication resources are created automatically as part of configuring autoscaling. When you create or update an LLMInferenceService resource that declares at least one trigger of type prometheus in spec.scaling.keda.triggers or spec.prefill.scaling.keda.triggers, the odh-model-controller creates the following resources in the namespace of the LLMInferenceService resource. All of the resources are named inference-prometheus-auth:

  • ServiceAccount: The identity that KEDA uses for Prometheus queries.
  • Secret: A secret of type kubernetes.io/service-account-token that contains the bearer token and the CA certificate for the service account.
  • Role: Grants read access to pods and pod metrics in the namespace.
  • RoleBinding: Binds the service account to the role.
  • TriggerAuthentication: References the token and CA certificate in the secret.

No manual step is required. The resources are available before KEDA evaluates the trigger, and they are removed automatically when no autoscaled service in the namespace requires them. For details of each resource, see the TriggerAuthentication resource reference section below.

Referencing the TriggerAuthentication resource from a trigger

The odh-model-controller does not modify the triggers in the LLMInferenceService specification. Each Prometheus trigger that queries OpenShift monitoring must reference the TriggerAuthentication resource and use the Thanos Querier tenancy port (9092) with the workload namespace:

triggers:
- type: prometheus
  metadata:
    serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
    namespace: llm-workloads
    query: sum(llm_d_epp_flow_control_queue_size{inference_pool="granite-chat-inference-pool"})
    threshold: "1"
    authModes: bearer
  authenticationRef:
    name: inference-prometheus-auth

serverAddress

The Thanos Querier tenancy port, which restricts queries to the namespace specified in the namespace field.

namespace

The namespace of the LLMInferenceService resource. The automatically created role grants access to this namespace only.

authModes: bearer

Configures KEDA to send the token as a bearer token.

authenticationRef.name

The name of the automatically created TriggerAuthentication resource.

Verifying TriggerAuthentication

  1. After you deploy an LLMInferenceService resource with a Prometheus trigger, verify that the TriggerAuthentication resource exists:

    $ oc get triggerauthentication inference-prometheus-auth -n <namespace>

    Example output:

    NAME                        PODIDENTITY   SECRET                      ENV   VAULTADDRESS
inference-prometheus-auth                 inference-prometheus-auth

  2. If the resource does not exist, review the odh-model-controller logs for reconciliation errors:

    $ oc logs -n redhat-ods-applications deployment/odh-model-controller | grep -i triggerauthentication

    The TriggerAuthentication resource is not created if KEDA is not installed, because the TriggerAuthentication CRD is not available. In this case, install the Custom Metrics Autoscaler Operator and create the KedaController resource.

Security considerations

Least privilege

The role grants only get access to pods and read access to pod and node metrics in the workload namespace. Queries through the tenancy port are restricted to that namespace.

Namespace scope

All authentication resources are created in the workload namespace. No resources are created in the openshift-monitoring namespace.

TLS verification

The ca parameter provides the CA bundle that KEDA uses to verify the Thanos Querier certificate.

Token lifetime

A token in a secret of type kubernetes.io/service-account-token is long-lived and is not rotated automatically. To rotate the token, delete the inference-prometheus-auth secret. The odh-model-controller re-creates the secret, Kubernetes issues a new token, and KEDA uses the new token for subsequent queries.

TriggerAuthentication resource reference

When you configure KEDA autoscaling with a Prometheus trigger in an LLMInferenceService resource, the odh-model-controller automatically creates the following resources in the same namespace. These resources provide the credentials that KEDA requires to query the OpenShift monitoring stack. You do not need to create them.

Automatically created resources

Resource type Name Namespace Purpose
ServiceAccount inference-prometheus-auth Same as the LLMInferenceService Identity for Prometheus queries
Secret inference-prometheus-auth Same as the LLMInferenceService Bearer token and CA certificate
Role inference-prometheus-auth Same as the LLMInferenceService Read access to pods and pod metrics
RoleBinding inference-prometheus-auth Same as the LLMInferenceService Binds the service account to the role
TriggerAuthentication inference-prometheus-auth Same as the LLMInferenceService References the token and CA certificate for KEDA

All resources have the label odh-model-controller: keda-reconciler.

ServiceAccount

The service account provides the identity that KEDA uses for Prometheus queries.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: inference-prometheus-auth
  namespace: llm-workloads
  labels:
    odh-model-controller: keda-reconciler

Secret

The secret is of type kubernetes.io/service-account-token. Kubernetes populates the token and ca.crt fields for the service account that is specified in the kubernetes.io/service-account.name annotation.

apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: inference-prometheus-auth
  namespace: llm-workloads
  labels:
    odh-model-controller: keda-reconciler
  annotations:
    kubernetes.io/service-account.name: inference-prometheus-auth
data:
  token: <base64-encoded-token>
  ca.crt: <base64-encoded-ca-certificate>

token: The bearer token that KEDA includes in the Authorization header of Prometheus queries.

ca.crt: The CA certificate that KEDA uses to verify the Thanos Querier TLS certificate.

Role

The role grants the read access that the Thanos Querier tenancy port (9092) requires for queries in the namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: inference-prometheus-auth
  namespace: llm-workloads
  labels:
    odh-model-controller: keda-reconciler
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]

RoleBinding

The role binding binds the service account to the role in the same namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: inference-prometheus-auth
  namespace: llm-workloads
  labels:
    odh-model-controller: keda-reconciler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: inference-prometheus-auth
subjects:
- kind: ServiceAccount
  name: inference-prometheus-auth
  namespace: llm-workloads

TriggerAuthentication

The TriggerAuthentication resource maps the fields of the secret to KEDA authentication parameters.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: inference-prometheus-auth
  namespace: llm-workloads
  labels:
    odh-model-controller: keda-reconciler
spec:
  secretTargetRef:
  - parameter: bearerToken
    name: inference-prometheus-auth
    key: token
  - parameter: ca
    name: inference-prometheus-auth
    key: ca.crt

bearerToken: The token that KEDA uses for authentication.

ca: The CA certificate that KEDA uses for TLS verification.

To use this resource, set authenticationRef.name to inference-prometheus-auth and authModes to bearer in each Prometheus trigger.

Resource lifecycle

Creation: The odh-model-controller creates the resources when an LLMInferenceService resource in the namespace declares at least one Prometheus trigger in spec.scaling.keda or spec.prefill.scaling.keda.

Sharing: The resources are created once for each namespace and are shared by all LLMInferenceService and InferenceService resources in the namespace that use Prometheus-based KEDA autoscaling. Each of these resources is added as an owner reference on the shared resources.

Updates: The odh-model-controller restores the expected configuration if the resources are modified. Changes to the triggers in the LLMInferenceService specification do not require changes to the authentication resources.

Deletion: When an LLMInferenceService resource is deleted or no longer declares a Prometheus trigger, the odh-model-controller removes its owner reference. The controller deletes the resources when no LLMInferenceService or InferenceService resource in the namespace requires them.

Custom authentication

The automatically created resources support queries through the Thanos Querier tenancy port (9092) in the workload namespace. For other requirements, create a separate TriggerAuthentication resource with a different name and reference it in the trigger. Examples include queries that span multiple namespaces through port 9091, or a Prometheus instance other than OpenShift monitoring. Do not modify the inference-prometheus-auth resources, because the odh-model-controller restores their expected configuration.

The following example grants cluster-wide query access through port 9091:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: custom-prometheus-reader
  namespace: llm-workloads
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: custom-prometheus-reader
  namespace: llm-workloads
  annotations:
    kubernetes.io/service-account.name: custom-prometheus-reader
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: llm-workloads-custom-prometheus-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-monitoring-view
subjects:
- kind: ServiceAccount
  name: custom-prometheus-reader
  namespace: llm-workloads
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: custom-prometheus-auth
  namespace: llm-workloads
spec:
  secretTargetRef:
  - parameter: bearerToken
    name: custom-prometheus-reader
    key: token
  - parameter: ca
    name: custom-prometheus-reader
    key: ca.crt

In the trigger, set serverAddress to https://thanos-querier.openshift-monitoring.svc.cluster.local:9091, remove the namespace field, and set authenticationRef.name to custom-prometheus-auth.

NOTE: If a namespace contains an LLMInferenceService resource with a Prometheus trigger, the odh-model-controller creates the inference-prometheus-auth resources in that namespace even if the trigger references a custom TriggerAuthentication resource.

Configuration

Configuring static allocation

Configure a fixed number of replicas for LLMInferenceService workloads. Static allocation does not create autoscaling resources.

Prerequisites

Static allocation does not require additional components.

Procedure

  1. If spec.scaling is set, remove it. For disaggregated deployments, also remove spec.prefill.scaling if it is set. A workload cannot set both replicas and scaling.

    When scaling is removed from a running service, the controller deletes the ScaledObject resource that it created, and KEDA deletes the associated HPA.

  2. Set spec.replicas:

    spec:
 replicas: 3

    For a standard deployment, this value is the number of model server replicas. For a disaggregated deployment, this value is the number of decode replicas. If replicas is not set, the workload runs one replica.

  3. For a disaggregated deployment, set the number of prefill replicas:

    spec:
 replicas: 5          // decode
 prefill:
   replicas: 3        // prefill

    where:

    spec.replicas: The fixed number of decode replicas.

    spec.prefill.replicas: The fixed number of prefill replicas.

    Each value is applied independently.

  4. Apply the configuration:

    $ oc apply -f llm-inference-service.yaml

Verification

  1. Verify the replica count of the model server workloads:

    $ oc get deployment -n llm-workloads granite-chat-kserve granite-chat-kserve-prefill

    Example output:

    NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
granite-chat-kserve           5/5     5            5           10m
granite-chat-kserve-prefill   3/3     3            3           10m

    For multi-node deployments, verify the LeaderWorkerSet resources instead by running oc get leaderworkerset -n llm-workloads.

  2. Verify that no autoscaling resources exist for the service:

    $ oc get scaledobject,hpa -n llm-workloads

    The output does not include the granite-chat-kserve-keda or keda-hpa-granite-chat-kserve-keda resources.

Changing capacity

To change capacity, update spec.replicas or spec.prefill.replicas and apply the configuration. Setting replicas to 0 stops all model server pods and releases their GPU resources. The workload remains at zero replicas until you increase the value.

Configuring KEDA autoscaling

Configure KEDA autoscaling for an LLMInferenceService workload so that the number of model server replicas follows the number of requests that vLLM is processing.

This procedure uses the vLLM metric vllm:num_requests_running, which reports the number of requests that each model server pod is processing. The workload scales between minReplicas and maxReplicas and does not scale to zero. To scale to zero during idle periods, see the scale-to-zero configuration section below.

Prerequisites

  • The Custom Metrics Autoscaler Operator is installed, and a KedaController resource exists.
  • Red Hat OpenShift AI is installed. No authentication setup is required for Prometheus triggers, because OpenShift AI creates the TriggerAuthentication resource automatically when you apply the autoscaling configuration.
  • The LLMInferenceService metrics are collected by OpenShift user workload monitoring.

About this task

Autoscaling is configured in the spec.scaling field of each workload:

  • spec.scaling.minReplicas (default 1, minimum 1) and spec.scaling.maxReplicas (required) define the replica range.
  • spec.scaling.keda contains the triggers and the KEDA settings: pollingInterval, cooldownPeriod, initialCooldownPeriod, idleReplicaCount, fallback, and advanced.

The KServe controller creates a ScaledObject resource named <name>-kserve-keda that targets the workload. The target is the Deployment <name>-kserve or, for multi-node deployments, the LeaderWorkerSet <name>-kserve-mn. KEDA then creates an HPA named keda-hpa-<name>-kserve-keda.

When the configuration includes a Prometheus trigger, OpenShift AI automatically creates a TriggerAuthentication resource named inference-prometheus-auth, together with its service account, secret, role, and role binding, in the namespace of the LLMInferenceService resource. Reference this resource in each Prometheus trigger as shown in the following procedure.

The spec.replicas and spec.scaling fields are mutually exclusive. Do not set spec.replicas on a workload that uses spec.scaling.

Replica calculation

  • The query field must return a single numeric value that represents the total across all pods. Use an aggregation such as sum(...), not avg(...). KEDA divides the query result by the current replica count before it compares the result with threshold. A query that uses avg(...) is therefore divided twice, which results in fewer replicas than required.
  • The threshold field specifies the target value per replica. With threshold: "10", a total of 35 running requests results in 4 replicas, within the range of minReplicas and maxReplicas.
  • When multiple triggers are configured, the HPA uses the highest replica count calculated across all triggers.

Procedure

  1. If spec.replicas is set on the workload, remove it.

  2. Define the replica range:

    spec:
 scaling:
   minReplicas: 1
   maxReplicas: 10

  3. Add a Prometheus trigger on the vLLM running requests metric:

    spec:
 scaling:
   keda:
     triggers:
     - type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: llm-workloads
         query: sum(vllm:num_requests_running{namespace="llm-workloads",model_name="<served-model-name>"})
         threshold: "10"
         authModes: bearer
       authenticationRef:
         name: inference-prometheus-auth

    serverAddress: The Thanos Querier tenancy port, which restricts queries to the namespace specified in the namespace field.

    namespace: The namespace of the LLMInferenceService resource.

    query: The total number of running requests for the model. Set model_name to the served model name.

    threshold: The target number of running requests per replica.

    authModes and authenticationRef: Configure KEDA to authenticate with the inference-prometheus-auth TriggerAuthentication resource. You do not create this resource. OpenShift AI creates it automatically when you apply this configuration.

  4. Optional: Configure the polling and cooldown settings:

    spec:
 scaling:
   keda:
     pollingInterval: 15
     initialCooldownPeriod: 600

    pollingInterval: The interval, in seconds, at which KEDA evaluates the triggers.

    initialCooldownPeriod: The period, in seconds, that KEDA waits after the ScaledObject resource is created before it evaluates triggers. Use this setting to allow time for the model to load.

  5. Optional: Configure fallback behavior for periods when Prometheus is unavailable:

    spec:
 scaling:
   keda:
     fallback:
       failureThreshold: 3
       replicas: 2
       behavior: currentReplicasIfHigher

    After 3 consecutive failed polling intervals, KEDA maintains 2 replicas or the current replica count, whichever is higher.

  6. Optional: Configure HPA stabilization windows:

    spec:
 scaling:
   keda:
     advanced:
       horizontalPodAutoscalerConfig:
         behavior:
           scaleUp:
             stabilizationWindowSeconds: 30
           scaleDown:
             stabilizationWindowSeconds: 300

    Do not set the horizontalPodAutoscalerConfig.name field. The controller manages the HPA name, and the API server rejects a configuration that sets this field.

  7. Apply the configuration:

    $ oc apply -f llm-inference-service.yaml

    The following example shows the complete configuration:

    apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
 name: granite-chat
 namespace: llm-workloads
spec:
 model:
   uri: hf://ibm-granite/granite-3.3-8b-instruct
   name: ibm-granite/granite-3.3-8b-instruct
 scaling:
   minReplicas: 1
   maxReplicas: 10
   keda:
     pollingInterval: 15
     initialCooldownPeriod: 600
     fallback:
       failureThreshold: 3
       replicas: 2
       behavior: currentReplicasIfHigher
     triggers:
     - type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: llm-workloads
         query: sum(vllm:num_requests_running{namespace="llm-workloads",model_name="ibm-granite/granite-3.3-8b-instruct"})
         threshold: "10"
         authModes: bearer
       authenticationRef:
         name: inference-prometheus-auth
     advanced:
       horizontalPodAutoscalerConfig:
         behavior:
           scaleUp:
             stabilizationWindowSeconds: 30
           scaleDown:
             stabilizationWindowSeconds: 300
 // router, template, and other fields omitted

Configuring autoscaling for a disaggregated prefill/decode deployment

In a disaggregated deployment, the decode workload and the prefill workload scale independently. Configure spec.scaling for the decode workload and spec.prefill.scaling for the prefill workload. The controller creates a separate ScaledObject resource for each workload:

Workload Configuration field ScaledObject Target
Decode spec.scaling <name>-kserve-keda Deployment <name>-kserve
Prefill spec.prefill.scaling <name>-kserve-prefill-keda Deployment <name>-kserve-prefill

Both workloads must use scaling.keda. A workload that uses scaling cannot also set replicas. You can configure autoscaling for one workload and static allocation for the other.

Each trigger must count only the requests on its own workload. Filter the vLLM metric by the pod label, which contains the pod name. Pod names have the format <deployment>-<hash>-<hash>: decode pods are named <name>-kserve-<hash>-<hash>, and prefill pods are named <name>-kserve-prefill-<hash>-<hash>:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: granite-chat
  namespace: llm-workloads
spec:
  // Decode workload
  scaling:
    minReplicas: 1
    maxReplicas: 8
    keda:
      pollingInterval: 15
      triggers:
      - type: prometheus
        metadata:
          serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
          namespace: llm-workloads
          query: sum(vllm:num_requests_running{namespace="llm-workloads",pod=~"granite-chat-kserve-[a-z0-9]+-[a-z0-9]+"})
          threshold: "10"
          authModes: bearer
        authenticationRef:
          name: inference-prometheus-auth
  // Prefill workload
  prefill:
    scaling:
      minReplicas: 1
      maxReplicas: 4
      keda:
        pollingInterval: 15
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: llm-workloads
            query: sum(vllm:num_requests_running{namespace="llm-workloads",pod=~"granite-chat-kserve-prefill-[a-z0-9]+-[a-z0-9]+"})
            threshold: "20"
            authModes: bearer
          authenticationRef:
            name: inference-prometheus-auth
  // model, router, template, and other fields omitted
  • The decode query matches pod names that contain exactly two segments after granite-chat-kserve-, which are the decode pods. Prometheus regular expressions match the full label value, so prefill pod names do not match.
  • The prefill query matches pod names that contain exactly two segments after granite-chat-kserve-prefill-, which are the prefill pods.
  • Set each threshold value according to the capacity of the corresponding replicas. Prefill requests are typically shorter than decode requests, so a prefill replica can usually process more concurrent requests.

For multi-node deployments, the workloads are LeaderWorkerSets and the pod names use a different format. Use pod=~"granite-chat-kserve-mn-[0-9]+(-[0-9]+)?" for decode and pod=~"granite-chat-kserve-mn-prefill-[0-9]+(-[0-9]+)?" for prefill.

Verification

  1. Verify that the ScaledObject resources exist:

    $ oc get scaledobject -n llm-workloads

    Example output for a disaggregated deployment:

    NAME                               SCALETARGETKIND      SCALETARGETNAME               MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED   TRIGGERS     AUTHENTICATIONS             AGE
granite-chat-kserve-keda           apps/v1.Deployment   granite-chat-kserve           1     8     True    True     False      False    prometheus   inference-prometheus-auth   2m
granite-chat-kserve-prefill-keda   apps/v1.Deployment   granite-chat-kserve-prefill   1     4     True    True     False      False    prometheus   inference-prometheus-auth   2m

  2. Verify that each ScaledObject resource is ready:

    $ oc get scaledobject granite-chat-kserve-keda -n llm-workloads \
   -o jsonpath='{.status.conditions[?(@.type=="Ready")]}'

  3. Verify that KEDA created the HPAs:

    $ oc get hpa -n llm-workloads

    Example output:

    NAME                                        REFERENCE                                TARGETS      MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-granite-chat-kserve-keda           Deployment/granite-chat-kserve           6/10 (avg)   1         8         2          2m
keda-hpa-granite-chat-kserve-prefill-keda   Deployment/granite-chat-kserve-prefill   4/20 (avg)   1         4         1          2m

  4. Verify that the TriggerAuthentication resource was created:

    $ oc get triggerauthentication inference-prometheus-auth -n llm-workloads

  5. Verify the scaling conditions on the LLMInferenceService resource:

    $ oc get llminferenceservice granite-chat -n llm-workloads -o jsonpath='{.status.conditions}'

Configuring scale-to-zero and scale-from-zero with flow control

Configure an LLMInferenceService workload to scale to zero replicas during idle periods and to scale up from zero when requests arrive. Scale-to-zero releases GPU resources while the service is not in use.

Prerequisites

  • KEDA autoscaling is available, as described in the KEDA autoscaling configuration section above.
  • Flow control is enabled on the scheduler (Endpoint Picker). For more information, see the managing mixed workloads with priority queuing documentation.

About scale-to-zero and scale-from-zero

Scale-to-zero

To enable scale-to-zero, set spec.scaling.keda.idleReplicaCount to 0. The value of idleReplicaCount must be less than minReplicas, and minReplicas must be set. The minReplicas field does not accept 0.

When the trigger reports a value that does not exceed its activationThreshold for the duration of cooldownPeriod, KEDA scales the workload to zero replicas.

Scale-from-zero

At zero replicas, no model server pods are running, so metrics that model server pods emit are not available. A trigger on those metrics cannot initiate a scale-up from zero.

The Endpoint Picker (EPP) continues to run when the model server is at zero replicas. With flow control enabled, the EPP holds incoming requests in a queue and reports the queue size in the llm_d_epp_flow_control_queue_size metric. When a request is queued, the metric value exceeds activationThreshold. KEDA scales the workload directly from zero to minReplicas, and the EPP dispatches the queued requests when the model server is ready.

The EPP metric is filtered by inference_pool="<name>-inference-pool", which is the name of the InferencePool that the controller creates. If you reference an existing InferencePool through spec.router.scheduler.pool.ref, use the name of that InferencePool instead.

Combining the queue trigger with a running requests trigger

The queue metric counts only requests that are waiting for capacity. When the available replicas can process all incoming requests, the queue size is 0, even while requests are being processed. As a result, a configuration that uses only the queue trigger has the following characteristics:

  • Under steady load, the HPA reduces the replica count toward minReplicas.
  • KEDA can scale the workload to zero after cooldownPeriod while long-running requests are still being processed, unless cooldownPeriod exceeds the longest expected request duration.

To address these characteristics, you can add a second Prometheus trigger on the vLLM running requests metric (vllm:num_requests_running), as described in the KEDA autoscaling configuration section above, to the same triggers list. When multiple triggers are configured, KEDA applies the following rules:

  • The workload is active while at least one trigger is active.
  • The HPA uses the highest replica count calculated across all triggers.

With both triggers configured, the queue trigger initiates the scale-up from zero, because it is the only metric that is available at zero replicas. While the workload is active, the running requests trigger keeps the workload active during request processing and determines capacity under steady load. KEDA scales the workload to zero only after both metrics have reported 0 for the duration of cooldownPeriod.

Procedure

  1. Set the replica range and enable scale-to-zero:

    spec:
 scaling:
   minReplicas: 1
   maxReplicas: 10
   keda:
     idleReplicaCount: 0

  2. Configure the polling and cooldown settings:

    spec:
 scaling:
   keda:
     pollingInterval: 5
     cooldownPeriod: 300
     initialCooldownPeriod: 600

    pollingInterval: A short interval reduces the time that KEDA requires to detect a queued request while the workload is at zero replicas.

    cooldownPeriod: The period during which the queue must remain empty before KEDA scales the workload to zero. Set this value higher than the longest expected request duration. For LLM workloads, use a value between 300 and 600 seconds.

    initialCooldownPeriod: Set this value to at least the model load time.

  3. Add the EPP flow control queue trigger:

    spec:
 scaling:
   keda:
     triggers:
     - type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: llm-workloads
         query: sum(llm_d_epp_flow_control_queue_size{inference_pool="granite-chat-inference-pool"})
         threshold: "1"
         activationThreshold: "0"
         authModes: bearer
       authenticationRef:
         name: inference-prometheus-auth

    threshold: "1": Targets approximately 1 queued request per replica. A queue size of 6 results in 6 replicas, within the range of minReplicas and maxReplicas.

    activationThreshold: "0": Scales the workload up from zero when the queue size is greater than 0.

    OpenShift AI automatically creates the inference-prometheus-auth TriggerAuthentication resource when you apply this configuration.

    NOTE: The queue size is 0 whenever the available replicas can process all incoming requests, including while requests are being processed. Set cooldownPeriod higher than the longest expected request duration, so that the workload does not scale to zero while requests are still being processed. Alternatively, add a running requests trigger, as described in the "Combining the queue trigger with a running requests trigger" section above.

  4. Optional: Configure fallback behavior:

    spec:
 scaling:
   keda:
     fallback:
       failureThreshold: 3
       replicas: 2
       behavior: currentReplicasIfHigher

    If Prometheus is unavailable for 3 consecutive polling intervals, KEDA maintains 2 replicas or the current replica count, whichever is higher. This prevents the workload from remaining at zero replicas during a metrics outage.

  5. Apply the configuration:

    $ oc apply -f llm-inference-service.yaml

    The following example shows the complete configuration:

    apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
 name: granite-chat
 namespace: llm-workloads
spec:
 model:
   uri: hf://ibm-granite/granite-3.3-8b-instruct
   name: ibm-granite/granite-3.3-8b-instruct
 scaling:
   minReplicas: 1
   maxReplicas: 10
   keda:
     idleReplicaCount: 0
     pollingInterval: 5
     cooldownPeriod: 300
     initialCooldownPeriod: 600
     fallback:
       failureThreshold: 3
       replicas: 2
       behavior: currentReplicasIfHigher
     triggers:
     - type: prometheus
       metadata:
         serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
         namespace: llm-workloads
         query: sum(llm_d_epp_flow_control_queue_size{inference_pool="granite-chat-inference-pool"})
         threshold: "1"
         activationThreshold: "0"
         authModes: bearer
       authenticationRef:
         name: inference-prometheus-auth
 // router, template, and other fields omitted

    This configuration results in the following behavior:

  • At zero replicas: When a request is queued, KEDA scales the workload to minReplicas (1).
  • While active: The HPA targets approximately 1 queued request per replica, up to 10 replicas.
  • Scale-to-zero: After the queue has been empty for 300 seconds, KEDA scales the workload to zero.

Configuring scale-to-zero for a disaggregated prefill/decode deployment

In a disaggregated deployment, configure scale-to-zero on both the decode workload (spec.scaling) and the prefill workload (spec.prefill.scaling). Each workload has its own ScaledObject resource.

The EPP queue metric is reported for the InferencePool as a whole, so both workloads use the same queue trigger. When a request is queued, both workloads scale up from zero. Use a different threshold value for each workload to reflect the capacity of its replicas:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: granite-chat
  namespace: llm-workloads
spec:
  // Decode workload
  scaling:
    minReplicas: 1
    maxReplicas: 8
    keda:
      idleReplicaCount: 0
      pollingInterval: 5
      cooldownPeriod: 300
      initialCooldownPeriod: 600
      triggers:
      - type: prometheus
        metadata:
          serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
          namespace: llm-workloads
          query: sum(llm_d_epp_flow_control_queue_size{inference_pool="granite-chat-inference-pool"})
          threshold: "1"
          activationThreshold: "0"
          authModes: bearer
        authenticationRef:
          name: inference-prometheus-auth
  // Prefill workload
  prefill:
    scaling:
      minReplicas: 1
      maxReplicas: 4
      keda:
        idleReplicaCount: 0
        pollingInterval: 5
        cooldownPeriod: 300
        initialCooldownPeriod: 600
        triggers:
        - type: prometheus
          metadata:
            serverAddress: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092
            namespace: llm-workloads
            query: sum(llm_d_epp_flow_control_queue_size{inference_pool="granite-chat-inference-pool"})
            threshold: "2"
            activationThreshold: "0"
            authModes: bearer
          authenticationRef:
            name: inference-prometheus-auth
  // model, router, template, and other fields omitted

This configuration results in the following behavior:

  • At zero replicas: When a request is queued at the EPP, both workloads scale up from zero to their minReplicas value.
  • While active: The decode workload targets approximately 1 queued request per replica, up to 8 replicas. The prefill workload targets approximately 2 queued requests per replica, up to 4 replicas.
  • Scale-to-zero: After the queue has been empty for 300 seconds, both workloads scale to zero.

Use the same cooldownPeriod value for both workloads, so that both workloads scale to zero and from zero together.

If you combine the queue trigger with a running requests trigger in a disaggregated deployment, filter the running requests metric for each workload by pod name, as described in the KEDA autoscaling configuration section above, so that each workload remains active only while its own pods are processing requests.

Verification

  1. While the service is idle, verify that the workload scales to zero after the cooldown period:

    $ oc get deployment granite-chat-kserve -n llm-workloads

    Example output:

    NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
granite-chat-kserve   0/0     0            0           30m

  2. Verify that the ScaledObject resource is inactive:

    $ oc get scaledobject granite-chat-kserve-keda -n llm-workloads

    The ACTIVE column displays False. The MIN column displays the value of minReplicas, not 0.

  3. Send a request to the service, and verify that the workload scales up from zero:

    $ oc get deployment granite-chat-kserve -n llm-workloads -w

    The READY column changes from 0/0 to 1/1 when the model server is ready. The request is processed after the model has loaded.

Additional considerations

Cold-start latency: The first request after a scale-up from zero waits for pod scheduling, image pull, and model download and loading. This can take several minutes for large models. For services that cannot tolerate this delay, do not set idleReplicaCount. See the KEDA autoscaling configuration section above.

Request timeouts: Configure client and gateway timeouts to exceed the expected cold-start time, so that queued requests are not rejected before the model server is ready.

Reference

KEDA autoscaling parameter reference

This reference lists KEDA autoscaling configuration parameters with their defaults, constraints, and LLM-specific recommendations.

Parameters

Parameter Type Default Constraint Recommendation for LLM workloads
spec.scaling.minReplicas integer 1 ≥ 1; ≤ maxReplicas; required when idleReplicaCount is set 1. Increase the value to maintain additional standby capacity.
spec.scaling.maxReplicas integer None (required) ≥ 1; ≥ minReplicas The number of replicas that the available GPU capacity can schedule
spec.scaling.keda.idleReplicaCount integer Not set (scale-to-zero disabled) ≥ 0; < minReplicas 0 to release GPU resources during idle periods
spec.scaling.keda.pollingInterval integer (seconds) 30 (KEDA default) ≥ 1 15–30
spec.scaling.keda.cooldownPeriod integer (seconds) 300 (KEDA default) ≥ 0 300–600
spec.scaling.keda.initialCooldownPeriod integer (seconds) 0 (KEDA default) ≥ 0 Equal to or greater than the model load time
spec.scaling.keda.fallback.failureThreshold integer Not set (fallback disabled) > 0 3
spec.scaling.keda.fallback.replicas integer Not set ≥ 0 1 or greater
spec.scaling.keda.fallback.behavior string static static, currentReplicas, currentReplicasIfHigher, currentReplicasIfLower currentReplicasIfHigher
spec.scaling.keda.triggers[] list None (required) At least one trigger See the activationThreshold section below
spec.scaling.keda.advanced object Not set horizontalPodAutoscalerConfig.name must not be set Stabilization windows and scaling policies

Parameter details

minReplicas

The minimum number of replicas for HPA scaling while the workload is active. The minimum accepted value is 1. To scale below one replica, use idleReplicaCount.

When idleReplicaCount is set and a trigger becomes active, KEDA scales the workload from idleReplicaCount directly to minReplicas in a single step. For example, with idleReplicaCount: 0 and minReplicas: 4, activity causes the workload to scale from 0 to 4 replicas. The HPA then scales the workload between 4 replicas and maxReplicas.

To avoid cold-start latency, do not set idleReplicaCount. The workload then maintains at least minReplicas replicas. The cold-start time for an LLM includes pod scheduling, image pull, and model download and loading, and can take several minutes for large models.

maxReplicas

The maximum number of replicas for HPA scaling. This parameter is required. Set the value to the number of replicas that the available GPU capacity can schedule. Replicas that exceed the available capacity remain in the Pending state.

idleReplicaCount

The number of replicas while no trigger is active. When this parameter is not set, the workload does not scale below minReplicas.

The value of idleReplicaCount must be less than minReplicas. Set the value to 0. Because of the way that KEDA coordinates with the HPA, 0 is the only value that KEDA supports reliably.

cooldownPeriod

The period that KEDA waits after the last active trigger before it scales the workload to idleReplicaCount. This parameter applies only to the transition to the idle replica count. Scale-down between maxReplicas and minReplicas is controlled by the HPA scaleDown behavior. A value of 0 causes an immediate scale-down.

For LLM workloads, set a value between 300 and 600 seconds to prevent repeated scaling to and from zero between bursts of traffic. Batch workloads with intervals between batches benefit from values at the upper end of this range.

pollingInterval

The interval, in seconds, at which KEDA evaluates triggers. This value determines how quickly KEDA detects activity while the workload is at zero replicas. Between minReplicas and maxReplicas, the HPA also reads metrics from the metrics API server at its own synchronization interval, which is 15 seconds by default.

A value between 15 and 30 seconds balances activation latency against the query load on Prometheus.

initialCooldownPeriod

The period after the ScaledObject resource is created during which KEDA does not evaluate triggers. Set the value to at least the model load time, so that KEDA does not make scaling decisions based on metrics that are unavailable or inaccurate before the model server is ready.

fallback

The fallback.replicas parameter specifies the number of replicas that KEDA maintains after fallback.failureThreshold consecutive failed trigger evaluations, for example, when Prometheus is unavailable. With pollingInterval: 30 and failureThreshold: 3, fallback takes effect after approximately 90 seconds of failures.

Fallback applies only to triggers with the AverageValue metric type, which is the default for Prometheus triggers.

The fallback.behavior parameter determines the replica count that KEDA maintains during fallback:

  • static (default): The value of fallback.replicas.
  • currentReplicas: The replica count at the time that the failures began.
  • currentReplicasIfHigher: The current replica count or fallback.replicas, whichever is higher. This value is recommended for LLM workloads, because it prevents a metrics outage from reducing the capacity of an active service.
  • currentReplicasIfLower: The current replica count or fallback.replicas, whichever is lower.

The fallback.behavior parameter requires a Custom Metrics Autoscaler Operator version that is based on a KEDA release that supports this field.

Set fallback.replicas to a value that can serve the expected traffic. A value of 1 keeps the service available but might not provide sufficient capacity. If fallback occurs frequently, investigate the availability of the metric source.

Trigger metadata: activationThreshold

The activationThreshold parameter is part of the metadata field of a trigger, not a field of spec.scaling.keda. It controls activation, which is scaling from idleReplicaCount to minReplicas. The threshold parameter controls scaling between minReplicas and maxReplicas.

  • The default value is 0.
  • KEDA activates the workload when the metric value is greater than activationThreshold. The comparison is strict: with a value of "0", any metric value greater than zero activates the workload.
  • The activation threshold takes precedence over the scaling threshold. If the metric value does not exceed activationThreshold, the workload remains at the idle replica count, even if the threshold value alone would result in additional replicas.

Cron triggers do not use activationThreshold. A cron trigger is active for the duration of its schedule window.

Related settings

Stabilization windows: Configured in spec.scaling.keda.advanced.horizontalPodAutoscalerConfig.behavior.{scaleUp,scaleDown}.stabilizationWindowSeconds. Stabilization windows affect scaling between minReplicas and maxReplicas. They do not affect scaling up from zero or the transition to the idle replica count.

Scaling policies: Configured in ...behavior.scaleUp.policies and ...behavior.scaleDown.policies. Scaling policies limit the rate of change, as a percentage of replicas or a number of pods per period. They do not affect activation.

Trigger-specific metadata: Each trigger type defines its own metadata fields. For example, Prometheus triggers use serverAddress, query, threshold, activationThreshold, namespace, and authModes.

Comments