Built-in observability dashboards for llm-d deployments

Red Hat OpenShift AI 3.5 includes built-in observability dashboards for Distributed Inference with llm-d deployments. The dashboards appear in the OpenShift AI Observe & Monitor section and offer pre-built views for monitoring traffic health, latency, and resource utilization without deploying a separate Grafana instance.

The kserve-module Operator deploys the dashboards automatically as PersesDashboard custom resources when you install the Cluster Observability Operator (COO) on your OpenShift cluster. COO manages the Red Hat build of Perses server instance that renders the dashboards.

Note: You must install COO manually from OperatorHub. OpenShift AI does not install COO automatically.

Default dashboards

LLM Traffic: Shows cluster-level request health indicators such as request rate, error rate, and token consumption. Use this dashboard to answer "Is my inference service healthy and handling traffic correctly?"

LLM Performance: Shows latency and cache diagnostics, including time to first token (TTFT) percentiles, inter-token latency (ITL), end-to-end latency, and KV cache hit rate. Use this dashboard to answer "Are my users experiencing acceptable latency?" and "Are my models meeting performance expectations?"

LLM Utilization: Shows per-replica resource consumption such as GPU utilization, CPU utilization, memory utilization, KV cache utilization, and request queue depth. Use this dashboard to answer "Are my replicas adequately resourced?"

Each of the three dashboards also includes panels that break down metrics by individual pod (Per-pod detail in Performance, Pod & pool detail in Utilization) and by model (Per-model detail in Traffic and Performance). The Phase latency panel provides model-specific prefill and decode timing, filtered by the model selector. The EPP scheduling panel reports Endpoint Picker (EPP) scheduling duration aggregated across models and pods within the selected namespace. These built-in panels replace the manual PromQL drill-down queries that were previously required for the same investigations.

Drill-down investigation workflow

The three dashboards support a structured investigation workflow for diagnosing issues with your Distributed Inference with llm-d deployments:

1. Start with LLM Traffic to detect error spikes, throughput drops, or abnormal token patterns that indicate a problem.
2. Move to LLM Performance to investigate whether the deployment meets latency targets and whether KV cache hit rates explain any latency regressions.
3. Drill into LLM Utilization to diagnose per-replica GPU, CPU, memory, and KV cache saturation as root causes.
4. Use the pod-level panels to determine whether an issue affects all pods evenly or is isolated to one pod.
5. Use the model-level panels to determine whether an issue is isolated to a specific model when your namespace serves more than one LLMInferenceService.
6. Use the per-phase and scheduling latency panels to distinguish prompt processing (prefill) time, token generation (decode) time, and endpoint selection time.

Common diagnostic patterns

The following patterns help you interpret metric combinations across dashboards.

Latency issue patterns

• High TTFT with normal inter-token latency indicates a prefill bottleneck from prompt processing or KV cache misses.
• High inter-token latency with normal TTFT indicates a decode bottleneck from token generation contention.
• Normal TTFT and inter-token latency with high end-to-end latency indicates scheduling delays. Check request queue depth in LLM Utilization.
• High prefill time in the Prefill / decode latency panel with normal decode time narrows a prefill bottleneck to prompt processing rather than token generation, and vice versa.
• A high EPP scheduling latency value indicates that time spent selecting an endpoint, rather than model execution, is contributing to end-to-end latency. This metric is aggregate across all models and pods, confirmed from its merged query definition (sum by (le) (rate(llm_d_epp_scheduler_e2e_duration_seconds_bucket{...})), no model_name or pod grouping), so it cannot be attributed to a specific model or pod.

Queue saturation

In LLM Utilization, compare running requests and waiting requests per pod. A steady running count with waiting requests near zero is healthy. Growing waiting requests indicate the pod cannot keep up. If queue depth is high across all pods, scale up. If queue depth is high on only one pod, investigate whether that node has a hardware issue.

KV cache pressure

A sudden drop in KV cache hit rate in LLM Performance often correlates with a TTFT regression. If KV cache usage in LLM Performance is at 100%, cache evictions have started. If preemption rates from the PromQL drill-down queries are high alongside low cache hit rates, memory pressure is the root cause.

Pod outlier detection

If aggregate metrics in LLM Traffic look healthy but users report intermittent slowness, check per-pod metrics in LLM Utilization. A single pod with higher latency, lower cache hit rate, or higher queue depth than its peers indicates unbalanced routing or a hardware issue on that node.

Built-in dashboards compared to community Grafana dashboards

OpenShift AI provides two approaches for monitoring Distributed Inference with llm-d deployments with dashboards. Both approaches source from the same underlying Prometheus metrics.

Built-in Perses dashboards: Deployed automatically on OpenShift when COO is installed. Rendered in the Observe & Monitor section. Require no separate infrastructure.

Community Grafana dashboards: Require a separately deployed Grafana instance and manual import of dashboard JSON files. Available on any Kubernetes platform. Offer a broader panel set including diagnostic drill-down and operational views.

On OpenShift with COO installed, use the built-in dashboards as your primary monitoring path. Community Grafana dashboards remain available for non-OpenShift platforms, environments where Grafana is already deployed, or when you need supplementary diagnostic views beyond the built-in panels.

Additional resources

• Dashboard panels and metrics for llm-d observability

Dashboard panels and metrics for llm-d observability

The three built-in observability dashboards for Distributed Inference with llm-d deployments contain panels that visualize Prometheus metrics from vLLM engines and the Endpoint Picker (EPP), also known as the inference scheduler. The following tables describe what each panel measures and how to interpret its data.

You can filter by namespace (exported_namespace label) and model name (model_name label). Selecting a filter limits all panels to the selected namespace or model.

The dashboards also include panels that break down traffic, performance, and utilization metrics by individual pod and model, and separate panels for per-phase and scheduling latency. Panel names, descriptions, and queries in this reference are taken from the merged dashboard definitions in the opendatahub-io/kserve repository (config/monitoring/llmisvc/dashboards/).

LLM Traffic dashboard

The LLM Traffic dashboard provides cluster-level request health indicators for your llm-d deployments.

LLM Traffic panels

Panel	Metrics	What it answers	Interpretation guidance
Request Rate	kserve_vllm:request_success_total	How many requests per second is my deployment handling?	A stable or growing rate matching expected load is healthy. A sudden drop might indicate upstream issues or routing failures.
Error Rate	llm_d_epp_request_error_total	Are requests failing?	An error rate near zero is healthy. Spikes in error rate require immediate investigation. Correlate with latency metrics in the LLM Performance dashboard.
Token Throughput	kserve_vllm:prompt_tokens_total, kserve_vllm:generation_tokens_total	How much work is the system doing?	Token volume consistent with workload patterns is healthy. An unchanged token count despite active clients might indicate that the system rejects requests before they reach the model.

Under Per-model detail, the Traffic dashboard also includes the following panels, confirmed on the merged dashboard definition:

Per-model detail panels (LLM Traffic dashboard)

Panel	Metrics	What it answers	Interpretation guidance
Error breakdown by code	sum by (error_code) (rate(llm_d_epp_request_error_total{...}[$__rate_interval]))	Which error codes are occurring, and how often?	Helps distinguish transient failures from systematic issues. Correlate a spike in a specific error code with recent configuration or deployment changes.
Per-model token consumption	kserve_vllm:prompt_tokens_total, kserve_vllm:generation_tokens_total grouped by model_name (input, output, and total series; rate(...), tokens/s)	What is the input, output, and total token throughput for each model?	Useful for monitoring token throughput and capacity planning across models, but the tokens-per-second values do not represent total token consumption over a billing period and are not sufficient for chargeback calculations. This panel does not calculate cost, identify a tenant, or provide billing controls.
Per-model traffic distribution	kserve_vllm:request_success_total grouped by model_name	How is request volume distributed across models?	Traffic proportions matching expected usage patterns are healthy. An unexpected shift might indicate a client misconfiguration or a change in consumer behavior.

LLM Performance dashboard

The LLM Performance dashboard provides latency percentiles and cache diagnostics for your llm-d deployments.

LLM Performance panels

Panel	Metrics	What it answers	Interpretation guidance
Time to First Token (TTFT)	kserve_vllm:time_to_first_token_seconds_bucket (P50, P95, P99)	How long do users wait before the model starts responding?	TTFT within your service-level objective (SLO) threshold is healthy. A spiking P99 might indicate prefill queue saturation or KV cache misses. Correlate with KV Cache Hit Rate: a drop in cache hits often precedes a TTFT regression.
End-to-End Latency	kserve_vllm:e2e_request_latency_seconds_bucket (P50, P95, P99)	How long does the full request lifecycle take?	Stable latency matching expected generation lengths is healthy. Increasing latency without increased output length suggests a bottleneck. If TTFT and inter-token latency are both normal, check request queue depth in the LLM Utilization dashboard for scheduling delays.
Inter-Token Latency (ITL)	kserve_vllm:time_per_output_token_seconds_bucket (P50, P95, P99)	Is token generation smooth for streaming responses?	Consistent ITL indicates smooth streaming. High variance causes choppy user experience and might indicate decode-phase contention.
KV Cache Hit Rate	kserve_vllm:prefix_cache_hits_total, kserve_vllm:prefix_cache_queries_total	Is prefix caching working effectively?	The dashboard calculates hit rate as the rate of cache hits divided by the rate of cache queries. A hit rate above 60-80% is healthy for workloads with repeated prefixes. A sudden drop in hit rate correlates with TTFT regressions.
KV Cache Usage	kserve_vllm:kv_cache_usage_perc	How full is the KV cache?	Usage below 90% is healthy. Usage at 100% indicates evictions have started, which degrades hit rate and increases TTFT. Correlate with the KV Cache Hit Rate panel: usage at 100% causes evictions that degrade hit rate.

The Performance dashboard also includes the following panels, confirmed on the merged dashboard definition:

Per-pod detail panels (LLM Performance dashboard)

Panel	Metrics	What it answers	Interpretation guidance
Per-pod cache hit rate	kserve_vllm:prefix_cache_hits_total, kserve_vllm:prefix_cache_queries_total grouped by k8s_pod_name	Which pods are benefiting from prefix caching?	A hit rate consistent with other pods is healthy. A pod with a much lower hit rate than its peers indicates that prefix-affinity routing to that pod is not working as expected.
Per-pod KV cache usage	kserve_vllm:kv_cache_usage_perc grouped by k8s_pod_name	Is the KV cache full on specific pods?	Pods near 100% usage are at risk of preemptions and increased TTFT.

Phase latency panel (LLM Performance dashboard)

Panel	Metrics	What it answers	Interpretation guidance
Prefill / decode latency	kserve_vllm:request_prefill_time_seconds_bucket, kserve_vllm:request_decode_time_seconds_bucket grouped by (le, model_name) (P50, P95, P99 for each phase)	Is a latency issue compute-bound (prefill) or memory-bound (decode)?	The model selector filters this panel, which provides per-model series. Prefill measures the wall-clock interval from request scheduling to first token generation; decode measures the interval from first token to last token. Both are wall-clock phase durations and can include preemption delays, so they are not pure execution time. TTFT includes the initial request queueing delay. The ITL panel queries time-per-output-token (TPOT), calculated by dividing decode duration by subsequent output tokens, so it excludes that initial delay.

EPP scheduling panel (LLM Performance dashboard)

Panel	Metrics	What it answers	Interpretation guidance
EPP scheduling latency	llm_d_epp_scheduler_e2e_duration_seconds_bucket (P50, P95, P99; aggregate across models and pods within the selected namespace)	Is endpoint selection time, rather than model execution, contributing to end-to-end latency?	Time spent in the EPP scheduler (filter/score/pick plugins) per request isolates endpoint selection time from model execution time. The panel aggregates this latency across models and pods within the selected namespace, so it cannot attribute scheduling delay to an individual model or pod.

Per-model detail panel (LLM Performance dashboard)

Panel	Metrics	What it answers	Interpretation guidance
Per-model TTFT	kserve_vllm:time_to_first_token_seconds_bucket grouped by model_name (P50, P95, P99)	Which model is slow?	TTFT consistent with each model's expected profile is healthy. A model with TTFT well outside its baseline, while the fleet-wide TTFT panel looks healthy, indicates a model-specific SLA breach masked by the aggregate view. To compare TTFT across models, select all models in the model filter; selecting a specific model limits the panel to that model.

Important: The model_name label reflects the base model that a serving pod was started with. It does not identify individual LoRA adapters served by that pod. If your deployment serves multiple LoRA adapters from a single base model, these panels report combined metrics for all adapters under that base model's name.

LLM Utilization dashboard

The LLM Utilization dashboard provides per-replica resource consumption metrics for your llm-d deployments.

LLM Utilization panels

Panel	Metrics	What it answers	Interpretation guidance
GPU Utilization	accelerator_gpu_utilization	How much GPU capacity does each replica consume?	Utilization between 40-80% is healthy. Sustained utilization above 95% indicates saturation. If one replica is significantly higher than others, investigate routing balance in the inference scheduler.
Request Queue Depth	kserve_vllm:num_requests_running, kserve_vllm:num_requests_waiting	Are replicas keeping up with incoming requests?	Running requests at a steady level with waiting requests near zero is healthy. Growing waiting requests indicate the replica cannot process requests fast enough.
Per-pod KV Cache Utilization	kserve_vllm:kv_cache_usage_perc	Is the KV cache full on specific replicas?	Per-pod cache usage below 90% is healthy. If specific pods show full cache while others do not, investigate routing and prefix affinity configuration.

Note: GPU utilization metrics require the NVIDIA Data Center GPU Manager (DCGM) exporter. If you have not deployed DCGM, the GPU Utilization panel does not display data.

The Utilization dashboard also includes the following panels, confirmed on the merged dashboard definition:

Pod & pool detail panels (LLM Utilization dashboard)

Panel	Metrics	What it answers	Interpretation guidance
Per-pod queue size (scheduler view)	llm_d_epp_per_endpoint_queue_size	Which pods have requests waiting for processing, as seen by the scheduler?	Compare with the vLLM-reported Request Queue Depth panel above to detect scraping lag or routing asymmetry. This metric is pool-wide and is not filtered by the model selector.
Preemption rate	kserve_vllm:num_preemptions_total grouped by k8s_pod_name	Which pods are evicting running requests due to memory pressure?	Preemptions near zero are healthy. A pod with a rising preemption count is experiencing KV cache pressure; correlate with that pod's KV cache usage.
EPP ready endpoints	llm_d_epp_ready_endpoints	Are all endpoints in the inference pool ready and healthy?	A ready endpoint count matching the expected pod count is healthy. A drop indicates pods failing health checks.
EPP average queue size	llm_d_epp_average_queue_size	Is the inference pool saturated across all endpoints?	A rising average queue size indicates pool-wide saturation rather than a single-pod issue.

Interpreting missing panel data

Most panels display No data rather than a zero value when no traffic has reached the filtered namespace, model, or pod during the selected time range. If a panel shows No data but you expect active traffic, check the following:

• The dashboard time range. The default range might not include the traffic you expect; extend it to include a longer or earlier window.
• The namespace and model filters. Confirm they match your deployment.
• Whether the underlying counter has ever recorded a value.

Note: The following five panels use an or vector(0) fallback in their merged dashboard queries: Error Rate (LLM Traffic dashboard), TTFT (LLM Performance dashboard), End-to-End Latency (LLM Performance dashboard), Inter-Token Latency (ITL) (LLM Performance dashboard), and EPP scheduling latency (LLM Performance dashboard). They can display zero instead of No data when the underlying metric is missing. A zero does not necessarily indicate healthy operation.

Metric prefixes in dashboard panels

llm-d metrics are available in two locations in the OpenShift UI, each using different metric naming:

OpenShift AI dashboard (Observe & Monitor)

The Perses dashboards query metrics from multiple sources:
- kserve_vllm:: vLLM engine metrics: TTFT, latency, cache, queue depth. These are recording rules that map to underlying vllm: metrics with normalized labels: exported_namespace instead of namespace, and k8s_pod_name instead of pod.
- llm_d_epp_: EPP/scheduler metrics for scheduling latency, error rate, and pool health. Queried directly, no recording rules.
- accelerator_gpu_utilization: GPU compute utilization. Queried directly.

OpenShift Console (Observe > Dashboards)

The Performance dashboards query raw Prometheus metric names such as vllm_gpu_cache_usage_perc and vllm_num_requests_waiting directly, without recording rules.

PromQL queries for llm-d monitoring

You can use the following PromQL queries to monitor your Distributed Inference with llm-d deployments.
The queries are organized into two groups: immediate failure and saturation indicators that drive operational alerts, and diagnostic drill-down queries that support deeper investigation.

Failure and saturation indicators

Use the following queries for dashboards and alerts that track service health and capacity limits.

Failure and saturation PromQL queries

Operational concern	PromQL query
Overall error rate	sum(rate(llm_d_epp_request_error_total[5m])) / sum(rate(llm_d_epp_request_total[5m]))
Per-model error rate	sum by(model_name) (rate(llm_d_epp_request_error_total[5m])) / sum by(model_name) (rate(llm_d_epp_request_total[5m]))
Error rate by error code	sum by(error_code) (rate(llm_d_epp_request_error_total[5m]))
Request rate by model	sum by(model_name, target_model_name) (rate(llm_d_epp_request_total[5m]))
Overall latency P99	histogram_quantile(0.99, sum by(le) (rate(llm_d_epp_request_duration_seconds_bucket[5m])))
Overall latency P90	histogram_quantile(0.90, sum by(le) (rate(llm_d_epp_request_duration_seconds_bucket[5m])))
Model-specific TTFT P99	histogram_quantile(0.99, sum by(le, model_name) (rate(vllm:time_to_first_token_seconds_bucket[5m])))
Model-specific time per output token (TPOT) P99	histogram_quantile(0.99, sum by(le, model_name) (rate(vllm:inter_token_latency_seconds_bucket[5m])))
Streaming ITL P99	histogram_quantile(0.99, sum by(le) (rate(llm_d_epp_request_streaming_itl_seconds_bucket[5m])))
Streaming ITL P90	histogram_quantile(0.90, sum by(le) (rate(llm_d_epp_request_streaming_itl_seconds_bucket[5m])))
EPP end-to-end latency P99	histogram_quantile(0.99, sum by(le) (rate(llm_d_epp_scheduler_e2e_duration_seconds_bucket[5m])))
Plugin processing latency P99	histogram_quantile(0.99, sum by(le, plugin_type) (rate(llm_d_epp_plugin_duration_seconds_bucket[5m])))
Scheduler health	avg_over_time(up{job=~".*epp.*"}[5m])
Request preemptions per vLLM instance	sum by(pod, instance) (rate(vllm:num_preemptions_total[5m]))

Diagnostic drill-down queries

Use the following queries for deeper investigation when failure and saturation indicators signal issues.

Model serving and scaling queries

Operational concern	PromQL query
KV cache utilization by pod	avg by(pod, model_name) (vllm:kv_cache_usage_perc)
Request queue length by pod	sum by(pod, model_name) (vllm:num_requests_waiting)
Model throughput in tokens per second	sum by(model_name, pod) (rate(vllm:prompt_tokens_total[5m]) + rate(vllm:generation_tokens_total[5m]))
Generation token rate by pod	sum by(model_name, pod) (rate(vllm:generation_tokens_total[5m]))
Active requests per pod	avg by(pod) (vllm:num_requests_running)

Routing and load balancing queries

Operational concern	PromQL query
Request distribution per instance	sum by(pod) (rate(llm_d_epp_request_total{target_model_name!=""}[5m]))
Token distribution across pods	sum by(pod) (rate(vllm:prompt_tokens_total[5m]) + rate(vllm:generation_tokens_total[5m]))
Routing decision latency P99	histogram_quantile(0.99, sum by(le) (rate(llm_d_epp_plugin_duration_seconds_bucket[5m])))

Prefix caching queries

Operational concern	PromQL query
Prefix cache hit rate	sum(rate(vllm:prefix_cache_hits_total[5m])) / sum(rate(vllm:prefix_cache_queries_total[5m]))
Per-instance prefix cache hit rate	sum by(pod) (rate(vllm:prefix_cache_hits_total[5m])) / sum by(pod) (rate(vllm:prefix_cache_queries_total[5m]))
KV cache utilization percentage	avg by(pod, model_name) (vllm:kv_cache_usage_perc * 100)
EPP prefix indexer size	llm_d_epp_prefix_indexer_size
EPP prefix indexer hit ratio P90	histogram_quantile(0.90, sum by(le) (rate(llm_d_epp_prefix_indexer_hit_ratio_bucket[5m])))

Prefill/decode disaggregation queries

Operational concern	PromQL query
Prefill worker utilization	avg by(pod) (vllm:num_requests_running{pod=~".*prefill.*"})
Decode worker KV cache utilization	avg by(pod) (vllm:kv_cache_usage_perc{pod=~".*kserve.*",pod!~".*prefill.*"})
Prefill queue length	sum by(pod) (vllm:num_requests_waiting{pod=~".*prefill.*"})
Disaggregation decision rate by type	sum by(decision_type) (rate(llm_d_inference_scheduler_disagg_decision_total[5m]))
Disaggregation decision ratio	sum(rate(llm_d_inference_scheduler_disagg_decision_total{decision_type="prefill-decode"}[5m])) / sum(rate(llm_d_inference_scheduler_disagg_decision_total[5m]))

NIXL KV cache transfer queries

Operational concern	PromQL query
NIXL transfer rate	sum(rate(vllm:nixl_xfer_time_seconds_count[5m]))
NIXL average transfer duration	rate(vllm:nixl_xfer_time_seconds_sum[5m]) / rate(vllm:nixl_xfer_time_seconds_count[5m])
NIXL bytes transferred per second	sum(rate(vllm:nixl_bytes_transferred_sum[5m]))
NIXL failed transfer rate	sum(rate(vllm:nixl_num_failed_transfers_total[5m]))
NIXL failed notification rate	sum(rate(vllm:nixl_num_failed_notifications_total[5m]))
NIXL failed transfer ratio	sum(rate(vllm:nixl_num_failed_transfers_total[5m])) / sum(rate(vllm:nixl_xfer_time_seconds_count[5m]))
Prompt token source distribution	sum by(source) (rate(vllm:prompt_tokens_by_source_total[5m]))
Decode queue depth	sum by(pod) (vllm:num_requests_waiting{pod=~".*kserve.*",pod!~".*prefill.*"})
Prefill latency P99	histogram_quantile(0.99, sum by(le) (rate(vllm:time_to_first_token_seconds_bucket{pod=~".*prefill.*"}[5m])))
Decode inter-token latency P99	histogram_quantile(0.99, sum by(le) (rate(vllm:inter_token_latency_seconds_bucket{pod=~".*kserve.*",pod!~".*prefill.*"}[5m])))

EPP and inference scheduler metrics for llm-d

The Endpoint Picker (EPP), or the inference scheduler, in Distributed Inference with llm-d deployments expose Prometheus metrics at the /metrics endpoint on the metrics service port.
You can use these metrics to monitor request routing decisions, scheduling latency, inference performance, and prefix cache indexing performance.

Note: Metrics in Distributed Inference with llm-d use the llm_d_epp_ prefix.
The inference_objective_, inference_extension_, and inference_pool_ metric prefixes are deprecated. These prefixes remain available for backward compatibility.

Inference objective metrics

The service-level request metrics track end-to-end inference performance at the Endpoint Picker (EPP).
These metrics measure user-perceived latency, which includes scheduler queue wait time and network latency between the scheduler and the model server.
These metrics use the llm_d_epp_request_ prefix.

Service-level request metrics

Metric name	Type	Labels	Description
llm_d_epp_request_total	counter	model_name, target_model_name, fairness_id, priority	The total number of inference requests processed by the scheduler.
llm_d_epp_request_error_total	counter	model_name, target_model_name, fairness_id, priority, error_code	The total number of inference request errors, broken down by error code.
llm_d_epp_request_duration_seconds	histogram	model_name, target_model_name, fairness_id, priority	The end-to-end inference request duration distribution in seconds.
llm_d_epp_request_ttft_seconds	histogram	model_name, target_model_name, fairness_id, priority	The time to first token (TTFT) distribution in seconds, measured from request receipt to first response byte. For non-streaming requests, this equals the total request duration.
llm_d_epp_request_streaming_tpot_seconds	histogram	model_name, target_model_name, fairness_id, priority	The average time per output token (TPOT) distribution in seconds for streaming requests, computed as (end-to-end latency - TTFT) / (output tokens - 1).
llm_d_epp_request_streaming_itl_seconds	histogram	model_name, target_model_name, fairness_id, priority	The inter-token latency (ITL) distribution in seconds for streaming requests, measured as the time between consecutive response body chunks. Use ITL to detect decode-phase lag spikes that TPOT averaging can mask. The EPP emits this metric only for streaming requests.
llm_d_epp_request_input_tokens	histogram	model_name, target_model_name, fairness_id, priority	The input token count distribution per request.
llm_d_epp_request_output_tokens	histogram	model_name, target_model_name, fairness_id, priority	The output token count distribution per request.
llm_d_epp_request_running	gauge	model_name, target_model_name, fairness_id, priority	The current number of active inference requests.

Scheduling and routing metrics

The scheduling and routing metrics track the internal performance of the Endpoint Picker.
These metrics use the llm_d_epp_ prefix.

Scheduling and routing metrics

Metric name	Type	Labels	Description
llm_d_epp_scheduler_e2e_duration_seconds	histogram	None	The end-to-end scheduling duration in seconds, measured from request receipt to endpoint selection. This metric backs the EPP scheduling latency panel in the LLM Performance dashboard; see Dashboard panels and metrics for llm-d observability. The merged panel query (llm_d_epp_scheduler_e2e_duration_seconds_bucket{exported_namespace=~"$namespace"}) confirms the panel uses this unprefixed metric name, not a kserve_-prefixed variant, and remains unlabeled by model_name or pod.
llm_d_epp_scheduler_attempts_total	counter	status, target_model_name, endpoint_name, namespace, port	The total number of scheduling attempts, broken down by status: success or failure.
llm_d_epp_plugin_duration_seconds	histogram	extension_point, plugin_type, plugin_name	The plugin processing duration in seconds, broken down by extension point, plugin type, and plugin name. Use this metric to identify slow plugins that impact scheduling latency.
llm_d_epp_model_rewrite_decisions_total	counter	model_rewrite_name, model_name, target_model	The total number of model rewrite decisions made by the scheduler.

Prefix cache indexer metrics

The prefix cache indexer metrics track KV cache prefix matching performance in the scheduler.
These metrics use the llm_d_epp_ prefix.

Prefix cache indexer metrics

Metric name	Type	Labels	Description
llm_d_epp_prefix_indexer_size	gauge	None	The current size of the prefix cache index maintained by the scheduler.
llm_d_epp_prefix_indexer_hit_ratio	histogram	None	The prefix cache hit ratio distribution. A higher hit ratio indicates that requests frequently match cached prefixes, reducing redundant computation.
llm_d_epp_prefix_indexer_hit_bytes	histogram	None	The prefix cache hit size distribution in bytes. Larger hit sizes indicate more effective prefix reuse.

Inference pool metrics

The inference pool metrics track aggregate statistics across all endpoints in a pool.
These metrics use the llm_d_epp_ prefix.

Inference pool metrics

Metric name	Type	Labels	Description
llm_d_epp_average_kv_cache_utilization	gauge	name	The average KV cache utilization across all endpoints in the inference pool.
llm_d_epp_average_queue_size	gauge	name	The average request queue size across all endpoints in the inference pool.
llm_d_epp_average_running_requests	gauge	name	The average number of running requests across all endpoints in the inference pool.
llm_d_epp_ready_endpoints	gauge	name	The number of ready endpoints in the inference pool.
llm_d_epp_per_endpoint_queue_size	gauge	name, model_server_endpoint	The request queue size for a specific endpoint in the inference pool.

Flow control metrics

The flow control metrics track priority-based queuing, request dispatching, and pool saturation when flow control is enabled.
These metrics use the llm_d_epp_flow_control_ prefix.

Flow control metrics

Metric name	Type	Labels	Description
llm_d_epp_flow_control_queue_size	gauge	fairness_id, priority, inference_pool, model_name, target_model_name	The current number of requests actively held in the flow control queue, broken down by fairness group and priority band.
llm_d_epp_flow_control_queue_bytes	gauge	fairness_id, priority, inference_pool, model_name, target_model_name	The current total size in bytes of requests actively held in the flow control queue, broken down by fairness group and priority band.
llm_d_epp_flow_control_pool_saturation	gauge	inference_pool	The current saturation level of the inference pool, ranging from 0.0 (empty) to 1.0 (fully saturated). When saturation reaches the configured threshold, dispatch halts for all priority bands.
llm_d_epp_flow_control_requests_total	counter	outcome, priority, inference_pool	The total number of requests processed by the flow control layer, broken down by outcome: Dispatched, RejectedCapacity, RejectedNoEndpoints, RejectedOther, EvictedTTL, EvictedContextCancelled, or EvictedOther.
llm_d_epp_flow_control_request_queue_duration_seconds	histogram	fairness_id, priority, outcome, inference_pool, model_name, target_model_name	The distribution of total time that requests wait in the flow control layer, measured from enqueue to final outcome.
llm_d_epp_flow_control_dispatch_cycle_duration_seconds	histogram	None	The distribution of time taken for each internal dispatch cycle in the flow control layer. Use this metric to identify dispatch processing bottlenecks.
llm_d_epp_flow_control_request_enqueue_duration_seconds	histogram	fairness_id, priority, outcome	The distribution of time taken to enqueue requests into the flow control layer.