Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode

1. Overview

Important:
This migration must be performed on RHOAI 2.25 if you are upgrading to RHOAI 3.3.

In RHOAI 3.x, ModelMesh and Serverless Deployment Mode in KServe are being deprecated.
Users must migrate existing InferenceServices to RawDeployment mode (shown as Standard in the UI).

This document provides:

Automated migration using helper scripts
Optional manual migration steps
Validation guidance
Cleanup procedures for legacy InferenceServices

After migration, users must:

Validate the new RawDeployment InferenceServices manually.
Confirm endpoints function correctly.
Remove legacy Serverless or ModelMesh InferenceServices.

The RHOAI team provides these instructions as a best-effort migration aid.

1.1 Purpose of This Migration

Because of the deprecation in RHOAI 3.x:

ModelMesh → RawDeployment migration is required
Serverless → RawDeployment migration is required

This guide covers:

Automated migration scripts
Optional manual conversion workflows
Validation procedures
Post-migration cleanup

1.2 Serverless → Raw Migration Helper Overview

This helper script:

Detects InferenceServices using Serverless mode
Prompts which models to migrate
For each selected service:
- Exports original InferenceService and ServingRuntime
- Generates new -raw resources in RawDeployment mode
- Handles authentication resources:
  - ServiceAccount
  - Roles
  - RoleBindings
  - Secrets
- Applies resources automatically (unless --dry-run)
Optionally preserves generated files for review

Tip:
If your namespace contains custom storage or permissions, apply those manually in the new namespace before migration.

Create the odh-cli pod that contains the scripts by running the following commands

These scripts will be run from within the container

oc run odh-cli --restart=Never --image=quay.io/rhoai/odh-cli-rhel9:rhoai-3.3 --overrides='{"spec":{"containers":[{"name":"odh-cli","image":"quay.io/rhoai/odh-cli-rhel9:rhoai-3.3","command":["sleep","infinity"]}]}}'


# Wait for the pod to be ready
oc wait --for=condition=Ready pod/odh-cli --timeout=120s


# Log in to the cluster from inside the pod, replace token and server with yours
oc exec -it odh-cli -- oc login --token=sha256~xxxx --server=https://api.my-cluster.p3.openshiftapps.com:6443

# Exec into the pod
oc exec -it odh-cli -- /bin/bash

1.3 ModelMesh → Raw Script Overview

This script automates migration from ModelMesh to KServe RawDeployment, including:

Environment validation and RBAC checks
Target namespace creation
ServingRuntime creation
Resource transformation:
- InferenceServices
- Storage configuration
- Authentication resources
Route exposure migration
Storage credential handling
Token regeneration when authentication is enabled
Optional manual review mode

User-Prompted Steps

Selecting models (single, multiple, or all)
Choosing storage behavior
Confirming ServingRuntime availability

2. Prerequisites

Scripts verify most requirements automatically, but ensure you have:

odh-cli pod is running and you are exec'd into the pod as we will be running all of the following commands inside of it
Required RBAC permissions in the target namespace
(not required for --dry-run)

3. Running the Scripts

See the Troubleshooting section at the bottom if issues occur.

3.1 Serverless → Raw Script

List namespaces containing Serverless InferenceServices:

rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*" --isvc-deployment-mode serverless

Dry Run

You can test out the output first by running the dry run mode.

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/serverless-to-raw.sh --dry-run -n <namespace>

Standard Execution

This is the recommended way of executing the script.

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/serverless-to-raw.sh -n <namespace>

Namespace defaults to the active OpenShift project.

Help

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/serverless-to-raw.sh -h

3.2 ModelMesh → Raw Script

List InferenceService names and namespaces with ModelMesh InferenceServices:

rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*" --isvc-deployment-mode modelmesh

Show help:

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/modelmesh-to-raw.sh --help

Example migration:

Note: the from-ns is the namespace where your modelmesh isvcs exist and the target-ns will be created automatically by just running the script

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/modelmesh-to-raw.sh \
  --from-ns public-models \
  --target-ns public-kserve

Key Parameters

--from-ns <source>
--target-ns <target>
--preserve-namespace
--ignore-existing-ns
--debug
--dry-run
--odh
--page-size <number>
-h | --help

4. Manual Migration

See Troubleshooting if problems occur.

4.1 Serverless Manual Migration

List InferenceService names and namespaces:

rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*" --isvc-deployment-mode serverless

Option A — Dry Run With Generated Files

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/serverless-to-raw.sh --dry-run -n <namespace>
oc apply -f <isvc_name>/raw/

Option B — Replace Existing Resources

oc delete route <isvc_name>-<namespace> -n istio-system
oc apply -f <isvc_name>/raw-original-names/

Generated directory structure:

<inference-service>/
├─ original/
├─ raw/
└─ raw-original-names/

4.2 ModelMesh Manual Migration

List InferenceService names and namespaces:

rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*" --isvc-deployment-mode modelmesh

Example dry-run output:

Run the dry-run command to create files:

/opt/rhai-upgrade-helpers/model-serving/before-upgrade/modelmesh-to-raw.sh \
  --from-ns public-models \
  --target-ns public-kserve --dry-run

The folder structure will be as follows:

migration-dry-run-<timestamp>/
├─ new-resources/
└─ original-resources/

Apply manually:

find migration-dry-run-<timestamp>/new-resources \
-name '*.yaml' -exec oc apply -f {} \;

Repeat per model as needed.

5. InferenceService Validation

Confirm

Ready=True status
Endpoint accessibility

Switch project:

oc project <project_name>

Runtime Endpoint Examples

Runtime	Endpoint Path
vLLM (CUDA/ROCM/S390x/PPC64le/Gaudi)	`v1/completions`
Caikit-TGis, OpenVINO	`v2/models`

Example Requests for Caikit-TGis or OpenVINO

NOTE: If you selected to have -raw appended to your InferenceService when running the serverless-to-raw script, then please make sure to use the new model name here which will be <old_model_name>-raw.

With Bearer Token

export RUNTIME_PATH=v2/models
export MODEL_NAME=<model_name>
export TOKEN=$(oc whoami -t)
export NAMESPACE=$(oc project -q)

URL=$(oc get isvc -n $NAMESPACE $MODEL_NAME \
  -o jsonpath='{.status.url}')

curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  $URL/$RUNTIME_PATH/$MODEL_NAME

Without Bearer Token

export RUNTIME_PATH=v2/models
export MODEL_NAME=<model_name>
export NAMESPACE=$(oc project -q)

URL=$(oc get isvc -n $NAMESPACE $MODEL_NAME \
  -o jsonpath='{.status.url}')

curl -sk \
  -H "Content-Type: application/json" \
  $URL/$RUNTIME_PATH/$MODEL_NAME

6. Clean Up Legacy InferenceServices

Critical Step:
Only delete old services after RawDeployment validation.

Identify Old Deployment Modes

Note: This only prints Serverless and ModelMesh InferenceServices for your convenience

rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*"

Delete Old InferenceServices

Option A — Manual

oc delete isvc <name> -n <namespace>

Repeat until none remain.

Option B — Bulk Delete

Preview first:

oc get isvc -A -o json | jq -r '.items[] | 
  select(
    .status.deploymentMode == "Serverless" or 
    .status.deploymentMode == "ModelMesh" or 
    .metadata.annotations["serving.kserve.io/deploymentMode"] == "Serverless" or 
    .metadata.annotations["serving.kserve.io/deploymentMode"] == "ModelMesh"
  ) | 
  "\(.metadata.namespace)/\(.metadata.name) - \(.status.deploymentMode // .metadata.annotations["serving.kserve.io/deploymentMode"] // "Unknown")"'

Then delete:

oc get isvc -A -o json | jq -r '.items[] | 
  select(
    .status.deploymentMode == "Serverless" or 
    .status.deploymentMode == "ModelMesh" or 
    .metadata.annotations["serving.kserve.io/deploymentMode"] == "Serverless" or 
    .metadata.annotations["serving.kserve.io/deploymentMode"] == "ModelMesh"
  ) | 
  "\(.metadata.namespace) \(.metadata.name)"' | 
while read -r ns name; do
  echo "Deleting InferenceService: $name in namespace: $ns"
  oc delete isvc "$name" -n "$ns"
done

Cleanup ModelMesh ServingRuntimes

List runtimes:

oc get servingruntimes.serving.kserve.io -A -o json |
jq -r '
["NAMESPACE","NAME","MULTI_MODEL","CREATED"],
(.items[] | select(.spec.multiModel==true) |
[.metadata.namespace,
.metadata.name,
(.spec.multiModel|tostring),
.metadata.creationTimestamp]) | @tsv' |
column -t

Delete:

oc delete servingruntime <name> -n <namespace>

Repeat until none remain in the output of the cli tool.

# One final check to make sure none remain.
rhai-cli lint --target-version 3.3.0 --verbose --checks "*kserve*"

7. Next Steps

At this stage:

Only RawDeployment (Standard) InferenceServices should exist.
Return to the KServe section of the RHOAI 3.3 upgrade guide.

Troubleshooting

Insufficient Resources

Serverless Migration

Pause old InferenceServices to free resources
Or delete legacy services before migration

ModelMesh Migration

ModelMesh cannot pause ISVCs
Namespace cleanup may be required

1. Overview

1.1 Purpose of This Migration

1.2 Serverless → Raw Migration Helper Overview

Create the odh-cli pod that contains the scripts by running the following commands

1.3 ModelMesh → Raw Script Overview

User-Prompted Steps

2. Prerequisites

3. Running the Scripts

3.1 Serverless → Raw Script

Dry Run

Standard Execution

Help

3.2 ModelMesh → Raw Script

Key Parameters

4. Manual Migration

4.1 Serverless Manual Migration

Option A — Dry Run With Generated Files

Option B — Replace Existing Resources

4.2 ModelMesh Manual Migration

5. InferenceService Validation

Confirm

Runtime Endpoint Examples

Example Requests for Caikit-TGis or OpenVINO

With Bearer Token

Without Bearer Token

6. Clean Up Legacy InferenceServices

Identify Old Deployment Modes

Delete Old InferenceServices

Option A — Manual

Option B — Bulk Delete

Cleanup ModelMesh ServingRuntimes

7. Next Steps

Troubleshooting

Insufficient Resources

Serverless Migration

ModelMesh Migration

Comments

Quick Links

Help

Site Info

Related Sites

Systems Status

About

Red Hat legal and privacy links

Red Hat legal and privacy links