Chapter 5. Model serving on OpenShift Data Science

Model serving is currently available as a Field Trial feature. For more information on the scope of Field Trial status, see OpenShift Data Science Service Definition.

As a data scientist, you can deploy your trained machine-learning models to serve intelligent applications in production. After you have deployed your model, applications can send requests to the model using its deployed API endpoint.

5.1. Configuring model servers

5.1.1. Configuring a model server for your data science project

Before you can successfully deploy a data science model on OpenShift Data Science, you must configure a model server. This includes configuring the number of replicas being deployed, the server size, the token authorization, and how the project is accessed.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have created a data science project that you can add a model server to.

Procedure

  1. From the OpenShift Data Science dashboard, click Data Science Projects.

    The Data science projects page opens.

  2. Click the name of the project that you want to configure a model server for.

    The Details page for the project opens.

  3. Click Configure server in the Models and model servers section.

    The Configure model server dialog appears.

  4. Configure your project’s model server in the Compute resources per replica section.

    1. Select the number of model server replicas to deploy.
    2. Select one of the following server sizes for your model:

      • Small
      • Medium
      • Large
      • Custom
    3. Optional: If you selected Custom, configure the following settings in the Model server size section to customize your model server:

      1. Enter the number of CPUs to use with your model in the CPUs requested field.
      2. Enter the maximum number of CPUs required for use with your model in the CPU limit field.
      3. Enter the requested memory for the model server in gigabytes (GB) in the Memory requested field.
      4. Enter the maximum memory limit for the model server in gigabytes (GB) in the Memory limit field.
    4. Optional: Select the Make deployed available via an external route check box in the Model route section to make your deployed model available externally.
    5. Optional: Select the Require token authorization check box in the Token Authorization section to apply token authentication to your model server.

      1. Edit the service account name for which the token will be generated in the Token secret field. The generated token is created and displayed when the model server is configured.
      2. To add an additional service account, click Add a service account and enter the relevant information in the Token secret field.

Verification

  • The model server that you configured is displayed in the Models and model servers section on the Details page for the project.

5.1.2. Updating a model server

You can update your data science project’s model server by changing details, such as the number of deployed replicas, the server size, the token authorization, and how the project is accessed.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have created a data science project that has a model server assigned.

Procedure

  1. From the OpenShift Data Science dashboard, click Data Science Projects.

    The Data science projects page opens.

  2. Click the name of the project whose model server details you want to update.

    The Details page for the project opens.

  3. In the Models and model servers section, locate the model server you want to update, and click the action menu () → Edit model server.

    The Configure model server dialog opens.

  4. Update the model server’s properties.

    1. Select the number of model server replicas to deploy.
    2. Select one of the following server sizes for your model:

      • Small
      • Medium
      • Large
      • Custom
    3. Optional: If you selected Custom, configure the following settings in the Model server size section to customize your model server:

      1. Enter the number of CPUs to use with your model in the CPUs requested field.
      2. Enter the maximum number of CPUs required for use with your model in the CPU limit field.
      3. Enter the requested memory for the model server in gigabytes (GB) in the Memory requested field.
      4. Enter the maximum memory limit for the model server in gigabytes (GB) in the Memory limit field.
    4. Optional: Select the Make deployed available via an external route check box in the Model route section to make your deployed model available externally.
    5. Optional: Select the Require token authorization check box in the Token Authorization section to apply token authentication to your model server.

      1. Edit the service account name for which the token will be generated in the Token secret field. The generated token is created and displayed when the model server is configured.
      2. To add an additional service account, click Add a service account and enter the relevant information in the Token secret field.
  5. Click Configure.

Verification

  • The model server that you updated is displayed in the Models and model servers section on the Details page for the project.

5.1.3. Deleting a model server

You can delete model servers that you have assigned to host your data science projects. This enables you to remove model servers that you no longer require. If you delete a project’s model server, models hosted by the server are then unavailable for applications to use. Red Hat recommends that you advise the users of these models before deleting the model server.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have created a data science project and an associated model server.

Procedure

  1. From the OpenShift Data Science dashboard, click Data Science Projects.

    The Data science projects page opens.

  2. Click the name of the project that you want to delete the model server from.

    The Details page for the project opens.

  3. Click the action menu () beside the project whose model server you want to delete in the Models and model servers section and click Delete model server.

    The Delete model server dialog opens.

  4. Enter the name of the model server in the text field to confirm that you intend to delete it.
  5. Click Delete model server.

Verification

  • The model server that you deleted is no longer displayed in the Models and model servers section on the project Details page.

5.2. Working with deployed models

5.2.1. Deploying a model in OpenShift Data Science

You can deploy trained models on OpenShift Data Science to enable you to test and implement them into intelligent applications. Deploying a model makes it available as a service that you can access using an API. This enables you to return predictions based on data inputs.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have created a data science project that contains an associated model server.

Procedure

  1. From the OpenShift Data Science dashboard, click Data Science Projects.

    The Data science projects page opens.

  2. Click the name of the project containing the model that you want to deploy.

    The Details page for the project opens.

  3. Click Deploy model in the Models and model servers section.

    The Deploy model dialog appears.

  4. Configure the properties for deploying your model.

    1. Enter a name for the model that you are deploying.
    2. Select a model framework for the model that you are deploying.
    3. Specify the location of your model.

      1. Select Existing data connection to use a data connection that you previously defined.

        1. Select the name of the data connection from the list.
        2. Enter the folder path containing the model.
      2. Select New data connection to add a new data connection that your model can access.

        1. Enter a name for the data connection.
        2. Enter your access key ID for Amazon Web Services in the AWS_ACCESS_KEY_ID field.
        3. Enter your secret access key for the account you specified in the AWS_ACCESS_KEY_ID field.
        4. Enter the endpoint of your AWS S3 storage in the AWS_S3_ENDPOINT field.
        5. Enter the default region of your AWS account in the AWS_DEFAULT_REGION field.
        6. Enter the name of the AWS S3 bucket in the AWS_S3_BUCKET field.
        7. Enter the folder path of the file containing the data.
    4. Click Configure.

Verification

  • The model you deployed is displayed in the Deployed models table on the Model Serving page, accessed from the dashboard.

5.2.2. Viewing a deployed model

To analyse the results of your work, you can view a list of deployed models on Red Hat OpenShift Data Science. You can also view the current statuses of deployed models and their endpoints.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • There are active and deployed data science models available on the Model Serving page.

Procedure

  1. From the OpenShift Data Science dashboard, click Model Serving.

    The Model Serving page opens.

  2. Review the list of deployed models.

    Inference endpoints are displayed in the Inference endpoint column in the Deployed models table.

  3. Optional: Click the Copy button ( osd copy ) on the relevant row to copy the model’s inference endpoint to the clipboard.

Verification

  • A list of previously deployed data science models is displayed on the Model Serving page.

5.2.3. Updating the deployment properties of a deployed model

You can update the deployment properties of a model that has been deployed previously. This allows you to change the model’s data connection and name.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have deployed a model on OpenShift Data Science.

Procedure

  1. From the OpenShift Data Science dashboard, click Model serving.

    The Model Serving page opens.

  2. Click the action menu () beside the model whose deployment properties you want to update and click Edit.

    The Deploy model dialog opens.

  3. Update the deployment properties of the model.

    1. Update the name of the model.
    2. Select an option to specify the location of your model.

      1. If you selected Existing data connection, update the folder path of your existing data connection.
      2. If you selected New data connection, update the relevant fields to add a new data connection.

        1. Update the data connection name.
        2. Update your access key ID for Amazon Web Services in the AWS_ACCESS_KEY_ID field.
        3. Update your secret access key for the account you specified in the AWS_ACCESS_KEY_ID field.
        4. Update the endpoint of your AWS S3 storage in the AWS_S3_ENDPOINT field.
        5. Update the default region of your AWS account in the AWS_DEFAULT_REGION field.
        6. Update the name of the AWS S3 bucket in the AWS_S3_BUCKET field.
        7. Update the folder path of the file containing the data.
    3. Click Configure.

Verification

  • The model whose deployment properties you updated is displayed on the Model Serving page.

5.2.4. Deleting a deployed model

You can delete models you have previously deployed. This enables you to remove deployed models that are no longer required.

Prerequisites

  • You have logged in to Red Hat OpenShift Data Science.
  • If you are using specialized OpenShift Data Science groups, you are part of the user group or admin group (for example, rhods-users) in OpenShift.
  • You have have deployed a model.

Procedure

  1. From the OpenShift Data Science dashboard, click Model serving.

    The Model Serving page opens.

  2. Click the action menu () beside the deployed model that you want to delete and click Delete.

    The Delete deployed model dialog opens.

  3. Enter the name of the deployed model in the text field to confirm that you intend to delete it.
  4. Click Delete deployed model.

Verification

  • The model that you deleted is no longer displayed on the Model Serving page.