Starting Jupyter Notebook Server at 96%, Invalid Image Name

Solution Unverified - Updated -

Environment

  • OpenShift Cluster Platform 4.x
  • Self Managed Red Hat OpenShift AI
    • 1.2x

Issue

The user is not able to finalize the spawning of a Jupyter Notebook, the Server is stuck at 96% and the notebook pod returns a Warning: failed to apply default image tag ":py3.8-v1": couldn't parse image reference ":py3.8-v1": invalid reference format

Resolution

Red Hat OpenShift Data Science(RHODS) use pre-built notebooks and they will be pulled to the internal registry.
For this reason, you must make sure that the internal registry is running in the cluster before you install RHODS.

Verify if the cluster has an image registry pod

  oc get pods -n openshift-image-registry
  NAME                                               READY   STATUS      RESTARTS   AGE
  cluster-image-registry-operator-759dbc577c-28qjt   1/1     Running     0          1d 
  image-registry-584cc85cf4-28nw6                    1/1     Running     0          1d    (<=== This is the image-registry)
  node-ca-q4f98                                      1/1     Running     0          1d

If you don't see any image registry pod is not running, you must to deploy it. In order to deploy an image registry pod, please refer to the Openshift Container Platform Documentation.

For a test cluster, you can use emptyDir which is NOT recommended for a production cluster.

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
config.imageregistry.operator.openshift.io/cluster patched
oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'
config.imageregistry.operator.openshift.io/cluster patched

Root Cause

At least, one image registry pod must run in an openshift container platform cluster. Without this, the cluster does not mean that it is ready to provide services properly.

Diagnostic Steps

  1. Install RHOAI self managed
  2. Log in as RHOAI user
  3. Spawn a notebook from the RHOAI Dashboard
  4. Check the notebook pod
$ oc get pods -n rhods-notebooks

Example output

NAME                 READY   STATUS             RESTARTS   AGE
jupyter-nb-admin-0   1/2     InvalidImageName   0          4m48s

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments