Chapter 4. Troubleshooting common problems in JupyterHub

If you are seeing errors in Red Hat OpenShift Data Science related to JupyterHub, your notebooks, or your notebook server, read this section to understand what could be causing the problem.

If you cannot see your problem here or in the release notes, contact Red Hat Support.

4.1. I see a 403: Forbidden error when I log in to JupyterHub

Problem

Your user name might not be added to the default user group or the default administrator group for OpenShift Data Science. Contact your administrator so that they can add you to the correct group/s.

Diagnosis

Check whether the user is part of either the default user group or the default administrator group.

  1. Find the names of groups allowed access to JupyterHub.

    1. Log in to OpenShift Dedicated web console.
    2. Click WorkloadsConfigMaps and click on the rhods-groups-config ConfigMap to open it.
    3. Click on the YAML tab and check the values for admin_groups and allowed_groups. These are the names of groups that have access to JupyterHub.

        data:
          admin_groups: rhods-admins
          allowed_groups: rhods-users
  2. Click User managementGroups and click on the name of each group to see its members.

Resolution

  • If the user is not added to any of the groups allowed access to JupyterHub, follow Adding users for OpenShift Data Science to add them.
  • If the user is already added to a group that is allowed to access JupyterHub, contact Red Hat Support.

4.2. My notebook server does not start

Problem

The OpenShift Dedicated cluster that hosts your notebook server might not have access to enough resources, or the JupyterHub pod may have failed. Contact your administrator so that they can perform further checks.

Diagnosis

  1. Log in to OpenShift Dedicated web console.
  2. Delete and restart the notebook server pod for this user.

    1. Click WorkloadsPods and set the Project to rhods-notebooks.
    2. Search for the notebook server pod that belongs to this user exists, for example, jupyterhub-nb-username-*.

      If the notebook server pod exists, an intermittent failure may have occurred in the notebook server pod.

      If the notebook server pod for the user does not exist, continue with diagnosis.

  3. Check the resources currently available in the OpenShift Dedicated cluster against the resources required by the selected notebook server image.

    If worker nodes with sufficient CPU and RAM are available for scheduling in the cluster, continue with diagnosis.

  4. Check the state of the JupyterHub pod.

Resolution

  • If there was an intermittent failure of the notebook server pod:

    1. Delete the notebook server pod that belongs to the user.
    2. Ask the user to start their notebook server again.
  • If the notebook server does not have sufficient resources to run the selected notebook server image, either add more resources to the OpenShift Dedicated cluster, or choose a smaller image size.
  • If the JupyterHub pod is in a FAILED state:

    1. Retrieve the logs for the jupyterhub-* pod and send them to Red Hat Support for further evaluation.
    2. Delete the jupyterhub-* pod.

      Warning

      Ensure that you delete the correct pod. Do not delete the jupyterhub-db-* pod by mistake.

  • If none of the previous resolutions apply, contact Red Hat Support.

4.3. I see a database or disk is full error or a no space left on device error when I run my notebook cells

Problem

You might have run out of storage space on your notebook server. Contact your administrator so that they can perform further checks.

Diagnosis

  1. Log in to JupyterHub and start the notebook server that belongs to the user having problems. If the notebook server does not start,
  2. Check whether the user has run out of storage space.

    1. Log in to OpenShift Dedicated web console.
    2. Click WorkloadsPods and set the Project to rhods-notebooks.
    3. Click the notebook server pod that belongs to this user, for example, jupyterhub-nb-username-*.
    4. Click Logs. The user has exceeded their available capacity if you see lines similar to the following:

      Unexpected error while saving file: XXXX database or disk is full

Resolution

  • Increase the user’s available storage by expanding their persistent volume: Expanding persistent volumes
  • Work with the user to identify files that can be deleted from the /opt/app-root/src directory to free up their existing storage space.