Data Science Pipelines workaround for how to view pipeline run pod logs in the Red Hat OpenShift AI dashboard

Solution In Progress - Updated -

Environment

Red Hat OpenShift AI 2.9

Issue

When using the log viewer in the OpenShift AI dashboard to view pipeline run pod logs, a pods not found error message might be shown. As a result, the user is unable to retrieve the pod logs.

Resolution

As a prerequisite to performing the steps in this workaround, ensure that you have the following:

  • You have administrator credentials.
  • A previously executed pipeline run is available.
  • You have a record of the error message that appeared when you could not view the pod logs.
  1. Inspect the error message that appeared when you could not view the pod logs and note the workflow name and the node display name. For example, if the pod logs for the node mynode could not be retrieved, an error message similar to the following appears:
pods "tutorial-data-passing-2-w7pns-2899757045" not found 

Disregard the final suffix. In this example, the workflow name is "tutorial-data-passing-2-w7pns" and the node display name is "mynode". Ensure that you retain an easily accessible record of this information.

  1. In the OpenShift web console, change into the Administrator perspective.
  2. In the left navigation pane, click Home > API Explorer.
  3. In the Filter by kind field, type Workflow.
  4. Select the Workflow that is in theargoproj.io group.
  5. Click the Instances tab.
  6. From the Project list, select your data science project.
  7. Click the workflow name that you identified from the error message.
  8. Click the YAML tab.
  9. Locate the .status.nodes field, and then identify the node that matches your node display name.
  10. Inspect the children field. The pods associated with this node are listed here.
  11. Note down the name of the pod that you cannot see logs for.
  12. In the left navigation pane, click Workloads > Pods.
  13. From the Project list, select your data science project.
  14. Search for a pod that you cannot see logs for, and then click the pod name.
  15. Click the Logs tab.
  16. From the Container list, select the main container.
  17. Inspect the logs for the main container. From here, you can view pipeline task node log outputs for the node.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments