How to use different image with toolbox and oc debug node in OCP 4

Solution Unverified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Red Hat Enterprise Linux CoreOS (RHCOS)

Issue

  • How to install/use troubleshooting packages in OCP 4 nodes.
  • The default container image for toolbox and oc debug node/<node> is rhel9/support-tools or rhel8/support-tools. This images is based off of the Red Hat Universal Base Image. Due to the limited packages in the RHEL UBI repos it is not possible to install various tools to diagnose and troubleshoot issues.

Resolution

To overcome the package limitation of RHEL UBI you can override the default settings for the toolbox or the oc debug node/<node> command to pull a custom image. That custom image can either be:

  • An image that uses RHEL9 Support Tools one as base (registry.redhat.io/rhel9/support-tools) but is built on an entitled RHEL machine, so it has access to all RHEL9 repos during the build.
  • An image that uses RHEL8 Support Tools one as base (registry.redhat.io/rhel8/support-tools) but is built on an entitled RHEL machine, so it has access to all RHEL8 repos during the build.
  • An image that uses RHEL7 Support Tools one as base (registry.access.redhat.com/rhel7/support-tools) but is built on an entitled RHEL machine, so it has access to all RHEL7 repos during the build. This is not recommended though, but it should be preferred to use RHEL 8 or 9, which are more modern.
  • A Fedora image, with access to all the fedora packages (note that we don't support Fedora).
  • Another custom image with the debugging tools of your choice (note that it is very likely you would be using an image not supported by Red Hat).

IMPORTANT: Depending on the image you choose, you may be running versions of different utilities that are incompatible with RHCOS, due to differences in expected kernel versions and several other factors. If you base your image in the support-tools (or UBI) one, it is recommended that you choose the RHEL version that matches the one utilized by RHCOS in your current OCP version (check this solution to find out the version).

How to use custom image with oc debug node/<node>

It is as simple as specifying it on --image parameter.

An example with a Fedora 33 image would be:

$ oc debug node/worker-1.example.com --image=quay.io/fedora/fedora:33-x86_64:latest
Creating debug namespace/openshift-debug-node-xxxxx ...
Starting pod/worker-1examplecom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.0.1
If you don't see a command prompt, try pressing enter.
sh-5.0#

How to use custom image with toolbox

For OCP 4, refer to Starting an alternative image with toolbox in OCP 4.

Note: in OCP 4, if it's not possible to execute an oc debug node, the toolbox container can be executed after connecting to the node using ssh and becoming root.

For OSD 4, refer to Starting an alternative image with toolbox in OSD 4.

Note: a message like Error: registry.fedoraproject.org/f33/fedora-toolbox:latest does not have a label of RUN is not an actual error, but just emitted while checking if the image has a RUN label. If it hasn't, it defaults to running a privileged container with host filesystem mounted (pretty similar to what RHEL8 support tools run label does). So, in most cases, this can be disregarded.

Root Cause

RHEL support tools use ubi repositories that don't have access to all the RHEL content and RHCOS nodes are not subscribed in the usual subscription-manager RHEL way, so toolbox containers cannot get access to RHEL content not available in ubi repositories.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments