Java not found after customize ose-jenkins-agent-base image

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (OCP 4)
  • Red Hat OpenShift Service on AWS (ROSA 4)
  • Red Hat Openshift Dedicated 4 (OSD 4)
  • Azure Red Hat Openshift (ARO 4)

Issue

After customizing the ose-jenkins-agent-base image, install/upgrade python 3.9

FROM registry.redhat.io/openshift4/ose-jenkins-agent-base:latest
RUN yum -y install python39

Jenkins slave pod is not able to start.

When checking the pod logs, we could found Permission denied and /usr/local/bin/run-jnlp-client: line 202: exec: java: not found in the logs

2023/02/08 20:49:19 [go-init] No pre-start command defined, skip
2023/02/08 20:49:19 [go-init] Main command launched : /usr/local/bin/run-jnlp-client
failed to create /var/lib/alternatives/java.new: Permission denied
alternatives version 1.19.1 - Copyright (C) 2001 Red Hat, Inc.
This may be freely redistributed under the terms of the GNU Public License.

usage: alternatives --install <link> <name> <path> <priority>
                    [--initscript <service>]
                    [--family <family>]
                    [--slave <slave_link> <slave_name> <slave_path>]*
       alternatives --remove <name> <path>
       alternatives --auto <name>
       alternatives --config <name>
       alternatives --display <name>
       alternatives --set <name> <path>
       alternatives --list
       alternatives --remove-all <name>
       alternatives --add-slave <name> <path> <slave_link> <slave_name> <slave_path>
       alternatives --remove-slave <name> <path> <slave_name>

common options: --verbose --test --help --usage --version --keep-missing --keep-foreign
                --altdir <directory> --admindir <directory>
OPENSHIFT_JENKINS_JVM_ARCH='', CONTAINER_MEMORY_IN_MB='300', using /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el8_4.x86_64/jre/bin/java
Downloading http://172.31.56.246:80//jnlpJars/remoting.jar ...
+ cd
+ exec java -Duser.home=/home/jenkins -Dcom.redhat.fips=false -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Xmx150m -cp /home/jenkins/remoting.jar hudson.remoting.jnlp.Main -headless -url http://172.31.56.246:80/ -tunnel 172.31.38.95:50000 c235b6ca627029169e8a59b1e8c689a3e31617a7463796ea8f5c81288e24b35a jenkins-worker-python-v4bl4
/usr/local/bin/run-jnlp-client: line 202: exec: java: not found
2023/02/08 20:49:19 [go-init] Main command failed
2023/02/08 20:49:19 [go-init] exit status 127
2023/02/08 20:49:19 [go-init] No post-stop command defined, skip

Resolution

Modify the Dockerfile and add an extra line to re-apply folder permissions

FROM registry.redhat.io/openshift4/ose-jenkins-agent-base:latest
RUN yum -y install python39
RUN yum clean all && \
    chown -R 1001:0 /home/jenkins && \
    chmod -R g+w /home/jenkins && \
    chmod -R 775 /etc/alternatives && \
    chmod -R 775 /var/lib/alternatives && \
    chmod -R 775 /usr/lib/jvm && \
    chmod 775 /usr/bin && \
    chmod 775 /usr/share/man/man1 && \
    chmod 775 /var/lib/origin

Root Cause

From the Base Image Dockerfile, the yum install RUN before a chmod RUN.

USER root
# Install headless Java
RUN INSTALL_PKGS="glibc-langpack-en bc gettext git java-11-openjdk-headless java-1.8.0-openjdk-headless lsof rsync tar unzip which zip bzip2 jq" && \
    yum install -y --setopt=tsflags=nodocs --disableplugin=subscription-manager $INSTALL_PKGS && \
    rpm -V  $INSTALL_PKGS && \
    yum clean all && \
    mkdir -p /home/jenkins && \
    chown -R 1001:0 /home/jenkins && \
    chmod -R g+w /home/jenkins && \
    chmod -R 775 /etc/alternatives && \
    chmod -R 775 /var/lib/alternatives && \
    chmod -R 775 /usr/lib/jvm && \
    chmod 775 /usr/bin && \
    chmod 775 /usr/share/man/man1 && \
    mkdir -p /var/lib/origin && \
    chmod 775 /var/lib/origin && \
    unlink /usr/bin/java && \
    unlink /usr/bin/jjs && \
    unlink /usr/bin/keytool && \
    unlink /usr/bin/pack200 && \
    unlink /usr/bin/rmid && \
    unlink /usr/bin/rmiregistry && \
    unlink /usr/bin/unpack200 && \
    unlink /usr/share/man/man1/java.1.gz && \
    unlink /usr/share/man/man1/jjs.1.gz && \
    unlink /usr/share/man/man1/keytool.1.gz && \
    unlink /usr/share/man/man1/pack200.1.gz && \
    unlink /usr/share/man/man1/rmid.1.gz && \
    unlink /usr/share/man/man1/rmiregistry.1.gz && \
    unlink /usr/share/man/man1/unpack200.1.gz

yum install -y python39 command will reset folder /var/lib/alternatives/ and /etc/alternatives permission from 775 to 755

So re-apply the folder permission should help with this case. it looks like only /etc/alternatives and /var/lib/alternatives was affected by this case, but it will do no harm to refresh the others too.

Diagnostic Steps

Use oc debug for testing

Before the yum install -y python39

/var/lib/alternatives/ folder permission is 775 drwxrwxr-x

$ oc debug --image registry.redhat.io/openshift4/ose-jenkins-agent-base
Starting pod/image-debug ...
Pod IP: 10.128.2.88
If you don't see a command prompt, try pressing enter.
sh-4.4# ls -ald /var/lib/alternatives/
drwxrwxr-x. 1 root root 172 Dec  9 23:49 /var/lib/alternatives/

After installing python39, it has been switching back to 755 which caused the permission error

$ oc debug --image registry.redhat.io/openshift4/ose-jenkins-agent-base
Starting pod/image-debug ...
Pod IP: 10.217.1.1
If you don't see a command prompt, try pressing enter.
sh-4.4# ls -ald /var/lib/alternatives/
drwxrwxr-x. 1 root root 172 Dec  9 23:49 /var/lib/alternatives/
sh-4.4#
sh-4.4#
sh-4.4# yum install python39 -y
Updating Subscription Management repositories.
... omitted...
Total download size: 13 M
Downloading Packages:
(1/7): python39-setuptools-50.3.2-4.module+el8.5.0+12204+54860423.noarch.rpm                                           1.1 MB/s | 871 kB     00:00
(2/7): python39-pip-wheel-20.2.4-7.module+el8.6.0+13003+6bb2c488.noarch.rpm                                            1.4 MB/s | 1.1 MB     00:00
(3/7): python39-3.9.13-2.module+el8.7.0+17195+44752b34.x86_64.rpm                                                      515 kB/s |  33 kB     00:00
(4/7): python39-setuptools-wheel-50.3.2-4.module+el8.5.0+12204+54860423.noarch.rpm                                     3.0 MB/s | 497 kB     00:00
(5/7): python39-pip-20.2.4-7.module+el8.6.0+13003+6bb2c488.noarch.rpm                                                  2.0 MB/s | 1.9 MB     00:00
(6/7): chkconfig-1.19.1-1.el8.x86_64.rpm                                                                               2.2 MB/s | 198 kB     00:00
(7/7): python39-libs-3.9.13-2.module+el8.7.0+17195+44752b34.x86_64.rpm                                                 8.7 MB/s | 8.2 MB     00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                  7.1 MB/s |  13 MB     00:01
... omitted...
Installed:
  python39-3.9.13-2.module+el8.7.0+17195+44752b34.x86_64                  python39-libs-3.9.13-2.module+el8.7.0+17195+44752b34.x86_64
  python39-pip-20.2.4-7.module+el8.6.0+13003+6bb2c488.noarch              python39-pip-wheel-20.2.4-7.module+el8.6.0+13003+6bb2c488.noarch
  python39-setuptools-50.3.2-4.module+el8.5.0+12204+54860423.noarch       python39-setuptools-wheel-50.3.2-4.module+el8.5.0+12204+54860423.noarch

Complete!
sh-4.4#
sh-4.4# ls -ald /var/lib/alternatives/
drwxr-xr-x. 1 root root 35 Feb 12 15:59 /var/lib/alternatives/
sh-4.4#

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments