Jenkins POD fails to start after updating plugins

Solution Verified - Updated -

Environment

  • Red Hat Openshift Container Platform (Openshift)
    • 3
    • 4
  • Jenkins / CICD (Continuous Integration Continuous Delivery)
  • Updating / installing common plugins

Issue

  • Pod shows as not ready

  • logs show errors like (Specific plugin not important):

    SEVERE: Failed Loading plugin Jenkins Git client plugin v3.2.1 (git-client)
java.io.IOException: Jenkins Git client plugin version 3.2.1 failed to load.
- Script Security Plugin version 1.74 failed to load. Fix this plugin first.

Resolution

  1. The namespaces to use need to be defined. The commands use a variable or placeholder for the image storage namespace called IMAGE_NAMESPACE. It also uses JENKINS_NAMESPACE for the namespace jenkins will be deployed into. The namespace "openshift" is special in that all the other namespaces will search there, plus the jenkins deployment is configured by default to use that for updates. this is the recommended place to put it, but it can also be put in the same namespace as the jenkins deployment, particularly in cases where the user doesn't have permissions to write to the "openshift" namespace. In these cases IMAGE_NAMESPACE and JENKINS_NAMESPACE will be the same.

  2. Login to the registry. See root cause for more information. Be sure to define the 4 variables.

    oc -n $IMAGE_NAMESPACE create secret docker-registry registry \
--docker-server=registry.redhat.io \
--docker-username=$DOCKER_USER_ID \
--docker-password=$DOCKER_PASSWORD \
--docker-email=$DOCKER_EMAIL

oc -n $IMAGE_NAMESPACE secrets link default registry --for=pull
oc -n $IMAGE_NAMESPACE secrets link builder registry

  3. Update the version of Jenkins. The image tags are relevant for Openshift 3 at the time of writing.

    $ oc -n $IMAGE_NAMESPACE import-image openshift3/jenkins-2-rhel7 --from=registry.redhat.io/openshift3/jenkins-2-rhel7 --confirm

    Find the correct Jenkins Image for your Openshift version. Pay special attention to the openshift version path "openshift3/" vs "openshift4/" for example. You can also at a tag to the end with ":" such as openshift3/jenkins-2-rhel7:latest

  4. Update the deployment config to pull the new image. This can also be done from the console.

    $ oc -n $JENKINS_NAMESPACE edit dc jenkins

    1. Change the image pull policy in the yaml to Always.

      imagePullPolicy: Always

    2. Check the triggers. This is the default text in the spec

      spec:
    ...
    triggers:
    ...
    - imageChangeParams:
        automatic: true
        containerNames:
        - jenkins
        from:
            kind: ImageStreamTag
            name: jenkins:2
            namespace: openshift
        lastTriggeredImage: image-registry.openshift-image-registry.svc:5000/openshift/jenkins@...

      Pay special attention to use the same namespace as the IMAGE_NAMESPACE variable, i.e replace the placeholder here with the actual value, and make the name the same as the image you pulled in the earlier step without the openshiftx/ prefix.

      from: 
    ...
    name: jenkins-2-rhel7:latest
    namespace: <IMAGE_NAMESPACE>
    ...

Root Cause

Jenkins is not up-to-date, so the image needs to be updated or the current plugin versions won't work.

For Openshift Dedicated or Red Hat Managed Integration (RHMI), you will need to pull the image into the jenkins project namespace because customer admin users don't have access to modify the openshift namespace.

When logging into registries to store images, the registry login and secrets need to be implemented in each namespace. For other examples than the one in the resolution, see RegistryAuthentication.

Diagnostic Steps

  • Check the logs with

    $ oc logs jenkins-#-<pod>

    Where the # is the number of the deployment and
    is the pod id. $ oc get pods will show the pods.

    In bash, an easy way to get the pod is oc get pods | grep 'jenkins.*Running' | awk '{ print $1; }', and it can be made one command with:

    $ oc logs $(oc get pods | grep 'jenkins.*Running' | awk '{ print $1; }')

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments