Jenkins POD fails to start after updating plugins
Environment
- Red Hat Openshift Container Platform (Openshift)
- 3
- 4
- Jenkins / CICD (Continuous Integration Continuous Delivery)
- Updating / installing common plugins
Issue
- Pod shows as not ready
-
logs show errors like (Specific plugin not important):
SEVERE: Failed Loading plugin Jenkins Git client plugin v3.2.1 (git-client) java.io.IOException: Jenkins Git client plugin version 3.2.1 failed to load. - Script Security Plugin version 1.74 failed to load. Fix this plugin first.
Resolution
-
The namespaces to use need to be defined. The commands use a variable or placeholder for the image storage namespace called
IMAGE_NAMESPACE
. It also usesJENKINS_NAMESPACE
for the namespace jenkins will be deployed into. The namespace "openshift" is special in that all the other namespaces will search there, plus the jenkins deployment is configured by default to use that for updates. this is the recommended place to put it, but it can also be put in the same namespace as the jenkins deployment, particularly in cases where the user doesn't have permissions to write to the "openshift" namespace. In these casesIMAGE_NAMESPACE
andJENKINS_NAMESPACE
will be the same. -
Login to the registry. See root cause for more information. Be sure to define the 4 variables.
oc -n $IMAGE_NAMESPACE create secret docker-registry registry \ --docker-server=registry.redhat.io \ --docker-username=$DOCKER_USER_ID \ --docker-password=$DOCKER_PASSWORD \ --docker-email=$DOCKER_EMAIL oc -n $IMAGE_NAMESPACE secrets link default registry --for=pull oc -n $IMAGE_NAMESPACE secrets link builder registry
-
Update the version of Jenkins. The image tags are relevant for Openshift 3 at the time of writing.
$ oc -n $IMAGE_NAMESPACE import-image openshift3/jenkins-2-rhel7 --from=registry.redhat.io/openshift3/jenkins-2-rhel7 --confirm
Find the correct Jenkins Image for your Openshift version. Pay special attention to the openshift version path "openshift3/" vs "openshift4/" for example. You can also at a tag to the end with ":" such as
openshift3/jenkins-2-rhel7:latest
-
Update the deployment config to pull the new image. This can also be done from the console.
$ oc -n $JENKINS_NAMESPACE edit dc jenkins
-
Change the image pull policy in the yaml to Always.
imagePullPolicy: Always
-
Check the triggers. This is the default text in the spec
spec: ... triggers: ... - imageChangeParams: automatic: true containerNames: - jenkins from: kind: ImageStreamTag name: jenkins:2 namespace: openshift lastTriggeredImage: image-registry.openshift-image-registry.svc:5000/openshift/jenkins@...
Pay special attention to use the same namespace as the IMAGE_NAMESPACE variable, i.e replace the placeholder here with the actual value, and make the name the same as the image you pulled in the earlier step without the
openshiftx/
prefix.from: ... name: jenkins-2-rhel7:latest namespace: <IMAGE_NAMESPACE> ...
-
Root Cause
Jenkins is not up-to-date, so the image needs to be updated or the current plugin versions won't work.
For Openshift Dedicated or Red Hat Managed Integration (RHMI), you will need to pull the image into the jenkins project namespace because customer admin users don't have access to modify the openshift namespace.
When logging into registries to store images, the registry login and secrets need to be implemented in each namespace. For other examples than the one in the resolution, see RegistryAuthentication.
Diagnostic Steps
-
Check the logs with
$ oc logs jenkins-#-<pod>
Where the # is the number of the deployment and
is the pod id. $ oc get pods
will show the pods.In bash, an easy way to get the pod is
oc get pods | grep 'jenkins.*Running' | awk '{ print $1; }'
, and it can be made one command with:$ oc logs $(oc get pods | grep 'jenkins.*Running' | awk '{ print $1; }')
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments