Chapter 4. Jenkins

4.1. Overview

The use of CI tooling is often required to gain the full benefit of continuous integration. Jenkins is a popular CI tool, with support for build and deployment to OpenShift environments through plugins and other approaches. Openshift provides a supported Jenkins image. Refer to the official OpenShift documentation for details on using this image.

This reference architecture explores two scenarios:

  • Using Jenkins outside OpenShift for deployment, where CI tooling is hosted outside of OpenShift and orchestrates the workflow inside OpenShift
  • Using an OpenShift Jenkins image, where OpenShift hosts the CI service, either as a central service or per project

4.2. External Jenkins Installation

4.2.1. Installing the OpenShift Pipeline Plugin

In order for Jenkins to communicate with a remote OpenShift installation, the OpenShift Pipeline Plugin needs to be installed first.

https://wiki.jenkins-ci.org/display/JENKINS/OpenShift+Pipeline+Plugin

Note

This plugin does NOT require the oc binary/CLI to be present. However, for certain features not provided in this plugin, (for example, new-app and start-build with binary source), it’s required to download and install the oc client binary on the machine where Jenkins is installed.

To install the OpenShift Pipeline Plugin, in Jenkins console, click Manage Jenkins, then choose the Manage Plugins link.

Java Servcies Diagram

Then select OpenShift Pipeline Jenkins Plugin

Java Servcies Diagram

4.2.2. Using the OpenShift Pipeline Plugin

After the OpenShift Pipeline Plugin is installed, the Jenkins project will expose more OpenShift related steps and options for Build and Post-build Actions.

Java Servcies Diagram

There is no new-app functionality among these options. We currently recommend that if oc new-app functionality is desired in your Jenkins jobs, to either use the OpenShift Jenkins image, where the oc binary is already provided and can be invoked from the Jenkins jobs, or if not using the OpenShift Jenkins image, include the oc binary in your Jenkins installation.

4.2.3. Security

In order for the Jenkins OpenShift plugin to operate against OpenShift resources, the OpenShift service account for the project(s) being operated against will need to have the necessary role and permissions. In particular, you will need to add the edit role to the default service account in the project those resources are located in.

For this project, called deploy-project, the command is:

oc policy add-role-to-user edit system:serviceaccount:deploy-project:default

Next, obtain the the bearer authorization token that Jenkins uses to talk to OpenShift. Use the commands oc describe serviceaccount default and oc describe secret <secret name> for obtaining actual token strings.

oc describe serviceaccount default
	Name:		default
	Namespace:	deploy-project
	Labels:		<none>

	Image pull secrets:	default-dockercfg-s4iuo

	Mountable secrets: 	default-token-zwlpl
		           	default-dockercfg-s4iuo

	Tokens:            	default-token-8tvtm
		           	default-token-zwlpl



oc describe secret default-token-zwlpl
Name:		default-token-zwlpl
Namespace:	deploy-project
Labels:		<none>
Annotations:	kubernetes.io/service-account.name=default
		kubernetes.io/service-account.uid=624012ea-e194-11e6-b40e-beeffeed0002

Type:	kubernetes.io/service-account-token

Data
====
namespace:	14 bytes
service-ca.crt:	2186 bytes
token:		eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXBsb3ktcHJvamVjdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLXp3bHBsIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnG1Z8DVjWB2QMYc4V6xzLcfO3nQbw65uXYtajDI9rTp1CywaBUypfvm0fN04zhuWI6dUB8AK9DYpwn8a7Qm2nNnZe37AXfeSpH1gi6f91YhNevOhjrsa8_XiUrPbGAheSYln5Wt2gWj_SKpw9xvlq_3n8P1yG2QUxNxsJYBHlO7sUpUXA
ca.crt:		1070 bytes

In the absence of a properly signed certificate that has been signed by a trusted certificate authority, the SKIP_TLS parameter needs to be used to avoid a warning:

Connection unsuccessful: You have not specified SKIP_TLS, but a cert is not accessible.  See https://github.com/openshift/jenkins-plugin#certificates-and-encryption

To avoid this exception, just add a string parameter as shown below, to the project configuration:

Skip TLS Exception

We can test the authorization token by adding a step like Trigger Openshift Build. There is a Test Connection button to test if the Cluster API URL and Authorization Token are correct or not:

Trigger Openshift Build

Once these settings are verified, other plugin configuration can be entered with the same values, for example, Trigger Openshift Deployment:

Trigger Openshift Deployment

4.2.4. The new-app flag

The OpenShift Pipeline Plugin does not provide the new-app flag or its equivalent functionality; this restricts the use of the plugin to existing OpenShift applications.

In cases where it’s absolutely required to start a new OpenShift application from Jenkins, the following two solutions can help bypass this limitation:

1- Install the Publish Over SSH Plugin, and invoke oc client on the OpenShift server.

Publish Over SSH Plugin

Using this plugin, Jenkins can remote execute the oc new-app command on the OpenShift server.

2- Install the oc client binary/CLI on the node where Jenkins is installed. Then invoke oc client from Jenkins.

Further details on these two solutions follows.

4.2.4.1. Publish Over SSH Plugin

The Publish over SSH plugin invokes the existing oc client on the OpenShift server.

After installing the Publish Over SSH Plugin, define the SSH Server in Manage Jenkins, under the Configure System section:

Define SSH Server

With this plugin in place, a new step called Send files or execute commands over SSH can be added and used in projects.

Example 1: Transfer war file to OpenShift server.

Define SSH Server

Console output for the build:

SSH: Transferred 1 file(s)

Example 2: Trigger oc new-app command on OpenShift.

Define SSH Server

Console output for the build:

SSH: EXEC: STDOUT/STDERR from command [oc new-app jboss-eap70-openshift:latest~. --name=presentation] ...
--> Found image 926ec6a (11 weeks old) in image stream "jboss-eap70-openshift" in project "openshift" under tag "latest" for "jboss-eap70-openshift:latest"

    JBoss EAP 7.0
    -------------
    Platform for building and running JavaEE applications on JBoss EAP 7.0

    Tags: builder, javaee, eap, eap7

    * A source build using binary input will be created
      * The resulting image will be pushed to image stream "presentation:latest"
      * Use 'start-build --from-dir=DIR|--from-repo=DIR|--from-file=FILE' to trigger a new build
      * WARNING: a binary build was created, you must specify one of --from-dir|--from-file|--from-repo when starting builds
    * This image will be deployed in deployment config "presentation"
    * Ports 8080/tcp, 8443/tcp, 8778/tcp will be load balanced by service "presentation"
      * Other containers can access this service through the hostname "presentation"

--> Creating resources with label app=presentation ...
    imagestream "presentation" created
    buildconfig "presentation" created
    deploymentconfig "presentation" created
    service "presentation" created
--> Success
    Use 'oc start-build presentation' to start a build.
    Run 'oc status' to view your app.
SSH: EXEC: completed after 600 ms

4.2.4.2. Using CLI with Jenkins

This solution requires the oc client binary/CLI on the node where Jenkins is installed, in order to invoke oc client from Jenkins.

Download command line interface (CLI) tools, as instructed in the official documentation.

After installing the CLI on the same machine where Jenkins is installed, the oc command can be invoked directly using the Execute Shell step. Remember to add CLI to the PATH, so that it can be invoked directly everywhere.

invoke local CLI

The OpenShift Pipeline Plugin has a step called Trigger OpenShift Build, but this step does not have a corresponding option to do binary source deployment. To deploy war files, use a similar approach to the above, to bypass the limitation.

4.3. OpenShift Jenkins Image

There are two major differences in terms of the functionality and usage of the Jenkins image, compared to the Jenkins installation outside of OpenShift:

  • No need to install the OpenShift Pipeline Plugin as it’s already included as part of the Jenkins image.
  • No need to configure security token, etc, in the Jenkins steps.

As documented on OpenShift Pipeline Plugin project page:

If that project is also where Jenkins is running out of, and hence you are using the OpenShift Jenkins image (https://github.com/openshift/jenkins[https://github.com/openshift/jenkins]), then the bearer authorization token associated with that service account is already made available to the plugin (mounted into the container Jenkins is running in at "/run/secrets/kubernetes.io/serviceaccount/token"). So you don't need to specify it in the various build step config panels.

Here is one example of configuring the Trigger OpenShift Build step on the Jenkins image. In contrast to the standalone Jenkins configuration, the only value that needs to be configured here is the BuildConfig name.

invoke local cli

However, there is still no new-app functionality in the Jenkins console. In order to start a new OpenShift application, either run the oc new-app directly in OpenShift before using the Jenkins image, or tigger it from the Jenkins console using Execute Shell.

Below is an example of triggering new-app from the Jenkins console:

invoke local cli

Console output:

[workspace] $ /bin/sh -xe /tmp/hudson6043280415050924590.sh
+ oc new-app jboss-eap70-openshift:latest~. --name=presentation
--> Found image 926ec6a (11 weeks old) in image stream "openshift/jboss-eap70-openshift" under tag "latest" for "jboss-eap70-openshift:latest"

    JBoss EAP 7.0
    -------------
    Platform for building and running JavaEE applications on JBoss EAP 7.0

    Tags: builder, javaee, eap, eap7

    * A source build using binary input will be created
      * The resulting image will be pushed to image stream "presentation:latest"
      * A binary build was created, use 'start-build --from-dir' to trigger a new build
    * This image will be deployed in deployment config "presentation"
    * Ports 8080/tcp, 8443/tcp, 8778/tcp will be load balanced by service "presentation"
      * Other containers can access this service through the hostname "presentation"

--> Creating resources ...
    imagestream "presentation" created
    buildconfig "presentation" created
    deploymentconfig "presentation" created
    service "presentation" created
--> Success
    Use 'oc start-build presentation' to start a build.
    Run 'oc status' to view your app.
Finished: SUCCESS

4.3.1. Jenkins Image Installation

Follow the instructions provided in the official OpenShift documentation to install and configure the Jenkins image.

To configure persistent volumes:

$ oc new-app jenkins-persistent

For the ephemeral mode, where configuration does not persist across pod restarts:

$ oc new-app jenkins-ephemeral
Note

If you instantiate the template against releases prior to v3.4 of OpenShift Container Platform, standard Jenkins authentication is used, and the default admin account will exist with a password of password.

4.4. Examples

4.4.1. Using Jenkins to deploy war files to a remote OpenShift

First create a project "MSA_create_all_services_cli" which builds the demo application through maven.

local cli create all project
local cli maven build

Then, in this project, add the Execute Shell step to invoke the locally installed CLI. Create all six services through the oc new-app command, and create routes to expose the service. This project should only need to be executed once for the whole lifecycle of the OCP application.

invoke local cli
invoke local cli

Since database services and the application route only need to be created once, there is no need to configure any additional steps for them. For Billing, Product, Sales, and _Presentation_services, multiple deployments are required for new code release, so proceed to define four new projects for these services:

redeploy services

For each, use a similar Execute Shell step to invoke the "oc start-build" command for war files.

Redeploy Billing:

redeploy Billing services

Redeploy Product:

redeploy Product services

Redeploy Sales:

redeploy Sales services

Redeploy Presentation:

redeploy Presentation

4.4.2. Using the Jenkins image to deploy with S2I

This example also creates 5 projects.

five projects

The first project, "MSA_create_all_services" uses the Execute Shell step to invoke CLI, creating all six services through the oc new-app command and creating a route to expose a service. This time, services are created through S2I, instead of an already-built war file being deployed in the case of the last example.

create all services through S2I

Since database services and the application route only need to be created once, there is no need to configure any additional steps for them. For Billing, Product, Sales, and _Presentation_services, multiple deployments are required for new code release, so proceed to define four new projects for these services:

Redeploy Billing:

redeploy Billing services

Redeploy Product:

redeploy Product services

Redeploy Sales:

redeploy Sales services

Redeploy Presentation:

redeploy Presentation

This approach uses S2I to build and deploy the application, and by relying on a Jenkins image running in OpenShift, the project configuration is simplified. It is enough to simply use the Trigger OpenShift Build step with the corresponding buildConfig name.

4.5. Experimental Jenkins Plugin

The OpenShift Pipeline Plugin does not have native support for some functionalities, including the new-app and start-build features with binary source. However, there is a new Jenkins plugin with support for various CLI features. This plugin wraps the oc client component. The OpenShift Jenkins Pipeline (DSL) Plugin is not currently supported and this paper therefore places less focus on it.

This plugin provides easier integration with OpenShift; for example, an Openshift Cluster can be configured simply with URL, credentials and the default project name.

Jenkins_DSL_plugin1

Sending custom command to OpenShift using this plugin is much easier. The oc command can be supplied after simply selecting the newly defined Openshift Cluster:

Jenkins_DSL_plugin2
Warning

As of the time of writing, this plugin is experimental and not supported by Red Hat.

4.6. Job DSL plugin

Job DSL is a very popular plugin for building Jenkins jobs. This plugin simplifies creating and managing pipelines and can define Jenkins jobs in plain text, using a Groovy-based domain specific language (DSL).

The DSL language output is a job configuration XML file (config.xml file under the job folder). DSL plugin provides high level API to work with this configuration file, and in the absence of high level API, the DSL supports configuration blocks to directly access the XML content of config.xml.

For example, we will explore using this DSL plugin to build the MSA project in an OpenShift environment.

First, install the Job DSL plugin from Manage Jenkins section of the administration web console.

Jenkins Job DSL

After installation, create a DSL seed job that uses the Free-style project style to run the Scripts.

Jenkins Job DSL

Seed jobs can use the DSL script directly in the console, or use a DSL script on the file system. In this example, a script from the file system is used, as it’s easier to edit, and the script can be stored in a Git repository.

Jenkins Job DSL

Details of the MSA_DSL_seed.groovy script:

The script sets up five jobs, which are: MSA_DSL_create_all, MSA_DSL_redeploy_billing, MSA_DSL_redeploy_product, MSA_DSL_redeploy_sales and MSA_DSL_redeploy_presentation.

It uses configuration blocks to define the same customWorkspace for all five jobs, since they all apply to the same project.

    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }

The script then sets up all jobs to use the same repository, git://github.com/RHsyseng/MSA-EAP7-OSE.git, through the SCM API.

    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

The following sections are similar to previous examples. The script uses the steps API to set up the build steps for each job, they will run a maven build, then will create services through a shell terminal, and finally deploy the war files through the oc client, which is installed locally next to Jenkins.

The full script content follows:

job('MSA_DSL_create_all') {

    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }

    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

    steps {
        maven('-e clean package')

        shell('oc new-app -e MYSQL_USER=product -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=product -e MYSQL_ROOT_PASSWORD=passwd mysql --name=product-db')
        shell('oc new-app -e MYSQL_USER=sales -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=sales -e MYSQL_ROOT_PASSWORD=passwd mysql --name=sales-db')
        shell('mkdir -p /tmp/nocontent')
        shell('sleep 60')

        shell('oc new-app jboss-eap70-openshift:latest~/tmp/nocontent --name=billing-service')
        shell('oc start-build billing-service --from-file=Billing/target/billing.war')

        shell('mkdir Product/target/deploy')
        shell('cp Product/target/product.war Product/target/deploy')
        shell('cp -r Product/configuration Product/target/deploy')
        shell('oc new-app -e MYSQL_USER=product -e MYSQL_PASSWORD=password jboss-eap70-openshift:latest~/tmp/nocontent --name=product-service')
        shell('oc start-build product-service --from-dir=Product/target/deploy')

        shell('mkdir Sales/target/deploy')
        shell('cp Sales/target/sales.war Sales/target/deploy')
        shell('cp -r Sales/configuration Sales/target/deploy')
        shell('oc new-app -e MYSQL_USER=sales -e MYSQL_PASSWORD=password jboss-eap70-openshift:latest~/tmp/nocontent --name=sales-service')
        shell('oc start-build sales-service --from-dir=Sales/target/deploy')

        shell('oc new-app jboss-eap70-openshift:latest~/tmp/nocontent --name=presentation')
        shell('oc start-build presentation --from-file=Presentation/target/ROOT.war')

        shell('oc expose service presentation --hostname=msa.example.com')

    }
}

job('MSA_DSL_redeploy_billing') {
    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }


    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

    steps {
        maven('-e clean package')
        shell('oc start-build billing-service --from-file=Billing/target/billing.war')
    }
}


job('MSA_DSL_redeploy_product') {
    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }


    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

    steps {
        maven('-e clean package')
        shell('mkdir Product/target/deploy')
        shell('cp Product/target/product.war Product/target/deploy')
        shell('cp -r Product/configuration Product/target/deploy')
        shell('oc start-build product-service --from-dir=Product/target/deploy')

    }
}


job('MSA_DSL_redeploy_sales') {
    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }


    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

    steps {
        maven('-e clean package')
        shell('mkdir Sales/target/deploy')
        shell('cp Sales/target/sales.war Sales/target/deploy')
        shell('cp -r Sales/configuration Sales/target/deploy')
        shell('oc start-build sales-service --from-dir=Sales/target/deploy')
    }
}

job('MSA_DSL_redeploy_presentation') {
    configure { project ->
        project / 'customWorkspace' << '${JENKINS_HOME}/workspace/MSA_DSL'
    }


    scm {
        git('git://github.com/RHsyseng/MSA-EAP7-OSE.git')
    }

    steps {
        maven('-e clean package')
        shell('oc start-build presentation --from-file=Presentation/target/ROOT.war')
    }
}

Building the MSA_seed project will generate these sub-projects:

Jenkins Job DSL

Inside each sub-project, the SCM, Maven and build steps are already defined. They are ready to be executed.

Jenkins Job DSL
Jenkins Job DSL

As demonstrated through these example, the Job DSL plugin is a powerful and user friendly plugin that can be used with oc client for deployment on OpenShift. It uses Groovy-based scripts to create new jobs, can invoke other plugins through their API, and to perform any other required function that’s not defined in the existing API, it can use configuration blocks to achieve the same effect. The resulting scripts can be checked into a source code repository, and easily reused in different environments.