Continuous integration + rpmbuild + Docker

Latest response

I have been going over the process of building RPMs and looking at ways to optimise the business and technical process involved in creating RPMs for deployment/consumption. Shipping software for RHEL in RPMs is something I am strangely passionate about and it is something that seems to be severely lacking in the wider market, even in the big end of town (large vendors). The discussion around 'proper' or even 'decent' vendor provided RPMs is a topic on its own, but I really want to start a topic on how to improve the process of creating RPMs and getting developers involved (DevOps!), especially if you have found a way in your own environment and can share some insight!

In a recent project I have been working with a group of extremely bright and progressive developers that preach to me about the endless benefits of continuous integration (CI) in the development process. They want to include the RPM build process into their workflow so a binary RPM is the output. Their build infrastructure builds on check in and runs multiple tests to validate a clean build and functional application output from the process. From a software development perspective, the OS dependant package (RPM) for deployment is currently much further down the line, and happens far less frequently. Currently the packaging of the application for RHEL specific deployment really falls outside the scope of the CI implementation but it is something we have been discussing moving 'up' into the CI automated tests.

So what? this has been done before.. and build servers exist that already pump out nightly builds... etc.

Where my interest lies is moving the process of the RPM build and deployment test so far 'up' in the development process that it can be run/verified with unit tests at each code check in. The greatest challenge to date has been making this happen fast enough to not impact the development process (the tests need to execute quickly).

A couple of methods I have considered are as follows (please suggest more if you have them!):
- Using VMs for the build and deployment validation and then rolling back for next use.
- Using VMs that are spun up or waiting as 'hot spares' to validate deployment same as above
- The latest kid on the block, 'Docker' for both build and deployment steps above

I really want to focus discussion on placing Docker into this role and would appreciate any feedback, positive/negative on anyone's experiences in using Docker in this role (build farms??).

Where I see Docker playing a role is in the following two places:

- RPM build cleanroom -
Using Docker to create a minimal Docker OS shell/image, and then build the the RPM in this environment. This ensures that you have all the build requirements correctly defined in your build and if prerequisite packages are missing the build will fail. On completion, the completed/validated RPM build is recovered and the Docker instance is destroyed. I have done some searching and found others leveraging Docker for rpmbuilds and I definitely think the method can provide some serious benefits.

- Deployment cleanroom -
Deploy the built RPM into what is essentially a Docker 'SOE' instance/cleanroom of the target environment, or a minimal RHEL installation. Once again, this will confirm that prerequisites (think RPM spec file) are correctly defined in the RPM and the deployment in the Docker image can then be validated.

The goal would be to optimise these steps so they can occur at code check-in time. If this occurs, developers will have much higher visibility of the impact the changes they are making to the code base has on the final package (eg. dependencies), which would hopefully bring greater attention to the importance of 'clean' RPMs.

The only current concern I have about leveraging Docker over KVM is that it will cause problems when developing packages that include SELinux components due to the MCS based implementation used for Docker containers. ie. the SELinux contexts within the container aren't representative of a 'real' RHEL environment.

I have heard of developers incorporating RPMs into their CI process but would really like to discuss it from the Sysadmin side and how your current processes work.
Has anyone implemented anything similar to above? Is it something you would do again?
Have you managed to get developers deeply involved in the RPM build process?

Responses

I agree with you about the importance of proper software packaging. It is a vital element to ensure consistent testing and deployment.

I have been involved in a C/C++ project (5+ full time developers). We wrote unit tests for all classes and before we checked in our code, we made sure that it built correctly and ran the unit tests. This was based on makefiles and googletest and ran on a single development system.

In my opinion unit test do not require a vm/docker to run as they are, by nature, elementary. Each unit test should create the required mock objects. Higher level, application tests, interacting with the system configuration, require a vm/docker to make sure each test runs in a defined environment.

Regards,
Siem Korteweg

Siem,

Thanks for the response.

I agree that depending on the project the docker/rpmbuild may need to be further along in the build process eg. if it incorporates multiple components that need to be combined in the packaging process.

What are your thoughts on when the rpmbuild style application tests should be run?, do you have a trigger in your current process? eg. version point release?, nightly?, manually triggered by developers?.

I am always conscious of the fact that the further the rpmbuild process is away from the 'coal face', the less time/focus it will receive and the more likely it will be to be rushed, which will impact package quality. I thought I would aim high initially with regard to where it is integrated in the process, but will likely settle for a bit further down!

When I left this project, the trigger for application level testing was the release near the end of each sprint (every 2 weeks) as we did not (yet) have the time to fully automate application level testing. Once a framework is in place, nightly test runs would be better and they could also be triggered manually.

I do not share your concerns about the rpmbuild process itself. Once the automated build process is in place, it keeps working unless you change the contents of the package. The package quality is the quality of the application, assuming the package process bundles all required files and contains the required pre and post processing.

Automating the testing process will increase the acceptance and use by developers and therefore increase application quality.

We have used koji servers to integrate rpm building into the dev cycle. We often do nightly rpm builds, which can then kick off builds which do a level of integration testing. There is no reason you can not take this same concept to move builds into every commit. My question would be, how often to commits actually impact the spec file? Would a daily rpm build be enough to catch them?

Thanks for the response Bryan,

It's not just changes to the spec I am concerned about catching/testing. The interest lies in developers making changes in the application that aren't captured in the spec eg:
- Did the code change add files that aren't correctly captured/deployed in the rpm?
- Are additional files generated as part of the general running of the application (ie. non-deployed) that aren't correctly cleaned up when the rpm is removed?
- Has an additional library requirement been added that isn't captured in the rpm spec as a dep? (so deploying the rpm to a clean room results in an application that won't run because of missing dependencies)

The benefit I see of having this at commit is you can quickly determine exactly which change/commit caused the problem, but depending on number of commits in a day, this decision is probably more environment specific.