Continuous integration + rpmbuild + Docker
I have been going over the process of building RPMs and looking at ways to optimise the business and technical process involved in creating RPMs for deployment/consumption. Shipping software for RHEL in RPMs is something I am strangely passionate about and it is something that seems to be severely lacking in the wider market, even in the big end of town (large vendors). The discussion around 'proper' or even 'decent' vendor provided RPMs is a topic on its own, but I really want to start a topic on how to improve the process of creating RPMs and getting developers involved (DevOps!), especially if you have found a way in your own environment and can share some insight!
In a recent project I have been working with a group of extremely bright and progressive developers that preach to me about the endless benefits of continuous integration (CI) in the development process. They want to include the RPM build process into their workflow so a binary RPM is the output. Their build infrastructure builds on check in and runs multiple tests to validate a clean build and functional application output from the process. From a software development perspective, the OS dependant package (RPM) for deployment is currently much further down the line, and happens far less frequently. Currently the packaging of the application for RHEL specific deployment really falls outside the scope of the CI implementation but it is something we have been discussing moving 'up' into the CI automated tests.
So what? this has been done before.. and build servers exist that already pump out nightly builds... etc.
Where my interest lies is moving the process of the RPM build and deployment test so far 'up' in the development process that it can be run/verified with unit tests at each code check in. The greatest challenge to date has been making this happen fast enough to not impact the development process (the tests need to execute quickly).
A couple of methods I have considered are as follows (please suggest more if you have them!):
- Using VMs for the build and deployment validation and then rolling back for next use.
- Using VMs that are spun up or waiting as 'hot spares' to validate deployment same as above
- The latest kid on the block, 'Docker' for both build and deployment steps above
I really want to focus discussion on placing Docker into this role and would appreciate any feedback, positive/negative on anyone's experiences in using Docker in this role (build farms??).
Where I see Docker playing a role is in the following two places:
- RPM build cleanroom -
Using Docker to create a minimal Docker OS shell/image, and then build the the RPM in this environment. This ensures that you have all the build requirements correctly defined in your build and if prerequisite packages are missing the build will fail. On completion, the completed/validated RPM build is recovered and the Docker instance is destroyed. I have done some searching and found others leveraging Docker for rpmbuilds and I definitely think the method can provide some serious benefits.
- Deployment cleanroom -
Deploy the built RPM into what is essentially a Docker 'SOE' instance/cleanroom of the target environment, or a minimal RHEL installation. Once again, this will confirm that prerequisites (think RPM spec file) are correctly defined in the RPM and the deployment in the Docker image can then be validated.
The goal would be to optimise these steps so they can occur at code check-in time. If this occurs, developers will have much higher visibility of the impact the changes they are making to the code base has on the final package (eg. dependencies), which would hopefully bring greater attention to the importance of 'clean' RPMs.
The only current concern I have about leveraging Docker over KVM is that it will cause problems when developing packages that include SELinux components due to the MCS based implementation used for Docker containers. ie. the SELinux contexts within the container aren't representative of a 'real' RHEL environment.
I have heard of developers incorporating RPMs into their CI process but would really like to discuss it from the Sysadmin side and how your current processes work.
Has anyone implemented anything similar to above? Is it something you would do again?
Have you managed to get developers deeply involved in the RPM build process?
