Using RHN Satellite for a dev/test/QA/Prod environment
I think I get this...
Let's say I have a base channel named rhel-x86_64-server-6. Let's say all my production servers are patched to be current as of, say, Aug. 1, 2013 and I have a one month patch cycle. Everything is registered with my local RHN Satellite and I keep it synced with the real RHN with a cron job that runs every weekend.
I want to have an environment with development, test, QA, and production servers, most of them virtual in a RHEV environment.
I think this is how I would use Satellite for this - tell me if I have this right.
On Aug. 1, I would clone the rhel-x86_64-server-6 channel and name the clone, say, RHEL-Test. I could probably even automate this somehow with a cron job. I would point yum on my test servers to this repository channel, yum update them, then run them through my application tests. Let's say during the testing, I run into an errata that breaks an app. I note it and remove the errata from my RHEL-Test channel. After a while, all those systems test good, so I clone my RHEL-Test channel and call this clone, say, RHEL-QA. I snapshot my production VMs and yum update them, now using the repository from RHEL-QA. The production VMs all look good, so now I clone RHEL-QA and call this one, say, RHEL-Prod, collapse the snapshots of my production servers, and point their yum repositories to RHEL-Prod.
By the time I'm done with this whole cycle, most of August is come and gone and I start the whole thing over again on Sept. 1 with a brand new RHEL-Test clone.
Am I getting the gist of this?
OK - so when I build new systems, I probably want my activation key and kickstart profile pointing to the "real", most up to date rhel-x86_64-server-6 channel, because this is my development channel with the newest RPMs of everythig - right? And then after a while, I would insert this new system into my dev/test/QA/Prod environment by updating its yum repository accordingly.
Great - so how do I point yum to the correct repositories in my various categories of systems? I'll bet there's something inside that template bootstrap.sh script for that. So then, as long as I keep my clone names consistent, my various groups of systems will have their own variations of relevant pieces of bootstrap.sh to point their yums to the correct repositories. Or I can keep those scripts as a config channel inside my Satellite server and apply them as needed I guess.
Am I getting this?
thanks
- Greg Scott
Responses
Greg,
There are two possible scenarios for managing monthly patching. One that I have heard about, but have not used would be the following:
- Create separate Channels & Activation Keys for your different environments (Dev, QA, Production)
- For any existing servers, subscribe them to the appropriate channels and child channels
- 1st of month, Satellite server runs satellite-sync
- Merge/Clone packages and errata from the RHEL channels to the Dev channel and update all Dev servers
- Once you're comfortable doing so, merge/clone your Dev channel to your QA channel and update QA servers
- Again, when ready, merge/clone from your QA channel to Production and update those servers.
This can be done according to a set schedule. I like the cleanliness of this model, but the Activation Key management was more that I was comfortable with. Therefore, what I have been doing is:
- Created custom channels and all my servers are subscribed to these channels
- 1st of the month, Satellite server runs satellite-sync
- 1st Monday of the month I do a merge/clone errata to my custom channels and deploy all errata to non-production systems (after hours)
- After two week period, I deploy patches to production servers based on maintenance windows.
My custom channels are based solely on the version of the OS. So I have a channel for RHEL5 32-bit, one for RHEL5 64-bit and one for RHEL6 64-bit. They have child channels for RHN Tools, Supplimentary, JBoss EWS and that sort of thing.
If you'd like more specific details, let me know. Otherwise, hopefully you can use one of these two methods. The first is better if you want to make sure nobody accidentally deploys unvetted patches into your production environment. Since I'm the only one doing patching, I'm not particularly worried about it in my environment.
Hope that helps some.
Greg,
Glad that helped. I know that I've read about the spacewalk-clone-by-date before, but honestly I don't remember much about it or what I was looking for when I came across it.
As for how I do the merge, the basic thing we do is:
- Satellite-sync so all new updates are syncd to my Satellite
- Log into Satellite Web UI and go to Channels --> Manage Software Channels
-
Go to the first custom channel, to the Packages tab for that channel and do a Compare
Just be careful when you do the compare...you don't want a 64-bit channel comparing to a 32-bit channel or you'll get a bunch of unwanted packages in your custom channel. Not fun!!
-
If the compare finds packages that aren't in the custom channel, go to the bottom of the page and select "Merge Differences"
-
I usually choose "Make identical..." and do Preview Merge
- At the bottom of the next page, do Select All (which then puts you back at the top of the page...a bit annoying) then go back to the bottom and do Merge Packages
- Back to the bottom again to Confirm
That will get all the packages into your custom channel, but then you'll probably still want to get all the Errata to be identified in your custom channel. At this point, they'll all just show up as Packages with no Errata. So, you'll need to clone the Errata.
- From Channels --> Manage Software Channels --> Your custom channel --> Errata tab, go to Clone
-
You'll get a list of errata and have to option to "Merge w/
" or "Clone as ..." I usually leave the default and at the bottom of the page do "Clone Errata" -
Back to the bottom of the next page, do "Update Errata"
Depending on how much Errata there is to update, this can take a few minutes. Your errata may not show up as such immediately, but if you go to they Systems tab, any servers that subscribe to the channel you just cloned and updated will eventually show some number of errata to be updated and some larger number of packages (Errata are all Packages, but not all Packages are Errata).
Another way to do it is to go to the Errata tab and then go to "Clone Errata" on the left. I haven't done it that way, but I suspect that may even be easier/faster. I just have to try it and let it sit long enough to find out if it does the same thing. I always ended up getting impatient, though and using the steps outlined above so when I started seeing errata show up, I wasn't sure if it was because I did Errata --> Clone Errata or if I went through and cloned each individual custom channel.
Good luck!!
We also have dev/test/prod host tiers, and the need to patch those tiers on a regular basis. At a high level, our approach is:
a) Prior to dev patching day, run spacewalk-clone-by-date to create new software channels:
'patching_august2013_rhel5'
'patching_august2013_rhel6'
b) Just prior to dev patching day, run a python script that uses the Satellite API to subscribe the dev systems to the appropriate channel based on RHEL release. The python script can either parse the system names to id dev servers (we have a standard hostnaming scheme), or use system group membership to id 'em.
c) On dev patching day, with the dev systems subscribed to the current patch channels, use Satellite to apply errata.
d) One week later, repeat steps b) and c) for the test systems.
e) One week later, repeat steps b) and c) for the prod systems.
f) Two weeks later (approx.), start over at step a).
In this way we're not really changing the channels once they're created. We just subscribe each set of hosts to the channels as we work through the cycle. After prod patching day, all hosts are on the same channel 'til the next cycle starts.
spacewalk-clone-by-date has done a nice job of managing packages and errata, which is a mixed blessing:
it keeps us from having to clone/copy/move errata and packages in the gui, but ...
it keeps us from gaining experience cloning/copying/moving errata and packages in the gui.
Greg,
Generally changing base channels on systems causes problems later, as you found. There are a few other cloning threads in the Satellite discussion group and I put together a blog on using clone-by-date to create a workflow where you move content not systems. http://blogs.dlt.com/content-workflow-rhn-satellite/
If you are set on using new base channels, Activation Keys are not the way to go. As you also found, those only take effect during provisioning. You can change Base and Child Channels in the GUI or via the API.
Second thought, wouldn't you want the newly built systems to match the environment, not the latest and greatest? If a new production system is built from a development channel, you will introduce inconsistencies that could lead to problems down the road.
Sorry if this is late.
-Matt
OK, so the date selection seems to be a common theme in a lot of spacewalk-clone-by-date discussions. I decided to look at the code and see how it selects the date to compare to the user input to_date. That code looks like
if self.use_update_date:
date_to_use = 'update_date'
else:
date_to_use = 'issue_date'
print "Using ", date_to_use
And it looks like it should log that info... Only problem is, that option is in the git upstream (-z/--use-update-date) so I believe it currently only uses issue_date on the package for comparison. If EPEL packages aren't publishing issue_date (which I'm not seeing on my local EPEL 5 repo), then clone-by-date probably just does the repo diff to determine what needs cloning.
That's just a quick analysis though, I could be very wrong.
-Matt
**** SUBSCRIPTION TO CUSTOM CHANNEL
This is the snippet of code we use to subscribe our hosts to channels based on environment (our naming standard starts with pd|qa|dv|tt, etc...) which we execute via bootstraph.sh
CLIENTHOSTNAME=hostname | cut -f1 -d.
SETTING THE ACTIVATION_KEYS IS A 2-STEP PROCESS
FIRST - FIGURE OUT IF THE HOST IN QA/DV/PD
PARSE CLIENTHOSTNAME STARTING WITH THE 6th CHAR THEN READ THE NEXT 6
SERVER_ENV=${CLIENTHOSTNAME: 0:2}
case $SERVER_ENV in
dv)
ACTIVATION_KEYS_ENV="1-dv-"
;;
qa)
ACTIVATION_KEYS_ENV="1-qa-"
;;
pd)
ACTIVATION_KEYS_ENV="1-pd-"
;;
*)
echo "ERROR: Unknown Host Environment"
echo "I.e. hostname should being with [dv|qa|pd]"
exit 9
;;
esac
*** CLONE BY DATE
The clone by date was a bit challenging to figure out how to NOT have a ton of manual intervention. At this point, there is only 1 step that is manual. The following script will create a .config file for each channel and environment (RHEL-5/6 and qa-pd)... after it does that, a human needs to go in and remove the last ',' in the section which describes the channel definitions. At this point, it simply displays the commands to run to the terminal so you can run them manually.
!/bin/bash
BASE_CHANNELS="rhel-x86_64-server-6" ENVS="dv pd"
SPACECMD config can be found in ~/.spacecmd/config OR SET THE FOLLOWING
SPACEADMIN="satmgr" SPACEPASS="satmgr01" SPACECMD="spacecmd -u $SPACEADMIN -p $SPACEPASS -s localhost"
FIRST CREATE A LIST OF CHILD CHANNELS FOR EACH BASE
for CHAN in $BASE_CHANNELS
do
echo ""
echo "$SPACECMD softwarechannel_listchildchannels $CHAN > ${CHAN}-childchannels.out"
$SPACECMD softwarechannel_listchildchannels $CHAN > ${CHAN}-childchannels.out
for ENV in $ENVS
do
echo "$ENV"
cat << EOF > ./.config-1-${ENV}-default-${CHAN}
{
"username":"satmgr",
"password":"satmgr01",
"assumeyes":true,
"skip_depsolve":false,
"security_only":false,
"blacklist": {
},
"removelist": {
},
"channels":[
{
EOF
echo "\"${CHAN}\": \"${ENV}-$CHAN\", " >> ${CHAN}-childchannels.out.tmp
for CHILDCHAN in cat ${CHAN}-childchannels.out
do
echo "\"${CHILDCHAN}\": \"${ENV}-$CHILDCHAN\", " >> ${CHAN}-childchannels.out.tmp
done
cat ${CHAN}-childchannels.out.tmp >> ./.config-1-${ENV}-default-${CHAN} cat << EOF >> ./.config-1-${ENV}-default-${CHAN} } ] } EOF echo "spacewalk-clone-by-date --config=/root/bin/.config-1-${ENV}-default-${CHAN} --to_date=2012-11-01" done done
echo "!!!!!!!!!!!! NOTE NOTE NOTE !!!!!!!!!!!! " echo "NOTE: You will need to remove the last comma ',' from each of the config files before you can run the clone" echo "!!!!!!!!!!!! NOTE NOTE NOTE !!!!!!!!!!!! " exit 0
cleanup() { rm -f .config-1* rm -f rhel-x86_64-server-?.tmp rm -f rhel-x86_64-server-?-childchannels.* }
remove() { echo "NOTE NOTE NOTE" echo " This is seriously distructive" exit 9
BUILD OUT THE LOGIC TO REMOVE THE ENV_SPECIFIC CHANNELS USING THE FOLLOWING
echo "spacewalk-remove-channel --unsubscribe --channel=${ENV}-${CHAN}"
}
Check out these articles in the KBase:
SOE Document for Sat5: https://access.redhat.com/articles/1156083
SOE Document for Sat6: https://access.redhat.com/articles/1169613
SOE-ish clone-by-date in-depth document: https://access.redhat.com/articles/469173
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
