Using RHN Satellite for a dev/test/QA/Prod environment

Latest response

I think I get this...

Let's say I have a base channel named rhel-x86_64-server-6. Let's say all my production servers are patched to be current as of, say, Aug. 1, 2013 and I have a one month patch cycle. Everything is registered with my local RHN Satellite and I keep it synced with the real RHN with a cron job that runs every weekend.

I want to have an environment with development, test, QA, and production servers, most of them virtual in a RHEV environment.

I think this is how I would use Satellite for this - tell me if I have this right.

On Aug. 1, I would clone the rhel-x86_64-server-6 channel and name the clone, say, RHEL-Test. I could probably even automate this somehow with a cron job. I would point yum on my test servers to this repository channel, yum update them, then run them through my application tests. Let's say during the testing, I run into an errata that breaks an app. I note it and remove the errata from my RHEL-Test channel. After a while, all those systems test good, so I clone my RHEL-Test channel and call this clone, say, RHEL-QA. I snapshot my production VMs and yum update them, now using the repository from RHEL-QA. The production VMs all look good, so now I clone RHEL-QA and call this one, say, RHEL-Prod, collapse the snapshots of my production servers, and point their yum repositories to RHEL-Prod.

By the time I'm done with this whole cycle, most of August is come and gone and I start the whole thing over again on Sept. 1 with a brand new RHEL-Test clone.

Am I getting the gist of this?

OK - so when I build new systems, I probably want my activation key and kickstart profile pointing to the "real", most up to date rhel-x86_64-server-6 channel, because this is my development channel with the newest RPMs of everythig - right? And then after a while, I would insert this new system into my dev/test/QA/Prod environment by updating its yum repository accordingly.

Great - so how do I point yum to the correct repositories in my various categories of systems? I'll bet there's something inside that template bootstrap.sh script for that. So then, as long as I keep my clone names consistent, my various groups of systems will have their own variations of relevant pieces of bootstrap.sh to point their yums to the correct repositories. Or I can keep those scripts as a config channel inside my Satellite server and apply them as needed I guess.

Am I getting this?

thanks

  • Greg Scott

Responses

Well, the response has been underwhelming so far. The scheme I laid out last week in the original post won't work.

Here's the Satellite model near as I can tell - Satellite uses a structure called an Activation Key to register systems. An Activation key has exactly one base channel and many child channels. Systems use a program named rhnreg_ks to register with the local Satellite and subscribe to the Channels listed in the specified Activation Key. This is important - once a system is registered, the Activation Key that registered it seems to be out of the picture. Change the channels in an Activation Key, and systems already registered with that Activation Key still use the original channels they registered with, not the new base channel specified in the updated Activation Key.

"By the book", my cloned Prod channel should freeze forever, because it's Prod after all, unless the customer adds carefully controlled and tested Errata to it.

What we want to do here is a little different. Instead of a static Prod channel, I want to have a "floating" Prod channel and a monthly patching cycle. So every month, I want to clone rhel-x86_64-server-6 and make a new channel with a unique name. I want to update my Activation Key and then have all systems registered with that Activation Key now subscribe to my new cloned Channel instead of the old one. And then I want to yum update my Prod systems to bring them up to date with my first of the month channel clone. I'll also have test and dev systems with more recent Channel clones, so by the time I'm ready to patch my production systems, the Channel I use will have gone through a test/QA cycle. So I may end up with weekly channel clones, deleting, say, the 5 week old clone every week.

But this doesn't work. When I change the channels in an Activation Key,the systems are still subscribed to the original channels, not the new channels. How do I know this? Well, I tried it with a test system.

I have a channel named rhel-x86_64-server-6. I used rhnreg_ks to register a test client system and, sure enough, yum update said the test system subscribed to the correct RHEL 6 channel when done. Next, I modified the Activation Key to use rhel-x86_64-server-5 as a base channel and saved it. Yum update on my test Satellite client system still showed the original RHEL 6 channels, not the RHEL 5 channels from my Activation Key. Woops. I ran rhnreg_ks again on my test system against my new Activation Key, and yum update now showed a zillion packages in the rhel-x86_64-server-5 channel. I changed the Activation Key again to use a clone RHEL 6 channel named clone-2013-0816-rhel-x86_64-server-6 and ran yum update on my test client system. It still wanted to use my bogus RHEL 5 channel. I ran rhnreg_ks again on my test client and, sure enough, now it's subscribed to my clone RHEL 6 channel.

So every time you change an Activation Key, you need to re-register the systems that use that Activation Key so those systems subscribe to the correct channels. This is doable, except that now all those systems have more than one RHN Satellite profile. So we need to come up with a way to remove the old profile first, then re-register each system. Removing the old profile manually with the Satellite GUI won't work - too many systems, too many profiles. I'm trying to find a scriptable way to remove old system profiles, but coming up short so far.

If I could run a script on a client system to remove its own RHN registration, that would work nicely. Or if I could run a script on the Satellite server to remove a specified list of profiles, that would work too.

Sure I cannot be the first one to come across this - how is everyone else doing it?

thanks

  • Greg Scott

Greg,

There are two possible scenarios for managing monthly patching. One that I have heard about, but have not used would be the following:

  1. Create separate Channels & Activation Keys for your different environments (Dev, QA, Production)
  2. For any existing servers, subscribe them to the appropriate channels and child channels
  3. 1st of month, Satellite server runs satellite-sync
  4. Merge/Clone packages and errata from the RHEL channels to the Dev channel and update all Dev servers
  5. Once you're comfortable doing so, merge/clone your Dev channel to your QA channel and update QA servers
  6. Again, when ready, merge/clone from your QA channel to Production and update those servers.

This can be done according to a set schedule. I like the cleanliness of this model, but the Activation Key management was more that I was comfortable with. Therefore, what I have been doing is:

  1. Created custom channels and all my servers are subscribed to these channels
  2. 1st of the month, Satellite server runs satellite-sync
  3. 1st Monday of the month I do a merge/clone errata to my custom channels and deploy all errata to non-production systems (after hours)
  4. After two week period, I deploy patches to production servers based on maintenance windows.

My custom channels are based solely on the version of the OS. So I have a channel for RHEL5 32-bit, one for RHEL5 64-bit and one for RHEL6 64-bit. They have child channels for RHN Tools, Supplimentary, JBoss EWS and that sort of thing.

If you'd like more specific details, let me know. Otherwise, hopefully you can use one of these two methods. The first is better if you want to make sure nobody accidentally deploys unvetted patches into your production environment. Since I'm the only one doing patching, I'm not particularly worried about it in my environment.

Hope that helps some.

Thanks Dan - yes, that helps a bunch. So the general idea is, don't make new cloned channels every month, merge the new content from the "real" channels into the cloned channels periodically. Yes, much much much cleaner. How do you do the merging?

I also logged a support case and a utility named spacewalk-clone-by-date was mentioned. Looking at the docs - Chapter 4 of the Satellite Users Guide has a section about this, but the title, "Cloning a machine", is funny. The idea is, I clone a source channel up to the specified date into the specified target channel. This looks promising.

  • Greg

Greg,

Glad that helped. I know that I've read about the spacewalk-clone-by-date before, but honestly I don't remember much about it or what I was looking for when I came across it.

As for how I do the merge, the basic thing we do is:

  1. Satellite-sync so all new updates are syncd to my Satellite
  2. Log into Satellite Web UI and go to Channels --> Manage Software Channels
  3. Go to the first custom channel, to the Packages tab for that channel and do a Compare

    Just be careful when you do the compare...you don't want a 64-bit channel comparing to a 32-bit channel or you'll get a bunch of unwanted packages in your custom channel. Not fun!!

  4. If the compare finds packages that aren't in the custom channel, go to the bottom of the page and select "Merge Differences"

  5. I usually choose "Make identical..." and do Preview Merge

  6. At the bottom of the next page, do Select All (which then puts you back at the top of the page...a bit annoying) then go back to the bottom and do Merge Packages
  7. Back to the bottom again to Confirm

That will get all the packages into your custom channel, but then you'll probably still want to get all the Errata to be identified in your custom channel. At this point, they'll all just show up as Packages with no Errata. So, you'll need to clone the Errata.

  1. From Channels --> Manage Software Channels --> Your custom channel --> Errata tab, go to Clone
  2. You'll get a list of errata and have to option to "Merge w/" or "Clone as ..." I usually leave the default and at the bottom of the page do "Clone Errata"

  3. Back to the bottom of the next page, do "Update Errata"

Depending on how much Errata there is to update, this can take a few minutes. Your errata may not show up as such immediately, but if you go to they Systems tab, any servers that subscribe to the channel you just cloned and updated will eventually show some number of errata to be updated and some larger number of packages (Errata are all Packages, but not all Packages are Errata).

Another way to do it is to go to the Errata tab and then go to "Clone Errata" on the left. I haven't done it that way, but I suspect that may even be easier/faster. I just have to try it and let it sit long enough to find out if it does the same thing. I always ended up getting impatient, though and using the steps outlined above so when I started seeing errata show up, I wasn't sure if it was because I did Errata --> Clone Errata or if I went through and cloned each individual custom channel.

Good luck!!

We also have dev/test/prod host tiers, and the need to patch those tiers on a regular basis. At a high level, our approach is:

a) Prior to dev patching day, run spacewalk-clone-by-date to create new software channels:
'patching_august2013_rhel5'
'patching_august2013_rhel6'

b) Just prior to dev patching day, run a python script that uses the Satellite API to subscribe the dev systems to the appropriate channel based on RHEL release. The python script can either parse the system names to id dev servers (we have a standard hostnaming scheme), or use system group membership to id 'em.

c) On dev patching day, with the dev systems subscribed to the current patch channels, use Satellite to apply errata.

d) One week later, repeat steps b) and c) for the test systems.

e) One week later, repeat steps b) and c) for the prod systems.

f) Two weeks later (approx.), start over at step a).

In this way we're not really changing the channels once they're created. We just subscribe each set of hosts to the channels as we work through the cycle. After prod patching day, all hosts are on the same channel 'til the next cycle starts.

spacewalk-clone-by-date has done a nice job of managing packages and errata, which is a mixed blessing:
it keeps us from having to clone/copy/move errata and packages in the gui, but ...
it keeps us from gaining experience cloning/copying/moving errata and packages in the gui.

Greg,

Generally changing base channels on systems causes problems later, as you found. There are a few other cloning threads in the Satellite discussion group and I put together a blog on using clone-by-date to create a workflow where you move content not systems. http://blogs.dlt.com/content-workflow-rhn-satellite/

If you are set on using new base channels, Activation Keys are not the way to go. As you also found, those only take effect during provisioning. You can change Base and Child Channels in the GUI or via the API.

Second thought, wouldn't you want the newly built systems to match the environment, not the latest and greatest? If a new production system is built from a development channel, you will introduce inconsistencies that could lead to problems down the road.

Sorry if this is late.

-Matt

Not late at all - this is all great stuff - thanks! I ended up using spacewalk-clone-by-date and a little script to do all the base and child channels we care about.

I have some cron jobs that do satellite-sync every Saturday night. So then every month around the 2nd or 3rd, I'll have another cron job that does spacewalk-clone-by-date as of the first of the current month. My clone channels are all labeled clone-prod-{channellabel}.

Production systems will subscribe to clone-prod-{channellabel}, dev systems to the real Red Hat channels. And if we get fancier with test and QA groups, we can accommodate that too.

Why not just use the GUI for the clone/merge every month? Well, from what I can see on the GUI, it clone/merge merges everything as of right now, not as of a specified date in the past. Also, I want to automate as much as possible and I can script out all the cloning and do it with cron jobs. So then patching doesn't depend on clicking a bunch of GUI buttons every month.

But nothing is simple. Spacewalk-clone-by-date apparently depends on repository cache directories in /var/cache/rhn/repodata. Satellite is supposed to populate that directory with repository metadata for all the base and child channels. But there's a known issue and naturally it doesn't always work properly without the latest and greatest Satellite updates. So most of the repodata directories for child channels are missing, and this breaks spacewalk-clone-by-date. Here is a link to one less than clear writeup on the problem:

https://access.redhat.com/site/solutions/319983

Put in all the latest updates and update the database schema and Satellite no longer messes up creating new repodata directories. But the missing directories are still missing and you have to create them, one by one, by hand. I have a script for that now too, based on that KB article.

So when I did spacewalk-clone-by-date against the child channels I cared about, it blew up with an error about missing repo data. That triggered several days of scrambling and homework, but I think I finally have it.

I'll post what I have so far when I get onsite.

  • Greg

Here is a script to regenerate those broken/missing repodata directories.

#!/bin/sh
# Regenerate yum cache repodata in case it gets messed up.  See the 
# directory /var/cache/rhn/repodata.  Also see the Red Hat KB article at
# https://access.redhat.com/site/solutions/319983
#
#  Pass one parameter, the label of the channel you want to regenerate
#
# Modify hostnames and directories as appropriate.

CHANNEL="$1"

echo ""
echo "See Red Hat KB article: https://access.redhat.com/site/solutions/319983"

if [ "$CHANNEL" =  "" ]
then
    read -p "Enter the RHN label for the channel you want to fix up? " CHANNEL
    if [ "$CHANNEL" =  "" ]
    then
        exit
    fi
fi

# Protect login info for the Satellite server.
# The .conf file below needs 2 lines:
# USER={AdminUser}
# PASS={AdminPass}
# Make sure to chmod it to 700 so the world can't see it.

. /var/www/html/pub/localscripts/includes/local-satellite-login.conf

spacewalk-api --server=satserver.example.com \
    --user=$USER --password=$PASS \
    channel.software.regenerateYumCache %session% $CHANNEL

echo ""
echo "Next steps:"
echo ""
echo "Login to the RHN Satellite WebUi"
echo "- Click on the \`Admin\` tab"
echo "- Click on \`Task Schedules\` on the left hand menu"
echo "- Click on \`Channel-repodata-bunch\` in the right-hand column."
echo "- Click on the \"Single Run Schedule\" button."
echo ""
echo "Monitor /var/log/rhn/rhn_taskomatic_daemon.log file for progress in"
echo "repodata re-generation and confirm whether repodata files got created under"
echo "/var/cache/rhn/repodata/channel-label directory or not."
echo ""
echo "# ls -l /var/cache/rhn/repodata/$CHANNEL"
echo ""
echo "Initially we can see files with .new extension which means that repodata"
echo "re-generation is in progress. Once .new extension is removed which means"
echo "that repodata generation is completed."

exit
  • Greg

As promised, here is a script to run spacewalk-clone-by-date and bring the base and child channel clones we care about current as of the first of this month. Modify as you see fit. This site also has a custom channel with RPMs for Ganglia and its dependencies, all downloaded from the Fedora EPEL repository. That's where the references to ganglia come from below.

spacewalk-clone-by-date seems to have some issues keeping my clone of my custom ganglia channel up to date. Since I use the first of this month as the date, I don't know exactly what that means - is this the date the RPM was created or the date it was added to the original channel? I ran into this because it took a few iterations to figure out all the RPMs I needed for ganglia, and when I added new RPMs to my original ganglia custom channel, spacewalk-clone-by-date would not clone those new RPMs into my clone channel. But that may be because I added them to the original after the first of the month - but if this is so, then why did spacewalk-clone-by-date add the original RPMs to my clone channel in the first place, since they were all added after the first of this month. So I'm still a little suspicious of spacewalk-clone-by-date, although I'm using the script below in a monthly cron job here.

  • Greg
#!/bin/sh

# clone-prod.sh - Clone RHEL 6-64 and RHEL 5-32 and 64 base and selected child 
# channels current as of the first of this month.  Production systems will
# update themselves from these cloned channels.
# Greg Scott 2013-Aug-22

date

# Protect the login info for the Satellite server.
# The file referenced below needs two lines:
# USER={SatelliteAdminUser}
# PASS={SatelliteAdminPassword}
# Be sure to chmod it to 700 so it's only visible to root.

. /var/www/html/pub/mylocalscripts/includes/satellite-login.conf

##THIS_MONTH="06"   # A couple months old for debugging
THIS_MONTH=$(date +%m)
THIS_YEAR=$(date +%Y)

echo "$(date) Cloning as of $THIS_YEAR-$THIS_MONTH-01"

echo "$(date) Cloning RHEL 6 64 bit base channel"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --channels=rhel-x86_64-server-6 clone-prod-rhel-x86_64-server-6
echo "$(date) Finished cloning RHEL 6 base channel"

echo "$(date) Cloning RHEL 6 64 bit child channels"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --parents=rhel-x86_64-server-6 clone-prod-rhel-x86_64-server-6 \
    --channels=rhel-x86_64-server-supplementary-6 clone-prod-rhel-x86_64-server-supplementary-6 \
    --channels=rhel-x86_64-rhev-mgmt-agent-6 clone-prod-rhel-x86_64-rhev-mgmt-agent-6 \
    --channels=rhn-tools-rhel-x86_64-server-6 clone-prod-rhn-tools-rhel-x86_64-server-6 \
    --channels=ganglia-rhel6-64 clone-prod-ganglia-rhel6-64
echo "$(date) Finished cloning RHEL 6 64 bit child channels"
echo ""

echo "$(date) Cloning RHEL 5 32 bit base channel"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --channels=rhel-i386-server-5 clone-prod-rhel-i386-server-5
echo "$(date)Finished cloning RHEL 5 32 bit base channel"

echo "$(date) Cloning RHEL 5 32 bit child channels"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --parents=rhel-i386-server-5 clone-prod-rhel-i386-server-5 \
    --channels=rhel-i386-server-vt-5 clone-prod-rhel-i386-server-vt-5 \
    --channels=rhn-tools-rhel-i386-server-5 clone-prod-rhn-tools-rhel-i386-server-5 \
    --channels=ganglia-rhel5-32 clone-prod-ganglia-rhel5-32
echo "$(date) Finished cloning RHEL 5 32 bit child channels"
echo ""

echo "$(date) Cloning RHEL 5 64 bit base channel"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --channels=rhel-x86_64-server-5 clone-prod-rhel-x86_64-server-5
echo "$(date)Finished cloning RHEL 5 64 bit base channel"

echo "$(date) Cloning RHEL 5 64 bit child channels"
spacewalk-clone-by-date -y \
    -d "$THIS_YEAR-$THIS_MONTH-01" \
    -u $USER -p $PASS \
    --parents=rhel-x86_64-server-5 clone-prod-rhel-x86_64-server-5 \
    --channels=rhn-tools-rhel-x86_64-server-5 clone-prod-rhn-tools-rhel-x86_64-server-5 \
    --channels=ganglia-rhel5-64 clone-prod-ganglia-rhel5-64
echo "$(date) Finished cloning RHEL 5 64 bit child channels"

date

Greg,

how do you keep organizations and channels ? What organizations you have with what channels ?

And here are the contab entries I use to sync everything:

# Minute  Hour  Day of month  Month  Day of week
0 20 * * 6 sh -v /var/www/html/pub/mylocalscripts/sync_rhel6-64.sh > /var/www/html/pub/mylocallogs/sync_rhel6-64.log 2>&1
0 22 * * 6 sh -v /var/www/html/pub/mylocalscripts/sync_rhel5-64.sh > /var/www/html/pub/mylocallogs/sync_rhel5-64.log 2>&1
0  0 * * 7 sh -v /var/www/html/pub/mylocalscripts/sync_rhel5-32.sh > /var/www/html/pub/mylocallogs/sync_rhel5-32.log 2>&1

0  2 1 * * sh -v /var/www/html/pub/mylocalscripts/sync_rhel6-64.sh > /var/www/html/pub/mylocallogs/sync_rhel6-64.log 2>&1
0  4 1 * * sh -v /var/www/html/pub/mylocalscripts/sync_rhel5-64.sh > /var/www/html/pub/mylocallogs/sync_rhel5-64.log 2>&1
0  6 1 * * sh -v /var/www/html/pub/mylocalscripts/sync_rhel5-32.sh > /var/www/html/pub/mylocallogs/sync_rhel5-32.log 2>&1

0  8 1 * * sh -v /var/www/html/pub/mylocalscripts/clone-prod.sh > /var/www/html/pub/mylocallogs/clone-prod.log 2>&1

OK, so the date selection seems to be a common theme in a lot of spacewalk-clone-by-date discussions. I decided to look at the code and see how it selects the date to compare to the user input to_date. That code looks like

        if self.use_update_date:
            date_to_use = 'update_date'
        else:
            date_to_use = 'issue_date'
         print "Using ", date_to_use

And it looks like it should log that info... Only problem is, that option is in the git upstream (-z/--use-update-date) so I believe it currently only uses issue_date on the package for comparison. If EPEL packages aren't publishing issue_date (which I'm not seeing on my local EPEL 5 repo), then clone-by-date probably just does the repo diff to determine what needs cloning.

That's just a quick analysis though, I could be very wrong.

-Matt


**** SUBSCRIPTION TO CUSTOM CHANNEL
This is the snippet of code we use to subscribe our hosts to channels based on environment (our naming standard starts with pd|qa|dv|tt, etc...) which we execute via bootstraph.sh

CLIENTHOSTNAME=hostname | cut -f1 -d.

SETTING THE ACTIVATION_KEYS IS A 2-STEP PROCESS

FIRST - FIGURE OUT IF THE HOST IN QA/DV/PD

PARSE CLIENTHOSTNAME STARTING WITH THE 6th CHAR THEN READ THE NEXT 6

SERVER_ENV=${CLIENTHOSTNAME: 0:2}
case $SERVER_ENV in
dv)
ACTIVATION_KEYS_ENV="1-dv-"
;;
qa)
ACTIVATION_KEYS_ENV="1-qa-"
;;
pd)
ACTIVATION_KEYS_ENV="1-pd-"
;;
*)
echo "ERROR: Unknown Host Environment"
echo "I.e. hostname should being with [dv|qa|pd]"
exit 9
;;
esac


*** CLONE BY DATE
The clone by date was a bit challenging to figure out how to NOT have a ton of manual intervention. At this point, there is only 1 step that is manual. The following script will create a .config file for each channel and environment (RHEL-5/6 and qa-pd)... after it does that, a human needs to go in and remove the last ',' in the section which describes the channel definitions. At this point, it simply displays the commands to run to the terminal so you can run them manually.