Advise on Installing a disconnected Redhat Satellite?

Latest response

I'm going to be honest, I've been trying to upgrade our Redhat Satellite from v5.4 to v5.6 for about 6 months now and running into a number of stumbling blocks. I currently one run one RH Satellite server that runs in disconnected mode. I'm told that typically in a DoD environment, there are two RH Satellite servers, one that talks to the Internet, which gets the latest and greatest content and then that is moved to the disconnected RH Satellite and updates are pushed from there.

Last week, after downloading all of the Base and Child Channels isos (around 400 GB of data), I ran the wrong command after copying over files that were having issues and deleted the whole OS. So now I have to start over.

I'm wondering if anyone can advise on how to set this up smoothly, so I don't get further behind on patches for the servers that I manage.

thanks

Responses

Hi Chris,

We have numerous disconnected satellites, and none that face the public internet. We download the iso channels directly from Red Hat. Then we grab the iso incremental channels as they are published. It sounds like a lot, but we've been doing this for years with no issues. We have a "forensic bridge" approved (hard drive with a approved one-way transfer device) that we were able to get approved via our security folks. The forensic bridge allows transfer to networks not facing the public internet in a verified one-way transfer with no ability for write-back to the drive where the iso channels are located from the lower network.

We use rsync often to an nfs share, and on the nfs server unpack them. We do have one satellite server with a gigantic amount of storage due because the network it is on has minimal data stores (nfs) available and we unpack/rsync the channels and do the satellite-sync on the server.

I can give more details on the process of unpacking iso channels in a sane way as well if you wish,

Kind Regards

That is where I had the issue, I have all of our ISOs on a data store and we have an environment made up of the following: RHEL 5 x32, RHEL 5 86_64, RHEL 6 x32 and RHEL 6 86_64.

So I was copying down the ISOs from the data store to the server, there were issues the copy and I tried to clean up the directory and bam! Wrong command issues and game over.

I know of what rsync and nfs is, however I'm not sure how to implement this to populate the channels on the RH Satellite, so I am looking for more info on unpacking the iso channels. Anything to get this back up and running.

thanks

Eef... This is where filesystem (or even LUN-level) snapshots are great. Or, failing that, having a good backup and recovery system in place.

This was on a Linux guest and because the disks were set to Thick Provisioning, I couldn't do a snapshot, because the data store would run out of space.

I'll be posting some detailed instructions shortly

Hi Chris,

Please treat the below like a buffet. Some things may work, some things might need to be altered for your environment(s).

There are probably cleaner ways to do some of what I’ve posted below. I suspect this will garner warranted replies with “we do this and it works better for this reason” and I don’t claim the below is the cleanest approach. There are people that have been doing UNIX/Linux tasks longer than I have and this bit below works for us for years, but there’s always room for improvement.

I recommend reading the man page for any command you are not comfortable with. Please check out the Satellite 5.7 documentation (click on 5.7 when you get to this link) as well. I have not tested the below against satellite 6 because only in this month have they finally released ISO channel dumps for satellite 6.x.

We copy our iso files and (every time without fail) verify the md5sums (or sha256 sums) of the iso files every time we copy them to their intended destination.

So, I see you say above...
-- you had issues with the copy
-- and some invalid commands were issued and the files were gone.

Here's our method we've been using for years

  • Download the RHEL ISO channels this link has the base and incremental channels. We are using satellite 5.6 and 5.7 in our environment, and are going to have strictly 5.7 soon. That being said, we have noticed the base and incremental channel dumps have the same md5sum between satellite versions. So we've never had to worry if we had to download separate channel dumps between satellite versions such as 5.5, 5.6, 5.7. So far, the md5sums for base and incremental channels work fine and match identically.
  • I can not stress the importance of verifying the md5sums AFTER the copy to a new network. Some may consider this Overkill(TM), however, I'd rather have Overkill(TM) then ingesting a corrupted ISO channel dump into my satellite server. The process is worth doing correctly and even being meticulous over whether or not your ISO disks are valid (and that's why Red Hat provides md5sums/sha256 sums to verify anyway) [/end rant]

  • Download all of the base channel dump ISOs, yes, there are a lot of them. Make sure to take the most current date.
    -- EXAMPLE As I type this --right now--, the most current date for Base channel dumps for RHEL 6 64-bit is:
    " RHEL 6 (x86_64) + EUS + AUS + RHN Proxy/Tools + Supplementary (Base 2014-12-31)"
    -- As mentioned previously, verify the ISO channel md5sums. Here's one method after downloading all ISOs into one directory:
    Make a text file with the md5sums copied from Red Hat's URL above named "md5sums_rhel6basechannels20141231.txt"

cd /data/rheldisks/rhel6/basechannels/20141231/
ls | grep iso$ | perl -ne 'print;chomp;system("md5sum $_ >> md5sum_results_after_download_time_date_stamp.txt")'
echo "the above command will seek iso files, perform an md5sum against each one and put the results into a file listed above"
echo "after that completes you can run the below to check"
awk '{print $1}' md5sum_results_after_download_time_date_stamp.txt | perl -ne 'print;chomp;system("grep $_ md5sums_rhel6basechannels20141231.txt")'

  • You should get a match for every ISO file, the above will grep the results of your md5sum to the md5sum file you created from Red Hat Inc.
  • If you do not get a match, redownload the iso file.

  • Once all your ISO files run clean with the md5sum/sha256 sum verification, then proceed. If any fail, redownload before proceeding.

NOTE: if you are using Solaris for your nfs servers, use "digest -a md5 name_of_file.iso" instead of the 'md5sum' command.

Copy the ISO files you verified to a hard drive (preferably) or burn individually to DVDs or a Blu-Ray if available
- If using a hard drive, verify beyond any shadow of a doubt that you have an -=approved=- method with your security folks for a product called "forensic bridge". A "forensic bridge" is a device that attaches between a hard disk and the system you are using to copy the files to (such as a network that will never face the public Internet that is secured for compelling reasons).

  • Either using the -=approved=- forensic bridge method, or the DVDs you have burned, copy the files to your disconnected network(s).
    -- We copy to an NFS share and sometimes this is through a windoze system to a Samba-presented file system.
    -- Yes, some consider this overkill, however, re-verify the md5sums for the iso channel dump files after copying because sometimes they do fail and the time you do not check it - that will likely be the time one of the isos fails and then you could have corrupted channels which is Bad(TM), and we've been there many years ago and we verify without fail every iso.
    -- AFTER verifying the md5sums, begin unpacking the iso channel dumps.

LINUX:

echo "verify the directory /iso exists, else create it."
/bin/mkdir -p /iso/notmounted;chmod 755 /iso
mountpoint /iso
echo "it should return "/iso is not a mountpoint"
mount -o loop /path/to/the.iso /iso
  • We often have an nfs share such as:
    /data/rhel/
    -- Then under that we have
    /data/rhel/rhel5/rhel5basechannels
    /data/rhel/rhel6/rhel6basechannels
    /data/rhel/rhel7/rhel7basechannels
    and
    /data/rhel/rhel5/rhel5incrementals
    /data/rhel/rhel6/rhel6incrementals
    /data/rhel/rhel7/rhel7incrementals

We will make a directory under any of those with a time date stamp such as:
/data/rhel/rhel6/rhel6basechannels/20141231
- And we unpack to something like:
/data/rhel/rhel6/rhel6basechannels/20141231/unpacked
- the same goes with incremental channels.

So one by one, mount each disk for the base channels and unpack

mount -o loop /path/to/the/basechannel_01.iso /iso

Then unpack it with rsync

rsync -au --progress /iso/ /data/rhel/rhel6/rhel6basechannels/20141231/unpacked/

"-a" is archive.
"-u" is update
NOTE 1: Read the man pages for rsync, the slashes above are very important and one should understand the impact of them.
NOTE 2: There are various options for rsync, the above works fine for us.
NOTE 3: Always run rsync more than once to verify all files copy and the rsync runs clean. The second run of rsync should have no output if all files copied properly. Again, read the use of rsync in the man page

  • SOLARIS:
    use
lofiadm -a /path/to/the.iso
echo "output will be something like below"
/dev/lofi/1
mount -F hsfs /dev/lofi/1  /iso
use rsync as previously cited
echo when complete, clean up the lofi device
lofiadm -d /dev/lofi/1

NOTE: Verify output because some other admin may have a "lofi" device they never bothered to clean up (to be fair, it may not be another admin)

REPEAT PROCESS ABOVE FOR EVERY ISO.

SHORTCUTS (should only be used when you have gained confidence in your process.
We have over time created up to 8 devices mounted at once and rsync 8 isos at once.
-- HIGHLY RECOMMEND THIS ONLY BE DONE WHEN you feel confident to do so and use extreme caution!!!

Read caution above, you have been warned
  • Mounting eight iso files at once to do an rsync of eight disks
  • Note: there is a limitation of 8 devices unless you adjust some config file that I've got listed someplace, but cannot immediately find under Linux. Under Solaris using lofiadm, I can mount all isos at once.
cd /path/to/rhel6/basechannels/
for i in {1..8};do /bin/mkdir -p /iso$i;done
for i in {1..8};do mount -o loop rhn-export-rhel-x86_64-6-20141231.0${i}.iso /iso$i ;done

df -PhT # output should show first eight disks mounted to their respective iso mountpoints

USE CAUTION, If you are not confortable mounting multiple iso files at once, do not proceed, or if you have any doubt, deal with this one disk at a time you have been warned.

  • Use rsync to copy the files from the eight mounted isos above to the unpacked directory.
rsync -au --progress /iso{1..8}/ /path/to/the/directory/called/unpacked/ 

NOTE 1: The characters and slashes above are important. Use extreme caution with rsync, it is like playing "Simon Says".
NOTE 2: Directory structures explained previously
NOTE 3: Always run rsync more than once. The second or subsequent runs should have no output because the first run should have copied the files. Yes, you could use "copy -a", however, I prefer rsync because you can see what is going on.
NOTE 4: ALWAYS umount ALL /iso directories before proceeding with the next batch of iso files

  • Umount the iso files after verifying the files were staged properly. The previoius sentence is written with qualifiers.
  • After successful (see above) movement using rsync, umount the mounted iso files
umount /iso*
df -PhT

NOTE: Verify no iso files are mounted before proceeding. If any iso files are --still-- mounted, take the usual methods to make sure to unmount the iso files (make sure someone did not cd to one of the /iso* directories, and so forth)

  • After verifying NO /iso* mountpoints are mounted with iso files, then proceed. Do not proceed until this is completed
  • (see important note below) Repeat above process with the next batch.
  • IMPORTANT: You will note the naming convention of the iso files is "0x" for the first 9 disks, and 1x" for the next batch, and you'll have to work around that as needed.

  • When complete, you should have unpacked all iso channels, for [ RHEL 6 (x86_64) + EUS + AUS + RHN Proxy/Tools + Supplementary (Base 2014-12-31)] disk collection, there are 22 (count them) disks AS OF TODAY

  • do an 'ls' of the unpacked directory and you should see "DISK_XX_OF_YY" for 22 total disks. VERIFY BEYOND ANY SHADOW OF A DOUBT before proceeding. note the number of disks is for the previously listed dated base channel set

IMPORTANT NOTE THAT IF IGNORED COULD CAUSE SEVERE CONSTERNATION if ignored:
If you are unpacking rhel5 channels one day, then the next day are going to unpack either rhel6, MAKE SURE beyond any shadow of a doubt there ar NO mounted iso files. We had one instance where an admin accidentally had a rhel5 incremental channel iso file STILL mounted, and then wondered why the channels were really complaining with the "--list-channels" argument of "satellite-sync". The answer was because they did not umount the previous rhel5 incremental channel iso before proceeding the next day with the rhel6 incrementals. Always verify the /iso* directories are umounted without fail every time!!!!!!!!

After verifying all isos are copied over to the "unpacked/" directory, do a recursive chmod 755 against it:

/bin/chmod 755 /path/to/the/unpacked/
  • ssh to the satellite server
  • Become root
  • mount the nfs file system where you unpacked this bit above.
  • cd to the proper directory under nfs (remember permissions)
  • Verify you have enough space under /var/satellite or the directory for 5.6 or 5.7 satelite vesions to ingest the channels
  • IMPORTANT: if your satellites are under VMware (most of ours are) DELETE ANY SNAPSHOTS FIRST OR YOU WILL FILL UP THE STORAGE LOCATION THE SATELLITE SERVER IS PARKED ON. I've never done this [eye roll], this is just for other people [eye roll]
  • Also make sure under satellite 5.6 the directory /var/lib/pgsql is not clsoe to being filled up (that's the datebase directory). If needed, resolve the storage issue properly before proceeding so you do not fill up your database for a /var/lib/pgsql filessytem that is inadquate in terms of volume.
  • The same warning in the line above applies for Satellite version 5.7 but for the the new directory located at:
    /opt/rh/postgresql92/root/var/lib/pgsql/
    -- mitigate any insufficient storage before proceeding, like if the directory is at 94% usage...

  • Make srue --no other admins are doing a satellite-sync before proceeding!!--

  • Do the initial satellite-sync with "--list-channels"

  • ***Please read the man page for satellite-sync for the meaning of "p" and "." (dot) for channels.
satellite-sync -m /path/to/the/directory/unpacked/ --list-channels

NOTE: This is decision time. You will need to decide which channels you want to import. I've seen some customers have me ingest all, or some of the channels. The instructions below are under the assumption you may want all channels. You may not want all channels, and there may be compelling reasons for either scenario. That being said...

Syncing all channels, make a file with all the channels for ingesting base channels

NOTE: ingesting incrementals is different!!!
NOTE: ingesting incrementals is different!!!
NOTE: ingesting incrementals is different!!!
  • This will create a file with all channels where you can edit it using vi/vim to create your final sync command
satellite-sync -m /path/to/the/rhel6basechannels/unpacked/ --list-channels | grep 'full import ' | awk '{print $3}' > /tmp/rhel6basechannel_list.txt
cp /tmp/rhel6basechannel_list.txt /tmp/rhel6basechannel_list.txt.backup

enter vi/vim or [shudder] gedit, and put a "-c " (dash c with a space after it) for every channel.
- If you are in vi/vim, you can then go to the top of the file and use "shift J" (capital "J") to force all lines into one line. There are creative ways of using vi/vim to do this, and you can even use sed in the original satellite-sync command above before redriecting it to achieve the same goal of putting a "-c " before each channel name.

NOTE 1: VERIFY you have enough storage space to ingest all base channels
NOTE 2: VERIFY you have no current snapshots on a VMware satellite before ingesting base channels (you have been warned)

At the beginning of the file you created above, you will want to enter:

satellite-sync -m /path/to/the/rhel6basechannels/unpacked/ -c channel_name -c another_channel_name -c yet_another_channel_name -c you_get_the_idea

The resulting file should have a VERY long list of channel names, each channel should have a "-c" before it
- My example I just examined (I did a cat of the file), is one very long line, with no line breaks that when I view it in a terminal window, it runs about 35 lines long (again no line breaks!!!!)

VERIFY THE COMMAND using the man page for satellite-sync before proceeding for sanity.

do a chmod 700 against the file to give it execute permissions, you'll be running this as root.

THIS IS A VERY LONG PROCESS!
- Use the "script -a" command prior to executing this so later you can look at the output, some servers will log you out after some inactivity. So if you run this and go away and come back, and it complets, and the server logs you out, you have --no-- idea if it ran clean or not. use the "script -a" commmand so you can view output later.

- (see man page for the "script" command)
script -a /tmp/20150402_rhel6basechannel_sync.txt
  • Execute the script you made above AFTER you have verified it for sanity using the man page.
/tmp/rhel6basechannel_list.txt
  • The output will show "completed" when done

GIVE THIS A LOT OF TIME - check in advance, make sure the VMware admins are NOT going to move your server to another VMware cluster, or do maintenance, or that there's a planned power outage, and so forth.

  • Give your satellite a lot of time to ingest this into it's database. I've found that the process takes some time even AFTER the sync shows "Completed"
  • You can go to the satellite server graphical web gui and select "admin" from the top menu" then select "task schedules" from the left menu, then select (AT LEFT) "channel-repodate-bunch" and WAIT until you see "FINISHED" instead of "SKIPPED". "SKIPPED" really means the satellite server is busy, and do NOT reboot it during that time.

You may need to issue a 'yum clean all;yum repolist" to all clients after you get "FINISHED" in the graphical menu above. I recommend this.

  • If the channels do not behave, seem to be wonky, give the satellite a few more hours before performing 'diagnosis', I've found being patient with the satellite after an ingest is really useful.
  • After ingesting one set of channels, wait a long while before ingesting another set (like wait after doing base channels for a long time, and I'd wait an additional 24 hours before ingeesting the incremental channels). The same goes with basically ingesting one set of incrementals, then wait a sensible amount of time until the database and "Bunch channel-repodate-bunch" is completed.

I recommend a tiered approach to patching, agree with your customer, your other teams on this.
- In our environment, we first patch to a portion of our test network. Then we patch the remainder of the test network. We then patch a portion of our production network, and then do the remainder of the production network. We thought long and hard over our patch plan. Always coordinate patches with database folks and it works better if you can do it with them together (we do this with our database team, works great).

Hope this helps, some of this is of course in the Satellite documentation, perhaps not in the detail I've given.

My examples may not fit all customers/networks. I'm sure there are folks that may have a cleaner approach to some of what I've posted above.

Chris,

All the above being said, I have one satellite server on a specific network that didn't (at the time) have sufficient space to host the base and incremental channels on nfs. So when I built the satellite on that network, I used a physical server with oodles of drive slots, created a sufficient raid5 for Overkill(TM) on all partitions to include a separate raid5 for all base/incremental channels with more than sufficient space to handle all base and incremental channels for rhel 5/6/7. So on that satellite, I do all unpacking operations listed previously on that system without any nfs.

Let me know if I can clarify anything or help further.

Been reading thru this thread.

In addition to creating a new RH Satellite Server, I would also have to setup another server with NFS Shares to host the ISO Base/Channel Content. This would probably work better because I don't believe I can run MD5 checks against the ISO, that are stored on a data store.

So this NFS Server could just have directories for the various channels and then copy from the NFS Server to the RH Satellite Server, correct?

Hi Chris, [post made to reply above, 7 April 2015]

The short answer (such as in the majority of cases (see below)) is yes. We have NFS on six of our eight satellites and those satellites can "see"/mount that NFS storage share. NOTE: I recommend the unpack/rsync bit be done on the nfs server hosting the storage if possible because we have seen strange/bizarre things occur when attempting to do this (that is too lengthy to describe now) when attempting to do the unpack operation on the satellite server to nfs storage (see original post I made).

Just to clarify why we used nfs... We selected NFS on six of our satellite servers because we had plenty of NFS storage under those specific networks. You could certainly have the storage under a properly provisioned satellite server (we do this on two physical servers where those networks do not have a lot of NFS storage).

We have eight disconnected satellite servers. Six of our satellites are VMware, two are physical servers. The two physical systems were made physical because the VMware storage on those two customer's networks did not have sufficient space under NFS and we stood up the satellite server on those two physical servers with PLENTY of storage to host the base/incremental servers.

See my instructions above for your last question posted 7 April 2015. For the six satellites we have that use NFS, we (as shown in the example above) copy, unpack, mount to the satellite server and then do the satellite sync (see detailed instructions previously typed in my previous post).

Chris, I selected an NFS share in some cases because I didn't have a lot of storage on many of my satellite servers (we have eight disconnected satellite servers). So NFS on most of them worked out as a place to put the collection of isos and to do the unpacking I described earlier.

Chris, I HIGHLY recommend you go straight to Satellite 5.7, and skip 5.6. I am in the process of migrating to Satellite 5.7 and have one in production that is working great. Even if you have to create a NEW satellite and migrate things from one satellite to another...

If you need to copy over a lot of configuration channels, I have a one-line script (previously posted in this discussion forum) that will copy off ALL configuration channels to a place where they can be ingested later on a new satellite. (see instructions in that link at that Red Hat discussion).

Kind Regards...

Going thru the RH Satellite 5.6 documentation.

I'm writing up my install plan and this question comes to mine.

For setting up space for /var/satellite, it needs a minimum of 40 GB per software channel. Does that mean 40 GB for both Base and Incremental? Or 40 GB each for Base and 40 GB for Incremental for a total of 80?

Ok, I'm still trying to forumlate a plan for this.

Here is a question and this is based on not knowing alot on RH Satellite, and once it is up and running.

Why even bother with an NFS Server and move over isos from this server to the RH Satellite Server? Why not just create directories on the RH Satellite server, run the MD5 check against the ISOs and then if they check out ok, then run your base/incremental upgrades? I'm thinking the rsync gives you a clean and safe way of knowing that your unpacked isos are going to copy over without an issue, correct?

Since this is a Disconnected Satellite, how is patching done in the long run? Do I wait for RH to list more updated ISOs, which seem to get published throughout the year? Or can I grab .rpms as errata is sent out via the Red Hat Network Alert? That way I can patch as new packages come out to keep our servers up to date?

thanks

I check with Red Hat routinely (like 2 or 3 times a week) and grab new ISO files as needed.

The method is what I described previously. I typically avoid grabbing individual rpm files because of a variety of reasons, to include it defeats the use of getting the incremental ISO dumps, not to mention being wrapped around the axle of every rpm request. My numerous customers for all eight of my satellites have no problem waiting the time it takes between incremental updates.

I do the rsync as listed previously, and then mount the drive in the cases using NFS as previously described. As I mentioned previously, there are two of our servers that are physical and we do not use NFS. As previously mentioned in the very long post above.

Is there a really compelling reason you -can not- go to satellite 5.7?? I'd recommend you skip satellite 5.6 and go straight to satellite 5.7. Really

partitioning? I take all channels for base channels for 5 and 6, I limit nothing. Yes, I'll probably get feedback on this, but I have enough storage to where this is not an issue, and I dont' care to reload some channel from the base channels that I missed (see previous description on base channel ingest previously described above).

I have both rhel 5 and 6 base and incremental channels on one server with 318G under /var/satellite with 194G used, and 107G remaining as I type this.(NOTE: this is where the rpms are stored after doing the satellite-sync to do the ingest)

My /var/lib/pgsql for satellite 5.6 (recommend using satellite 5.7, not 5.6) is 45G, with 17G used, with 25G remaining.

My two satellite 5.7 servers have even more space because I wanted more space under the database (see previous notes for that partition location listed previously in my comments above).

Hope this helps, let me know if I can clarify anything else,

kind Regards

Right, two of my satellite servers have sufficient space (I think I mentioned this a bit previously) and the only reason I used NFS on the other six servers was because I had oodles of space on NFS and not a lot of space on my VMware servers to house both base and incremental channel iso dumps. (I think I spoke on this previously... maybe)

Kind Regards.

Just so I understand, the unpacked isos are done on the nfs server and not on the RHSatellite Server, correct?

From the RHSatellite Server, we mount the nfs share and from here we run the satellite-sync command, correct?

I'm getting ready to re-build the RHSatellite Server and just want to make sure.

thanks

Hi Chris,
- To answer your question, yes. I have eight disconnected satellite servers. For those satellite servers whereI have nfs-presented satellite ISOs, I unpack on the same NFS server where the ISOs are located.
-- one thing I forgot in my original post I made where on the NFS server I do a recursive permission chmod of 755 against the unpacked directory, where the rsync was sent to, so in that directory called "unpacked" that is where I do a chmod of 755 recursively so the satellite can see the channel when I go to import it.
- For the satellite servers where I do not use NFS to present the ISO channels, I do the unpack/rsync on the satellite server itself (where I have a very large raid5 presented)..

Kind Regards,
R. Hinton

Latest RHEL 6.7 Base channels released very recently at this link https://rhn.redhat.com/rhn/software/channel/downloads/Download.do?cid=24541

Chris, hope this helped, resurrecting this based on your discussion https://access.redhat.com/discussions/2323171... maybe this thread might help again.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.