Advise on Installing a disconnected Redhat Satellite?
I'm going to be honest, I've been trying to upgrade our Redhat Satellite from v5.4 to v5.6 for about 6 months now and running into a number of stumbling blocks. I currently one run one RH Satellite server that runs in disconnected mode. I'm told that typically in a DoD environment, there are two RH Satellite servers, one that talks to the Internet, which gets the latest and greatest content and then that is moved to the disconnected RH Satellite and updates are pushed from there.
Last week, after downloading all of the Base and Child Channels isos (around 400 GB of data), I ran the wrong command after copying over files that were having issues and deleted the whole OS. So now I have to start over.
I'm wondering if anyone can advise on how to set this up smoothly, so I don't get further behind on patches for the servers that I manage.
thanks
Responses
Hi Chris,
We have numerous disconnected satellites, and none that face the public internet. We download the iso channels directly from Red Hat. Then we grab the iso incremental channels as they are published. It sounds like a lot, but we've been doing this for years with no issues. We have a "forensic bridge" approved (hard drive with a approved one-way transfer device) that we were able to get approved via our security folks. The forensic bridge allows transfer to networks not facing the public internet in a verified one-way transfer with no ability for write-back to the drive where the iso channels are located from the lower network.
We use rsync often to an nfs share, and on the nfs server unpack them. We do have one satellite server with a gigantic amount of storage due because the network it is on has minimal data stores (nfs) available and we unpack/rsync the channels and do the satellite-sync on the server.
I can give more details on the process of unpacking iso channels in a sane way as well if you wish,
Kind Regards
Hi Chris,
Please treat the below like a buffet. Some things may work, some things might need to be altered for your environment(s).
There are probably cleaner ways to do some of what I’ve posted below. I suspect this will garner warranted replies with “we do this and it works better for this reason” and I don’t claim the below is the cleanest approach. There are people that have been doing UNIX/Linux tasks longer than I have and this bit below works for us for years, but there’s always room for improvement.
I recommend reading the man page for any command you are not comfortable with. Please check out the Satellite 5.7 documentation (click on 5.7 when you get to this link) as well. I have not tested the below against satellite 6 because only in this month have they finally released ISO channel dumps for satellite 6.x.
We copy our iso files and (every time without fail) verify the md5sums (or sha256 sums) of the iso files every time we copy them to their intended destination.
So, I see you say above...
-- you had issues with the copy
-- and some invalid commands were issued and the files were gone.
Here's our method we've been using for years
- Download the RHEL ISO channels this link has the base and incremental channels. We are using satellite 5.6 and 5.7 in our environment, and are going to have strictly 5.7 soon. That being said, we have noticed the base and incremental channel dumps have the same md5sum between satellite versions. So we've never had to worry if we had to download separate channel dumps between satellite versions such as 5.5, 5.6, 5.7. So far, the md5sums for base and incremental channels work fine and match identically.
-
I can not stress the importance of verifying the md5sums AFTER the copy to a new network. Some may consider this Overkill(TM), however, I'd rather have Overkill(TM) then ingesting a corrupted ISO channel dump into my satellite server. The process is worth doing correctly and even being meticulous over whether or not your ISO disks are valid (and that's why Red Hat provides md5sums/sha256 sums to verify anyway) [/end rant]
-
Download all of the base channel dump ISOs, yes, there are a lot of them. Make sure to take the most current date.
-- EXAMPLE As I type this --right now--, the most current date for Base channel dumps for RHEL 6 64-bit is:
" RHEL 6 (x86_64) + EUS + AUS + RHN Proxy/Tools + Supplementary (Base 2014-12-31)"
-- As mentioned previously, verify the ISO channel md5sums. Here's one method after downloading all ISOs into one directory:
Make a text file with the md5sums copied from Red Hat's URL above named "md5sums_rhel6basechannels20141231.txt"
cd /data/rheldisks/rhel6/basechannels/20141231/
ls | grep iso$ | perl -ne 'print;chomp;system("md5sum $_ >> md5sum_results_after_download_time_date_stamp.txt")'
echo "the above command will seek iso files, perform an md5sum against each one and put the results into a file listed above"
echo "after that completes you can run the below to check"
awk '{print $1}' md5sum_results_after_download_time_date_stamp.txt | perl -ne 'print;chomp;system("grep $_ md5sums_rhel6basechannels20141231.txt")'
- You should get a match for every ISO file, the above will grep the results of your md5sum to the md5sum file you created from Red Hat Inc.
-
If you do not get a match, redownload the iso file.
-
Once all your ISO files run clean with the md5sum/sha256 sum verification, then proceed. If any fail, redownload before proceeding.
NOTE: if you are using Solaris for your nfs servers, use "digest -a md5 name_of_file.iso" instead of the 'md5sum' command.
Copy the ISO files you verified to a hard drive (preferably) or burn individually to DVDs or a Blu-Ray if available
- If using a hard drive, verify beyond any shadow of a doubt that you have an -=approved=- method with your security folks for a product called "forensic bridge". A "forensic bridge" is a device that attaches between a hard disk and the system you are using to copy the files to (such as a network that will never face the public Internet that is secured for compelling reasons).
- Either using the -=approved=- forensic bridge method, or the DVDs you have burned, copy the files to your disconnected network(s).
-- We copy to an NFS share and sometimes this is through a windoze system to a Samba-presented file system.
-- Yes, some consider this overkill, however, re-verify the md5sums for the iso channel dump files after copying because sometimes they do fail and the time you do not check it - that will likely be the time one of the isos fails and then you could have corrupted channels which is Bad(TM), and we've been there many years ago and we verify without fail every iso.
-- AFTER verifying the md5sums, begin unpacking the iso channel dumps.
LINUX:
echo "verify the directory /iso exists, else create it."
/bin/mkdir -p /iso/notmounted;chmod 755 /iso
mountpoint /iso
echo "it should return "/iso is not a mountpoint"
mount -o loop /path/to/the.iso /iso
- We often have an nfs share such as:
/data/rhel/
-- Then under that we have
/data/rhel/rhel5/rhel5basechannels
/data/rhel/rhel6/rhel6basechannels
/data/rhel/rhel7/rhel7basechannels
and
/data/rhel/rhel5/rhel5incrementals
/data/rhel/rhel6/rhel6incrementals
/data/rhel/rhel7/rhel7incrementals
We will make a directory under any of those with a time date stamp such as:
/data/rhel/rhel6/rhel6basechannels/20141231
- And we unpack to something like:
/data/rhel/rhel6/rhel6basechannels/20141231/unpacked
- the same goes with incremental channels.
So one by one, mount each disk for the base channels and unpack
mount -o loop /path/to/the/basechannel_01.iso /iso
Then unpack it with rsync
rsync -au --progress /iso/ /data/rhel/rhel6/rhel6basechannels/20141231/unpacked/
"-a" is archive.
"-u" is update
NOTE 1: Read the man pages for rsync, the slashes above are very important and one should understand the impact of them.
NOTE 2: There are various options for rsync, the above works fine for us.
NOTE 3: Always run rsync more than once to verify all files copy and the rsync runs clean. The second run of rsync should have no output if all files copied properly. Again, read the use of rsync in the man page
-
SOLARIS:
use
lofiadm -a /path/to/the.iso
echo "output will be something like below"
/dev/lofi/1
mount -F hsfs /dev/lofi/1 /iso
use rsync as previously cited
echo when complete, clean up the lofi device
lofiadm -d /dev/lofi/1
NOTE: Verify output because some other admin may have a "lofi" device they never bothered to clean up (to be fair, it may not be another admin)
REPEAT PROCESS ABOVE FOR EVERY ISO.
SHORTCUTS (should only be used when you have gained confidence in your process.
We have over time created up to 8 devices mounted at once and rsync 8 isos at once.
-- HIGHLY RECOMMEND THIS ONLY BE DONE WHEN you feel confident to do so and use extreme caution!!!
Read caution above, you have been warned
- Mounting eight iso files at once to do an rsync of eight disks
- Note: there is a limitation of 8 devices unless you adjust some config file that I've got listed someplace, but cannot immediately find under Linux. Under Solaris using lofiadm, I can mount all isos at once.
cd /path/to/rhel6/basechannels/
for i in {1..8};do /bin/mkdir -p /iso$i;done
for i in {1..8};do mount -o loop rhn-export-rhel-x86_64-6-20141231.0${i}.iso /iso$i ;done
df -PhT # output should show first eight disks mounted to their respective iso mountpoints
USE CAUTION, If you are not confortable mounting multiple iso files at once, do not proceed, or if you have any doubt, deal with this one disk at a time you have been warned.
- Use rsync to copy the files from the eight mounted isos above to the unpacked directory.
rsync -au --progress /iso{1..8}/ /path/to/the/directory/called/unpacked/
NOTE 1: The characters and slashes above are important. Use extreme caution with rsync, it is like playing "Simon Says".
NOTE 2: Directory structures explained previously
NOTE 3: Always run rsync more than once. The second or subsequent runs should have no output because the first run should have copied the files. Yes, you could use "copy -a", however, I prefer rsync because you can see what is going on.
NOTE 4: ALWAYS umount ALL /iso directories before proceeding with the next batch of iso files
- Umount the iso files after verifying the files were staged properly. The previoius sentence is written with qualifiers.
- After successful (see above) movement using rsync, umount the mounted iso files
umount /iso*
df -PhT
NOTE: Verify no iso files are mounted before proceeding. If any iso files are --still-- mounted, take the usual methods to make sure to unmount the iso files (make sure someone did not cd to one of the /iso* directories, and so forth)
- After verifying NO /iso* mountpoints are mounted with iso files, then proceed. Do not proceed until this is completed
- (see important note below) Repeat above process with the next batch.
-
IMPORTANT: You will note the naming convention of the iso files is "0x" for the first 9 disks, and 1x" for the next batch, and you'll have to work around that as needed.
-
When complete, you should have unpacked all iso channels, for [ RHEL 6 (x86_64) + EUS + AUS + RHN Proxy/Tools + Supplementary (Base 2014-12-31)] disk collection, there are 22 (count them) disks AS OF TODAY
-
do an 'ls' of the unpacked directory and you should see "DISK_XX_OF_YY" for 22 total disks. VERIFY BEYOND ANY SHADOW OF A DOUBT before proceeding. note the number of disks is for the previously listed dated base channel set
IMPORTANT NOTE THAT IF IGNORED COULD CAUSE SEVERE CONSTERNATION if ignored:
If you are unpacking rhel5 channels one day, then the next day are going to unpack either rhel6, MAKE SURE beyond any shadow of a doubt there ar NO mounted iso files. We had one instance where an admin accidentally had a rhel5 incremental channel iso file STILL mounted, and then wondered why the channels were really complaining with the "--list-channels" argument of "satellite-sync". The answer was because they did not umount the previous rhel5 incremental channel iso before proceeding the next day with the rhel6 incrementals. Always verify the /iso* directories are umounted without fail every time!!!!!!!!
After verifying all isos are copied over to the "unpacked/" directory, do a recursive chmod 755 against it:
/bin/chmod 755 /path/to/the/unpacked/
- ssh to the satellite server
- Become root
- mount the nfs file system where you unpacked this bit above.
- cd to the proper directory under nfs (remember permissions)
- Verify you have enough space under /var/satellite or the directory for 5.6 or 5.7 satelite vesions to ingest the channels
- IMPORTANT: if your satellites are under VMware (most of ours are) DELETE ANY SNAPSHOTS FIRST OR YOU WILL FILL UP THE STORAGE LOCATION THE SATELLITE SERVER IS PARKED ON. I've never done this [eye roll], this is just for other people [eye roll]
- Also make sure under satellite 5.6 the directory /var/lib/pgsql is not clsoe to being filled up (that's the datebase directory). If needed, resolve the storage issue properly before proceeding so you do not fill up your database for a /var/lib/pgsql filessytem that is inadquate in terms of volume.
-
The same warning in the line above applies for Satellite version 5.7 but for the the new directory located at:
/opt/rh/postgresql92/root/var/lib/pgsql/
-- mitigate any insufficient storage before proceeding, like if the directory is at 94% usage... -
Make srue --no other admins are doing a satellite-sync before proceeding!!--
-
Do the initial satellite-sync with "--list-channels"
- ***Please read the man page for satellite-sync for the meaning of "p" and "." (dot) for channels.
satellite-sync -m /path/to/the/directory/unpacked/ --list-channels
NOTE: This is decision time. You will need to decide which channels you want to import. I've seen some customers have me ingest all, or some of the channels. The instructions below are under the assumption you may want all channels. You may not want all channels, and there may be compelling reasons for either scenario. That being said...
Syncing all channels, make a file with all the channels for ingesting base channels
NOTE: ingesting incrementals is different!!!
NOTE: ingesting incrementals is different!!!
NOTE: ingesting incrementals is different!!!
- This will create a file with all channels where you can edit it using vi/vim to create your final sync command
satellite-sync -m /path/to/the/rhel6basechannels/unpacked/ --list-channels | grep 'full import ' | awk '{print $3}' > /tmp/rhel6basechannel_list.txt
cp /tmp/rhel6basechannel_list.txt /tmp/rhel6basechannel_list.txt.backup
enter vi/vim or [shudder] gedit, and put a "-c " (dash c with a space after it) for every channel.
- If you are in vi/vim, you can then go to the top of the file and use "shift J" (capital "J") to force all lines into one line. There are creative ways of using vi/vim to do this, and you can even use sed in the original satellite-sync command above before redriecting it to achieve the same goal of putting a "-c " before each channel name.
NOTE 1: VERIFY you have enough storage space to ingest all base channels
NOTE 2: VERIFY you have no current snapshots on a VMware satellite before ingesting base channels (you have been warned)
At the beginning of the file you created above, you will want to enter:
satellite-sync -m /path/to/the/rhel6basechannels/unpacked/ -c channel_name -c another_channel_name -c yet_another_channel_name -c you_get_the_idea
The resulting file should have a VERY long list of channel names, each channel should have a "-c" before it
- My example I just examined (I did a cat of the file), is one very long line, with no line breaks that when I view it in a terminal window, it runs about 35 lines long (again no line breaks!!!!)
VERIFY THE COMMAND using the man page for satellite-sync before proceeding for sanity.
do a chmod 700 against the file to give it execute permissions, you'll be running this as root.
THIS IS A VERY LONG PROCESS!
- Use the "script -a" command prior to executing this so later you can look at the output, some servers will log you out after some inactivity. So if you run this and go away and come back, and it complets, and the server logs you out, you have --no-- idea if it ran clean or not. use the "script -a" commmand so you can view output later.
- (see man page for the "script" command)
script -a /tmp/20150402_rhel6basechannel_sync.txt
- Execute the script you made above AFTER you have verified it for sanity using the man page.
/tmp/rhel6basechannel_list.txt
- The output will show "completed" when done
GIVE THIS A LOT OF TIME - check in advance, make sure the VMware admins are NOT going to move your server to another VMware cluster, or do maintenance, or that there's a planned power outage, and so forth.
- Give your satellite a lot of time to ingest this into it's database. I've found that the process takes some time even AFTER the sync shows "Completed"
- You can go to the satellite server graphical web gui and select "admin" from the top menu" then select "task schedules" from the left menu, then select (AT LEFT) "channel-repodate-bunch" and WAIT until you see "FINISHED" instead of "SKIPPED". "SKIPPED" really means the satellite server is busy, and do NOT reboot it during that time.
You may need to issue a 'yum clean all;yum repolist" to all clients after you get "FINISHED" in the graphical menu above. I recommend this.
- If the channels do not behave, seem to be wonky, give the satellite a few more hours before performing 'diagnosis', I've found being patient with the satellite after an ingest is really useful.
- After ingesting one set of channels, wait a long while before ingesting another set (like wait after doing base channels for a long time, and I'd wait an additional 24 hours before ingeesting the incremental channels). The same goes with basically ingesting one set of incrementals, then wait a sensible amount of time until the database and "Bunch channel-repodate-bunch" is completed.
I recommend a tiered approach to patching, agree with your customer, your other teams on this.
- In our environment, we first patch to a portion of our test network. Then we patch the remainder of the test network. We then patch a portion of our production network, and then do the remainder of the production network. We thought long and hard over our patch plan. Always coordinate patches with database folks and it works better if you can do it with them together (we do this with our database team, works great).
Hope this helps, some of this is of course in the Satellite documentation, perhaps not in the detail I've given.
My examples may not fit all customers/networks. I'm sure there are folks that may have a cleaner approach to some of what I've posted above.
Chris,
All the above being said, I have one satellite server on a specific network that didn't (at the time) have sufficient space to host the base and incremental channels on nfs. So when I built the satellite on that network, I used a physical server with oodles of drive slots, created a sufficient raid5 for Overkill(TM) on all partitions to include a separate raid5 for all base/incremental channels with more than sufficient space to handle all base and incremental channels for rhel 5/6/7. So on that satellite, I do all unpacking operations listed previously on that system without any nfs.
Let me know if I can clarify anything or help further.
Hi Chris, [post made to reply above, 7 April 2015]
The short answer (such as in the majority of cases (see below)) is yes. We have NFS on six of our eight satellites and those satellites can "see"/mount that NFS storage share. NOTE: I recommend the unpack/rsync bit be done on the nfs server hosting the storage if possible because we have seen strange/bizarre things occur when attempting to do this (that is too lengthy to describe now) when attempting to do the unpack operation on the satellite server to nfs storage (see original post I made).
Just to clarify why we used nfs... We selected NFS on six of our satellite servers because we had plenty of NFS storage under those specific networks. You could certainly have the storage under a properly provisioned satellite server (we do this on two physical servers where those networks do not have a lot of NFS storage).
We have eight disconnected satellite servers. Six of our satellites are VMware, two are physical servers. The two physical systems were made physical because the VMware storage on those two customer's networks did not have sufficient space under NFS and we stood up the satellite server on those two physical servers with PLENTY of storage to host the base/incremental servers.
See my instructions above for your last question posted 7 April 2015. For the six satellites we have that use NFS, we (as shown in the example above) copy, unpack, mount to the satellite server and then do the satellite sync (see detailed instructions previously typed in my previous post).
Chris, I selected an NFS share in some cases because I didn't have a lot of storage on many of my satellite servers (we have eight disconnected satellite servers). So NFS on most of them worked out as a place to put the collection of isos and to do the unpacking I described earlier.
Chris, I HIGHLY recommend you go straight to Satellite 5.7, and skip 5.6. I am in the process of migrating to Satellite 5.7 and have one in production that is working great. Even if you have to create a NEW satellite and migrate things from one satellite to another...
If you need to copy over a lot of configuration channels, I have a one-line script (previously posted in this discussion forum) that will copy off ALL configuration channels to a place where they can be ingested later on a new satellite. (see instructions in that link at that Red Hat discussion).
Kind Regards...
I check with Red Hat routinely (like 2 or 3 times a week) and grab new ISO files as needed.
The method is what I described previously. I typically avoid grabbing individual rpm files because of a variety of reasons, to include it defeats the use of getting the incremental ISO dumps, not to mention being wrapped around the axle of every rpm request. My numerous customers for all eight of my satellites have no problem waiting the time it takes between incremental updates.
I do the rsync as listed previously, and then mount the drive in the cases using NFS as previously described. As I mentioned previously, there are two of our servers that are physical and we do not use NFS. As previously mentioned in the very long post above.
Is there a really compelling reason you -can not- go to satellite 5.7?? I'd recommend you skip satellite 5.6 and go straight to satellite 5.7. Really
partitioning? I take all channels for base channels for 5 and 6, I limit nothing. Yes, I'll probably get feedback on this, but I have enough storage to where this is not an issue, and I dont' care to reload some channel from the base channels that I missed (see previous description on base channel ingest previously described above).
I have both rhel 5 and 6 base and incremental channels on one server with 318G under /var/satellite with 194G used, and 107G remaining as I type this.(NOTE: this is where the rpms are stored after doing the satellite-sync to do the ingest)
My /var/lib/pgsql for satellite 5.6 (recommend using satellite 5.7, not 5.6) is 45G, with 17G used, with 25G remaining.
My two satellite 5.7 servers have even more space because I wanted more space under the database (see previous notes for that partition location listed previously in my comments above).
Hope this helps, let me know if I can clarify anything else,
kind Regards
Right, two of my satellite servers have sufficient space (I think I mentioned this a bit previously) and the only reason I used NFS on the other six servers was because I had oodles of space on NFS and not a lot of space on my VMware servers to house both base and incremental channel iso dumps. (I think I spoke on this previously... maybe)
Kind Regards.
Hi Chris,
- To answer your question, yes. I have eight disconnected satellite servers. For those satellite servers whereI have nfs-presented satellite ISOs, I unpack on the same NFS server where the ISOs are located.
-- one thing I forgot in my original post I made where on the NFS server I do a recursive permission chmod of 755 against the unpacked directory, where the rsync was sent to, so in that directory called "unpacked" that is where I do a chmod of 755 recursively so the satellite can see the channel when I go to import it.
- For the satellite servers where I do not use NFS to present the ISO channels, I do the unpack/rsync on the satellite server itself (where I have a very large raid5 presented)..
Kind Regards,
R. Hinton
Latest RHEL 6.7 Base channels released very recently at this link https://rhn.redhat.com/rhn/software/channel/downloads/Download.do?cid=24541
Chris, hope this helped, resurrecting this based on your discussion https://access.redhat.com/discussions/2323171... maybe this thread might help again.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
