Red Hat Satellite (Disconnected) 5.6 drops channel content
This has now happened to me twice and Red Hat GSS can't explain whys, so I'm posting here to see what others know.
Within the last few weeks, I'm not able to sync or get the latest content after importing in update isos for RHEL5 x32 and x86_64 channels. Since I operate a disconnected RH Satellite, its going to take forever to download everything and import it back in.
This happened earlier this year with RHEL6 x86_64 content too.
Red Has GSS has attributed this to either ISO file corruption and/or Intermittent sync issue. How do I know that this is happening or is there a way to prevent this from happening?
thanks
Responses
Hey Christopher, Interesting. I run two Satellite servers (both disconnected) one version 5.7 and the second 6.1 in testing. I haven't had any issues with the last batch of isos for 5.7 sync'ing (only running RHEL 6 x86_64), but the batch from 8 April for sat 6.1 (RHEL 7 Server) will not sync at all. The directory structure appears to be missing for that batch. All the checksums were good on the 9 isos. I've got a support case open with them, but haven't gotten anything resolved yet. I take it the checksums were good on the isos you downloaded?
Ryan Holt
[re-written/edited]
Chris, I imagine you've unpacked && ingested your iso files in some fashion as I mentioned to you in this thread), correct?
I see you've checked the checksums...
What happens when you unpack the rpms, then make the permissions proper and do a satellite-sync -m /path/to/the/unpacked/specific/channel/ --list-channels
Also, what happens when you run satellite-sync -m /path/to/the/unpacked/specific/channel/
I have an nfs server that only serves the unpacked files to my satellite server. I made sure the permissions to the unpacked rpms on the nfs server was proper. I mounted that nfs mount to the satellite server and then did satellite sync with the --list-channels option for sanity. We did the unpacking on the nfs server, and one directory each for each separate set of channels (segregate out base channels, incremental channels and architectures to separate directories). So, rhel6 64-bit gets it's own directory, same with rhel5 (and for you of course a separate one for 32 & 64 bit), and base channels and incremental channels are segregated. Always (always) do base channels before any incremental (sorry, you probably have already done this, but I write it because others will read this).
Verify in your web gui that the "channel-repodate-bunch" task shows "FINISHED" before ingesting a new set of channels!!! 1) Go to the web gui as an Red Hat Satellite Admin (special role), and 2) look at the top menu, 3) click on "Admin" 4) then on the left, click on "Task Schedules" then 5) look for "Channel-repo-data-bunch" (right side, top link, generally) and 6) make sure it has a status of "FINISHED" before ingesting another set of ISOs
Hi Chris
I'd avoid the recent incremental channel of [2015-10-10 spanning to 2016-05-11 rhel 6 64-bit], and just take the base channels.
But I see the consternation you're facing is with the 32-bit channels. Did you see that as of 2016-05-11 there's a new set of Base Channels for [RHEL 6 (i386) + EUS + AUS + RHN Tools + Supplementary (Base 2016-05-11)] ? (there are 19 DVDs in that set)
If you are using VMware for your satellite server, make sure to delete any snapshots (to the Red Hat Satellite in VMware) you have, particularly before ingesting them or could easily fill up your ESX storage pool that your Red Hat Satellite is parked on. (This would cause consternation for any server parked on that storage LUN if it fills up).
When you unpack the 19 DVDs, have a place that can accommodate 19 DVDs, and the DVDs unpacked (I see you are using /tmp/). If using NFS, (and if you have enough storage on nfs), you can unpack them on an nfs storage and present it as an NFS mount (I've done this in a pinch on a workstation with a huge local hard drive and then we activate the nfs service that only presents (via /etc/exports) the files to the satellite server and the nfs service is typically off with chkconfig off).
When you do the ingest of the Base Channels, you'll have to use "-c" with every channel you wish to import. You don't necessarily have to import all the channels, some are older channels like 6.0, 6.1 etc. I have at times ingested them all, but I have enough storage on my satellite server to take it all. (I mentioned this a bit in that guide I wrote for you in that other thread I cited earlier)
Chris, just in case this applies (I don't know if it does/does not for your case) ... RHEL 6.8 base channels were released 5/11/2016. make sure to download/install the base channels first before applying any incrementals. If you have applied incremental ISO channel dumps that followed after the milestone of 5/11/2016, it will certainly bugger your channels, as base channels ==must== be ingested first...
Chris, just an FYI, also see this discussion https://access.redhat.com/discussions/1597433, and also this discussion https://access.redhat.com/discussions/2327371
Chris,
If (and only if) you really wanted to take in all of any given RHEL Base channel (and you might not), this would make the script: (this is in reply to something you posted a bit ago, and you said you were scripting the ingest...
echo -e 'satellite-sync -m /path/to/unpacked/rpms' `satellite-sync -m /path/to/unpacked/rpms --list-channels | grep 'full support' | awk '{print "-c " $3}'` >/tmp/verify_this_script_and_change_to_700_perms_if_good
Also, in reply to 20 May 2016 1:28 PM - Regarding what you mentioned on:
--steps=[channels|rpms|packages|errate|kickstarts]
I've never used that option.
Did the ingest of the channels return "Completed"? Did you get "FINISHED" in the "channel-repodate-bunch" 1) Go to the web gui as an Red Hat Satellite Admin (special role), and 2) look at the top menu, 3) click on "Admin" 4) then on the left, click on "Task Schedules" then 5) look for "Channel-repo-data-bunch" (right side, top link, generally) and 6) make sure it has a status of "FINISHED". Then do a yum clean all against a client, and echo n | yum update or yum check-update to see if more packages are available.
What is the content of "./satellite_channel_sync_rhel5_x32.sh" script?
Also looks like that sync did not brought any new packages (see "Fetching any missing RPMs: rhn-tools-rhel-i386-server-5 (NONE MISSING)") and/or erratas (see "etrieving / parsing errata data: rhn-tools-rhel-i386-server-5 (NONE RELEVANT)"), so if your clients are up2date, there should not be any new update available for them.
If you suspect issue with Taskomatic daemon and its channel repodata bunch, check Admin -> Task Engine Status -> Channel Repodata or for more details go to Admin -> Task Schedules -> channel-repodata-bunch and check status column for newest record in the table (should be "FINISHED" and start time should be recent - by default that bunch gets started every minute).
That's odd... On my Satellite 5.6 and 5.7 servers, I have a link in that location named "Channel-repo-data-bunch" with a status of "FINISHED".
Chris, do you get this error at this Red Hat Solution --> https://access.redhat.com/solutions/319983 ? Please check to see if that solution is a fit.
This could be a contributing problem, maybe. Look in the tomcat logs within that area in the satellite web interface, there's a link for tomcat logs. I'd also recommend calling support with this at this point. Cite this discussion area in the ticket.
ADDED #1: ADDED #2:Chris, In the Satellite web Gui - go to "Admin" (top bar), then on the left, go to "Task Schedules", then:
++ Look at the (probably) top link named "channel-repodate-default" and is there anything in the "Frequency" column? ++ Click on "channel-repodate-default" link and you should see a page with:
Schedule Name: channel-repodata-default
Bunch: channel-repodata-bunch # this is a hyperlink
Frequency: Select a schedule:
<various "radio buttons" & drop down menus to schedule something>
Do you see anything like what's in that code block (on the web page in the satellite gui)? If you do, what do you get when you click on "channel-repodata-bunch within that specific page?
ADDED #3:Chris, Is there anything ... bad ... in /var/log/rhn/rhn_taskomatic_daemon.log ?
ADDED #4Chris, also see this https://www.redhat.com/archives/spacewalk-list/2015-February/msg00038.html what happens when you do a rhn_satellite restart ? what does the /var/log/tomcat6/catalina.out file give too?
We run the disconnected Satellite. We had an issue with the imports earlier this year. They definitely posted a corrupt incremental. I rebuilt the server from scratch, updated to the latest Satellite packages and then RH delivered a newer Base Channel content. They have also increased the frequency of the incremental updates sometimes releasing an incremental that only contains updates between the last incremental and the current one and and incremental that goes way back to the Current date. I believe they did that with the RHEL 6-x86_64 date 20160521. Do you have the base channel content dowloaded again? You could always unsubscribe all your systems from the base channel and delete the channels and then re-import.
Daniel, the latest base channels for RHEL 6 are 2016-05-11 so if that incremental you cited [(Incremental 2016-05-11:2016-05-21)] was taken before it's parent base channel, that would certainly cause consternation... in case that was the case.
I too have had to rebuild a satellite server when it went south, they're not terribly difficult to rebuild, it's the ingesting of the channels, and rejoining of the client systems that's no fun.
I suspect there may be a way to redo the channels without rebuilding the server if it comes to that... Chris check my last posts
Chris, have you ever used bootstrap, the bootstrap script to bring in a system? 1) make an activation key 2) assign groups, software channels, software packages to the activation key 3) go to the web gui, there's a simple method to generate it from the web gui. Once it's generated, you can go to http://yoursatellitehostname.something.xyz/pub/bootstrap/bootstrap.sh and then within that script, add your activation key (type it in the generated script), and also decide if you want to have the bootstrap script fully update the system upon registration or not. I can give you more details next week, but this week I'm at the RH Summit.
Chris, is there any chance you're having this issue OutOfMemoryError https://access.redhat.com/solutions/43122?
Is ipv6 blacklisted within the various /etc/modprobe.d/xyz.conf files? SIDE ISSUE: I found nfs mounts failed on certain systems where I went out of my way to blacklist IPV6, a while back and I mitigated that.
Also see https://www.redhat.com/archives/spacewalk-list/2012-July/msg00080.html
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
