Does Red Hat bother to monitor its CDN for erratum delivery?
Rewording post:
We have repeatedly been told that Red Hat monitors its CDN for erratum delivery. In practice, this does not appear to be the case. The question is why does Red Hat not monitor its CDN and instead makes customers jump through all kinds of hoops attempting to get Red Hat to actually look at their CDN and see the obvious problem with old metadata or missing packages.
The other frustrating aspect is to see Red Hat erratum delivered by CentOS and even Oracle before we can get them via the RHN CDN. It is not a matter of slowness, but the erratum are not available!
You can see these on the RHN hosted console when looking at how auto-applied-errata failed to be applied. If I worked at Red Hat within RHN, I would immediately write this query and create a dashboard.
"How many systems on the CDN have auto-apply-errata checked and failed to install a given errata?" Seeing a very high error rate would raise some sort of internal red flags, one would think.
original rant text:
Just curious why Red Hat does not actively monitor that its CDN is delivering erratum. It is frustrating to consistently see Oracle and now Centos deliver errata faster than Red Hat RHEL RHN does.
Responses
When you mention "deliver errata faster" - are you referring to download speeds for packages?
If you have some definitive numbers indicating the download speeds (including date/time and the URL) I am fairly certain Red Hat would be interested in that data. I imagine that Red Hat is like the rest of us and has a service level agreement with their providers and if that is level is not being met, they should know.
With Oracle/CentOS, are you using the fastestmirror plugin?
I would look into using:
yum-fastestmirror RHEL 5
yum-plugin-fastestmirror RHEL 6 Optional Channel
NOTE: We use a Satellite in my environment so I don't actually use the fastestmirror plugin, but I believe it should help.
I am interested in what you are experiencing also. Are you not getting the notification about the Errata or is it that the packages are not available when you expect them to be?
I looked for openjdk and found https://access.redhat.com/errata/RHSA-2014:0907. If this is what you are referencing then we can use this as an example of how things could be better.
Daryl, consider adding your request to this discussion where Red Hat is asking for "what would you change".
So on July 17 I received email announcing java-1.7.0-oracle updates bringing it up to version 7 update 65. Now it is July 28 and when I check on an otherwise current RHEL7 Workstation using the CDN it offers update 55 still as the most recent. I have seen this sort of thing happen multiple times since RHEL7 was released and I'm about to the point where I just do not trust the CDN. I reported this CDN problem to Red Hat on July 18. If the CDN can't deliver critical security errata it is actually harming me by leaving machines I entrust to it vulnerable.
It is not an Oracle Java channel problem. It has happened with even the base workstation channel on RHEL7 too. Sometimes the metadata isn't rebuilding in the CDN properly and it has happened to at least several different channels.
I don't think a RHEL6 errata can shed any light on the particular problem I've seen as that has been RHEL7 specific, but not channel specific.
rhel-7-workstation-rpms and rhel-7-workstation-fastrack-rpms are getting metadata dated Aug 7 today - so again missing available updates from the 12th. I did not check other channels so these may not be the only ones not updating.
Another day another security update (openssl) is released but not available via the CDN. I'm really frustrated with this continuing to happen.
Every errata released since the Aug 7 since that was the last time the metadata for the repos was updated. I'm not where I can check the state of this today but as of yesterday that included at least RHBA-2014:1045, RHSA-2014:1042, RHEA-2014:1046, RHBA-2014:1049, RHSA-2014:1052, RHBA-2014:1058, RHBA-2014:1057, and RHBA-2014:1055.
Bryan: With this morning's errata (Aug. 18) both rhel-7-workstation-optional-rpms and rhel-7-workstation-rpms have updated metadata and all the old missing rpms appear to available from those channels although I only spot checked them.
The rhel-7-workstation-fastrack-rpms channel is still stuck on Aug. 7 metadata and so is not providing RHBA-2014:1045-1 which was released on Aug. 12.
same problem here. glibc update is not available through the CDN. but is available in CentOS the same day it was issued.
for the moment, is faster to get updates from CentOS.
"for the moment, is faster to get updates from CentOS."
Unfortunately this is getting true. In addition to RHEL 5, 6 and 7 systems, I am running CentOS and Scientific Linux. Yesterday's bash update from RH via 'yum' came through many hours after it became available from CentOS and SL.
I am aware Red Hat has been working on the issue but the delay in critical updates like yesterday's is becoming a serious issue.
Hello,
Thank you for continuing to let us know about this issue. I am gathering this information and will coordinate with the teams to work through these known cases. Although we will continue to monitor this discussion I also wanted to make sure that everyone was aware they can raise these types of issues with us through the support channels.
I also wanted to make sure that everyone was aware they can
raise these types of issues with us through the support channels.
I just wanted to note that I first opened a support case for the glibc update issue (case #01182854). However, the initial answer I received there did not address the issue. So, I turned to the discussion session hoping to get more info from others. I then posted this thread link in my support case and finally got the appropriate response.
After getting less than satisfactory response from opening a support ticket earlier about this I ended up here instead and I'm pretty sure I am not the only one who found the "official" support channel not at all acceptable in this case. I was for the first time hoping to get the evaluation questionnaire after closing the last support ticket I had open for this but didn't.
All I really would like to see at this point is for Red Hat to figure out how to monitor its errata release process so when whatever fails in the process a flag goes up and you guys can sort it out without me needing to spend my time telling you that updates are broken again. If I can notice errata missing I really can't understand why Red Hat can't notice it.
Please don't get me wrong. We are having this issue since back in 2010. Within most tickets we've been told this our problem. Our proxy is not working properly, etc. Despite pointing out using direct connections, seening the issue at our company and our customers.
Maybe it would be worth to train the support crew to also check the CDN. And after four years ... please make this CDN stuff work.
This thread is not about user error or local cache problems. It is about the CDN not delivering the updates. No matter how many times I yum clean all I still get metadata dated Aug 26 for the rhel-7-workstation-rpms channel which does not include the glibc update or any other released after Aug 26.
I have verified that the packages that were missing are now showing up for Errata released from Aug 26th.
I also wanted everyone to know that a team at Red Hat met today to discuss this issue and actions that we need to take.
Some of them are going to take a while to implement but in the short term there are a few things we want to put into place. Give us a week to work through some of this and will update this discussion with findings and actions being taken.
The glibc packages that were missing in the rhel-7-workstation-rpms should be available now. You will likely need to do "yum clean all && yum update" if you've tried earlier today to download the packages or they still appear not to show up. If you still have problems obtaining the packages, please email GSS for assistance. Our apologies for all of this -- I'm still unclear as to why this happened, but at least it's corrected now. Thanks.
Hello,
I just wanted to follow up with a note. This week we worked on updating some systems in order to improve their performance. We also increased some of our monitoring capabilities in order to catch issues faster if they do happen. We will continuing with longer term improvements that will decrease the possibilities of this type of issue from even happening.
Thank you for your patience.
Where are you looking? I see the errata posted in several locations:
https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.0/x86_64/product-errata
https://access.redhat.com/downloads/content/69/ver=/rhel---6/6.6%20Beta/x86_64/product-errata
https://rhn.redhat.com/errata/rhel-server-7-errata.html
https://rhn.redhat.com/errata/rhel-server-6-errata.html
Daryl,
The monitoring yesterday did identify that we were having issues and the teams went to work on it. There were a lot of things queued up and it took longer then we wanted for things to complete. As Bryan mentioned they are available in multiple channels now.
The teams involved continue to work on improvements and I highlight this discussion to show impact to customers.
Hi Daryl,
I have not heard of any reports of missing that package and was able to find it in the package list by filtering on "libvirt" https://access.redhat.com/downloads/content/rhel---7/x86_64/2508/libvirt/1.1.1-29.el7_0.3/x86_64/fd431d51/package ...
So as far as I can tell there was not an issue. We do need to improve the package search for the new downloads and I am putting together requirements around that now.
Shea
Can we get clarification on one assumption I have?
I assume there are several CDN locations which have to be synced and maintained (as opposed to a single CDN repository)?
And the reason I ask is we focus on the end-user experience and then ask specifics, which in many cases do not apply universally for this situation.
The new bash update has been announced. I yum cleaned all and also manually deleted the relevant /var/cache/yum directory before running yum. The repodata are still dated Sept 25 (about 12 hrs back). Let's see how long it will take this time.
[EDIT] Got the update on RHEL-6 and -7 (Server). It was quick this time. Thanks.
[EDIT2] RHEL-5 (Server) as well.
Pages
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
