Does Red Hat bother to monitor its CDN for erratum delivery?

Latest response

Rewording post:
We have repeatedly been told that Red Hat monitors its CDN for erratum delivery. In practice, this does not appear to be the case. The question is why does Red Hat not monitor its CDN and instead makes customers jump through all kinds of hoops attempting to get Red Hat to actually look at their CDN and see the obvious problem with old metadata or missing packages.

The other frustrating aspect is to see Red Hat erratum delivered by CentOS and even Oracle before we can get them via the RHN CDN. It is not a matter of slowness, but the erratum are not available!

You can see these on the RHN hosted console when looking at how auto-applied-errata failed to be applied. If I worked at Red Hat within RHN, I would immediately write this query and create a dashboard.

"How many systems on the CDN have auto-apply-errata checked and failed to install a given errata?" Seeing a very high error rate would raise some sort of internal red flags, one would think.

original rant text:
Just curious why Red Hat does not actively monitor that its CDN is delivering erratum. It is frustrating to consistently see Oracle and now Centos deliver errata faster than Red Hat RHEL RHN does.

Responses

When you mention "deliver errata faster" - are you referring to download speeds for packages?

If you have some definitive numbers indicating the download speeds (including date/time and the URL) I am fairly certain Red Hat would be interested in that data. I imagine that Red Hat is like the rest of us and has a service level agreement with their providers and if that is level is not being met, they should know.

With Oracle/CentOS, are you using the fastestmirror plugin?

I would look into using:

yum-fastestmirror RHEL 5
yum-plugin-fastestmirror RHEL 6 Optional Channel

NOTE: We use a Satellite in my environment so I don't actually use the fastestmirror plugin, but I believe it should help.

Hello, Are you getting RHEL7 errata delivered via the CDN, like openjdk update?

I, actually, have not done much with RHEL7 yet (other than download the DVD ISO). Hopefully someone else can lend some advice. I don't actively monitor our Satellite syncs either. So, unfortunately I don't have much actual experience with the CDN.

I am interested in what you are experiencing also. Are you not getting the notification about the Errata or is it that the packages are not available when you expect them to be?

I looked for openjdk and found https://access.redhat.com/errata/RHSA-2014:0907. If this is what you are referencing then we can use this as an example of how things could be better.

Hello,

Do you have a RHEL7 system registered to RHN hosted? If so, compare what errata you can get via yum vs what is shown on the portal for released errata.

Ok will look into what you are saying and discuss with some of the support team.

Daryl, consider adding your request to this discussion where Red Hat is asking for "what would you change".

So on July 17 I received email announcing java-1.7.0-oracle updates bringing it up to version 7 update 65. Now it is July 28 and when I check on an otherwise current RHEL7 Workstation using the CDN it offers update 55 still as the most recent. I have seen this sort of thing happen multiple times since RHEL7 was released and I'm about to the point where I just do not trust the CDN. I reported this CDN problem to Red Hat on July 18. If the CDN can't deliver critical security errata it is actually harming me by leaving machines I entrust to it vulnerable.

John:

Thank you for pointing this out. The packages are on the CDN now. We are digging into why the original push failed, and why the escalation did not reach the appropriate teams. Please let me know if the issue is not resolved.

IBM java appears to have an update today. I don't know how (exactly) to validate, but I wonder if today's event would be a good example.

https://rhn.redhat.com/rhn/errata/details/Details.do?eid=27349

I'm curious whether this is an issue specific to the new Oracle Java channels.

It is not an Oracle Java channel problem. It has happened with even the base workstation channel on RHEL7 too. Sometimes the metadata isn't rebuilding in the CDN properly and it has happened to at least several different channels.

I don't think a RHEL6 errata can shed any light on the particular problem I've seen as that has been RHEL7 specific, but not channel specific.

Thanks for the additional clarification John.

We are still doing root cause analysis of the issue. As you find these issues, please post them here or PM me as I can reach out to get them resolved and help identify the problems.

rhel-7-workstation-rpms and rhel-7-workstation-fastrack-rpms are getting metadata dated Aug 7 today - so again missing available updates from the 12th. I did not check other channels so these may not be the only ones not updating.

Another day another security update (openssl) is released but not available via the CDN. I'm really frustrated with this continuing to happen.

John: Which erratum can you not find. RHSA-2014:1052?

Every errata released since the Aug 7 since that was the last time the metadata for the repos was updated. I'm not where I can check the state of this today but as of yesterday that included at least RHBA-2014:1045, RHSA-2014:1042, RHEA-2014:1046, RHBA-2014:1049, RHSA-2014:1052, RHBA-2014:1058, RHBA-2014:1057, and RHBA-2014:1055.

Bryan: With this morning's errata (Aug. 18) both rhel-7-workstation-optional-rpms and rhel-7-workstation-rpms have updated metadata and all the old missing rpms appear to available from those channels although I only spot checked them.

The rhel-7-workstation-fastrack-rpms channel is still stuck on Aug. 7 metadata and so is not providing RHBA-2014:1045-1 which was released on Aug. 12.

Bryan,

You are having the same CDN issue now.

An important security update to glibc was announced on Aug 29, 2014:

https://rhn.redhat.com/errata/RHSA-2014-1110.html
Advisory: RHSA-2014:1110-1

As of Sept 1, 2014, the referenced glibc update is not available through RHN.

same problem here. glibc update is not available through the CDN. but is available in CentOS the same day it was issued.

for the moment, is faster to get updates from CentOS.

"for the moment, is faster to get updates from CentOS."

Unfortunately this is getting true. In addition to RHEL 5, 6 and 7 systems, I am running CentOS and Scientific Linux. Yesterday's bash update from RH via 'yum' came through many hours after it became available from CentOS and SL.

I am aware Red Hat has been working on the issue but the delay in critical updates like yesterday's is becoming a serious issue.

I'll add at that this time I see this update is also missing on x86_64 RHEL6 Workstation and RHEL5 Server using SM and the CDN. So it isn't just a RHEL7 issue.

Hello,
Thank you for continuing to let us know about this issue. I am gathering this information and will coordinate with the teams to work through these known cases. Although we will continue to monitor this discussion I also wanted to make sure that everyone was aware they can raise these types of issues with us through the support channels.

I also wanted to make sure that everyone was aware they can
raise these types of issues with us through the support channels.

I just wanted to note that I first opened a support case for the glibc update issue (case #01182854). However, the initial answer I received there did not address the issue. So, I turned to the discussion session hoping to get more info from others. I then posted this thread link in my support case and finally got the appropriate response.

After getting less than satisfactory response from opening a support ticket earlier about this I ended up here instead and I'm pretty sure I am not the only one who found the "official" support channel not at all acceptable in this case. I was for the first time hoping to get the evaluation questionnaire after closing the last support ticket I had open for this but didn't.

All I really would like to see at this point is for Red Hat to figure out how to monitor its errata release process so when whatever fails in the process a flag goes up and you guys can sort it out without me needing to spend my time telling you that updates are broken again. If I can notice errata missing I really can't understand why Red Hat can't notice it.

Please don't get me wrong. We are having this issue since back in 2010. Within most tickets we've been told this our problem. Our proxy is not working properly, etc. Despite pointing out using direct connections, seening the issue at our company and our customers.
Maybe it would be worth to train the support crew to also check the CDN. And after four years ... please make this CDN stuff work.

If you're trying to get the glibc update (or any other) and it's not showing up, try doing a "yum clean all && yum update"; sometimes if the cache isn't stale enough this is required to force a re-download of the appropriate metadata for the update to show up.

This thread is not about user error or local cache problems. It is about the CDN not delivering the updates. No matter how many times I yum clean all I still get metadata dated Aug 26 for the rhel-7-workstation-rpms channel which does not include the glibc update or any other released after Aug 26.

Thanks for specifying that, John. The system I was looking at is using rhel-7-server-rpms and glibc is most definitely there. We are looking into why it apparently didn't make it to all of the channels.

I am using the same channel rhel-7-server-rpms, and I got the update 19 hours ago ( 4 days after CentOS ).

I have verified that the packages that were missing are now showing up for Errata released from Aug 26th.
I also wanted everyone to know that a team at Red Hat met today to discuss this issue and actions that we need to take.

Some of them are going to take a while to implement but in the short term there are a few things we want to put into place. Give us a week to work through some of this and will update this discussion with findings and actions being taken.

Thanks Thomas

The glibc packages that were missing in the rhel-7-workstation-rpms should be available now. You will likely need to do "yum clean all && yum update" if you've tried earlier today to download the packages or they still appear not to show up. If you still have problems obtaining the packages, please email GSS for assistance. Our apologies for all of this -- I'm still unclear as to why this happened, but at least it's corrected now. Thanks.

Hello,
I just wanted to follow up with a note. This week we worked on updating some systems in order to improve their performance. We also increased some of our monitoring capabilities in order to catch issues faster if they do happen. We will continuing with longer term improvements that will decrease the possibilities of this type of issue from even happening.

Thank you for your patience.

Thank you for the update. Very much appreciated.

Hi Shea,

Thanks for the response. What is the monitoring reporting today for the bash erratum delivery? Kind of annoying to now have gotten 3 alerts from Red Hat about it (blogs, direct email, errata email), but not have it yet available on the CDN.

It also defeats the purpose of 'auto-apply-errata' as the system attempts to apply it, but does not find the package on the CDN :( 'Error while executing packages action: empty transaction [[6]]'

daryl

Where are you looking? I see the errata posted in several locations:

https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.0/x86_64/product-errata

https://access.redhat.com/downloads/content/69/ver=/rhel---6/6.6%20Beta/x86_64/product-errata

https://rhn.redhat.com/errata/rhel-server-7-errata.html

https://rhn.redhat.com/errata/rhel-server-6-errata.html

Bryan,

Thanks for your response, hopefully Shea's response after yours helped. For an extact timeline yesterday, here's what I have now found:

  1. At 9:16 AM CDT, rhel-x86_64-workstation-6 repomd.xml was generated at RH
  2. At 10:28 AM CDT, I got the bash erratum alert email
  3. At 10:55 AM CDT, my system was scheduled for errata application on RHN hosted
  4. At 12:56 PM CDT, my system picked up the scheduled action via its 4 hour check interval.
  5. At 12:56 PM CDT, the system failed to install the errata since it was not available for download, rhn shows error "Error while executing packages action: empty transaction [[6]]" (code -1)
  6. At 2:51 PM CDT, I manually deleted metadata (yum clean metadata) and attempted to manually (yum update) only to find the channel metadata was still dated 22 September.
  7. At 9:00 PM CDT, I again tried to manually clean the metadata and apply the errata, it was not available for download
  8. At ~10:30 PM CDT, a collegue of mine reports that it was finally available on the CDN, but I was snoring away.
  9. At 7:15 AM CDT today, I was able to manually update the package.

So again, RHN hosted 'auto-apply-errata' failed and the CDN was very late to update. CentOS and Oracle systems got the errata in a timely fashion.

daryl

Daryl,
The monitoring yesterday did identify that we were having issues and the teams went to work on it. There were a lot of things queued up and it took longer then we wanted for things to complete. As Bryan mentioned they are available in multiple channels now.

The teams involved continue to work on improvements and I highlight this discussion to show impact to customers.

Hello Shea,

Is your monitoring indicating a current problem with the rhel-7-workstation-rpms channel? Currently dated: Sep 30 23:12 and missing libvirt-1.1.1-29.el7_0.3

daryl

Hi Daryl,
I have not heard of any reports of missing that package and was able to find it in the package list by filtering on "libvirt" https://access.redhat.com/downloads/content/rhel---7/x86_64/2508/libvirt/1.1.1-29.el7_0.3/x86_64/fd431d51/package ...

So as far as I can tell there was not an issue. We do need to improve the package search for the new downloads and I am putting together requirements around that now.

Shea

Shea,

Did you click on that link? It 404 errors for me.

The link you followed may have expired.
Click on the back button in your browser to refresh the page and try again.
Download links are dynamic and time-sensitive.
If you used a bookmark to try to reach /content/origin/rpms/libvirt/1.1.1/29.el7_0.3/fd431d51/libvirt-1.1.1-29.el7_0.3.x86_64.rpm please delete it. Instead, bookmark the page describing the object.

daryl

Good morning,

No change this morning. Channel metadata still old, package unavailable from CDN. Your monitoring is not showing this issue or is not checking for it?

daryl

I realize today I did not actually click on it and am following up on why it's 404 error now.

The package is now available for download. I do not have answers yet as to why it was not picked up.

Thanks, the channel metadata on the CDN is still old. Will it update soon as well?

The same problem with rhel-7-server-rpms. Just checked after 'yum clean all'. repomd.xml is dated Sept 30 and the latest libvirt in there is 1.1.1-29.el7_0.1.

Well, the hope that the tzdata errata would 'fix' this does not appear to be the case. Sigh.

Can we get clarification on one assumption I have?
I assume there are several CDN locations which have to be synced and maintained (as opposed to a single CDN repository)?

And the reason I ask is we focus on the end-user experience and then ask specifics, which in many cases do not apply universally for this situation.

The new bash update has been announced. I yum cleaned all and also manually deleted the relevant /var/cache/yum directory before running yum. The repodata are still dated Sept 25 (about 12 hrs back). Let's see how long it will take this time.

[EDIT] Got the update on RHEL-6 and -7 (Server). It was quick this time. Thanks.
[EDIT2] RHEL-5 (Server) as well.

Pages