Does Red Hat bother to monitor its CDN for erratum delivery?

Latest response

Rewording post:
We have repeatedly been told that Red Hat monitors its CDN for erratum delivery. In practice, this does not appear to be the case. The question is why does Red Hat not monitor its CDN and instead makes customers jump through all kinds of hoops attempting to get Red Hat to actually look at their CDN and see the obvious problem with old metadata or missing packages.

The other frustrating aspect is to see Red Hat erratum delivered by CentOS and even Oracle before we can get them via the RHN CDN. It is not a matter of slowness, but the erratum are not available!

You can see these on the RHN hosted console when looking at how auto-applied-errata failed to be applied. If I worked at Red Hat within RHN, I would immediately write this query and create a dashboard.

"How many systems on the CDN have auto-apply-errata checked and failed to install a given errata?" Seeing a very high error rate would raise some sort of internal red flags, one would think.

original rant text:
Just curious why Red Hat does not actively monitor that its CDN is delivering erratum. It is frustrating to consistently see Oracle and now Centos deliver errata faster than Red Hat RHEL RHN does.


It was just brought to my attention that tzdata is not available via the CDN (RHN Hosted this time) for 32-bit RHEL5 clients. Freshly downloaded metadata is dated Sept. 30 still.

New php packages (5.3.3-27.el6_5.2) was released in RHEL6 optional channel on tuesday october 7 (maybe monday), but no corresponding php packages (i.e. php, php-common) was released in the updates channel. Errata was from september 30. Not only late, but incomplete release in CDN. Am i the only one seeing this? This breaks dependencies.


Which architecture and variant are you using? From what I can tell, the packages were available on as of October 1st with the exception of one repo: Workstation-Optional (x86_64). That was detected on October 6th and the packages were pushed to that repo at that point. I'm not seeing any missing php content on CDN.

-- Dennis

We use RHEL6 Server x86_64. We also use RHEL6 Client x86_64, but there the packages all came in at the same time. In the client variant all (i think) php packages are in the optional channel. I will have to check again for errors on our side, I guess.

Well, I can't see anything on our end making this not work. We use lftp mirror against redhat cdn and we still have no updated php packages from updates channel. Could 'our' cdn mirror have some problems?

Here's how you can check:

curl -s --cacert /etc/rhsm/ca/redhat-uep.pem --cert /etc/pki/entitlement/917147791860476371.pem --key /etc/pki/entitlement/917147791860476371-key.pem | grep timestamp | head -1

date -d '@1412693341.53'
Tue Oct 7 10:49:01 EDT 2014

You can compare that timestamp to what you have locally. The URL I used is for RHEL 6 Server x86_64. Adjust accordingly for other architectures or variants.

Thanks for the tip - i now see that I can download the missing packages with curl so they are definitely available in the cdn. So this is a completely off topic I guess, but any guess why mrepo (using lftp mirror) does not download the packages?

Wow, somebody edited the subject/title of my posting. Conspiracy!

A "conspiracy" would require two or more participants. This was just me modifying the post title, so it's a "coverup". ;)

In all seriousness though, I removed the "Answer: No" portion of the title because it was not accurate to the content in the thread and potentially misleading to people skimming the discussions.

Not accurate? Up until even two weeks ago, we pointed out a CDN issue that was not captured by Red Hat's monitoring? Are you saying Red Hat now monitors its CDN?

You could have asked first, before censoring.

This response is not intended to simply be argumentative - but I have some perspective.

In all fairness - I'm not sure it is accurate to claim that Red Hat does not monitor their CDN. It is obvious that portions of the CDN is not updated occasionally, and they are aware (as indicated by the responses on this thread - and some others) and they have stated they are working on correcting the issue and trying to avoid the issue in the future. Since this thread has changed it's "spirit" and has become something different than when it was first started and the subject (which is what I see when I glance at my inbox) is no longer representative of what this thread is now discussing.

Again the RHEL 7 Workstation channels are wedged on the CDN. rhel-7-workstation-rpms is still giving out metadata from 10/28 and rhel-7-workstation-fastrack-rpms is giving out metadata from 10/23 both of which should have been updated yesterday, today, or both.

Hi. I noticed this as well earlier today. At this point RHEL 7 Workstation has been updated and I'm verifying the rest of the repos as well.

It appears that rhel-i386-server-5 on RHN hosted is stuck with metadata dated 10/21 too. :(

I can confirm this. On my RHEL 5 32-bit system, rhel-i386-server-5 has:

1466 Oct 21 12:33 repomd.xml

RHEL 5 x86_64 seems to be up to date. rhel-x86_64-server-5 has:

1466 Oct 29 12:09 repomd.xml

Is there any word on when yesterday's security updates for wget and php will be available from Red Hat? They still don't show up on the CDN or on RHN hosted but have been available from Oracle and CentOS for a long time now. I've tried various versions and flavors of RHEL6 and RHEL7 on both the CDN and on RHN hosted so I'm assuming they just aren't there at all this time.

John, I just wanted to acknowledge that we are aware of this and working on it.

I have waited and waited for the updated wget package to become available (for RHEL 6 server).

Today, after trying (yet again) a "rm -fr /var/cache/yum/*; yum check-update" sequence without success, I resorted to downloading the source package file and then building the package set within "mock".


Alan - I am unable to respond to your recent post.
I'm not sure when it was published, but it is there now. Do you use Satellite or RHN?

[root@apoc html]# yum list wget
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Installed Packages
wget.x86_64                                              1.12-5.el6_6.1                                               @rhel-x86_64-server-6

[root@apoc html]# rpm -qi wget
Name        : wget                         Relocations: (not relocatable)
Version     : 1.12                              Vendor: Red Hat, Inc.
Release     : 5.el6_6.1                     Build Date: Fri 24 Oct 2014 10:53:14 AM EDT
... ... 

I'm not sure which channel wget-debuginfo is included in.

Thank you James.

I use RHN Classic and can now confirm that the wget package is indeed available.

A "yum reinstall wget" replaced my self-built package with the corresponding Red Hat package.

It has been a while where updates seemed to be working well, at least I didn't notice any not working at the times I checked them. But this morning I still see none of the updates from yesterday (2014-12-18) including RHSA-2014:2010-1 in the 7 workstation channels. Could someone kindly check if the CDN needs poked to get this security advisory pushed through to us? Thanks in advance.

Thanks John, we are looking into it.

RHBA-2015:0007 (mariadb) and RHSA-2015:0008 (libvirt) from yesterday are not available on RHEL7 Workstation via the SM CDN.

Anybody from Red Hat wish to comment if their current monitoring of the CDN shows any issues? I suspect it should be....

Daryl: are you seeing an issue? We have improved the monitoring, and are resolving issues if they arise.

-- bk

The jasper errata

released on 2015-01-22 is missing from the base RHEL6 server x86_64 channel on the hosted side which is still giving out metadata for that channel dated 2015-01-20.

(wonders if the forum software is going to work today...)

Bryan, jasper was finally available this Saturday morning, a day and a half late. Was it showing up as an issue with your monitoring?

As of Sunday. 25 Jan 2015 at 11:42 AM PDT, the jasper update for my 32-bit RHEL 6 self-support Client has still not appeared via yum. Even after 'yum clean metadata', the newly downloaded metadata files are still dated 2015-01-20.

I do get an alert for the update under "System Status" in Classic Subscription Management, and everything else appears to be in order. I finally just downloaded and applied the RPM manually.

We have been reporting these problems now for over 6 months and I'm wondering if any progress is being made on resolving the underlying cause of the failures to update some repo metadata periodically?

I have unhappy people who gave up today and also fixed things by downloading the rpms and installing them locally. We should not ever need to do this for routine updates. I don't know what to tell them at this point, saying I've been trying to work with Red Hat for the past 6 months to get this resolved isn't making either of us look good at this point.

Hello John,
We have been monitoring our systems and have started testing for availability on every errata. We still have some challenges to overcome but are making progress. In this particular case our monitoring and review have not been able to determine an issue on the Red Hat side.

We are currently investigating if there may be issues outside of Red Hat that may be contributing to what you are seeing. I would like to help troubleshoot this with you through a support case in order to make sure we capture everything and you get timely responses. If you will open a case and reference this discussion I will coordinate with the various teams at Red Hat.

Shea DeAntonio

Case 01340557

Thanks Shea.

Great I have let the team know.

Unbelievable, GSS asks for a spacewalk report? Seriously, this is crazy. Please humor us, what exact repo metadata time does your monitoring show for the RHEL6 channels available on the CDN? I doubt you'll tell us.

Crickets... So I sit here today watching the various channels on the CDN hoping for delivery of critical errata and not seeing much yet. Kind of lame to see CDN slowness noted in bugzilla as well. So frustrated with Red Hat right now.

Still awaiting the RHEL7 kernel to appear on the CDN for workstation 64bit (RHSA-2015:0102)....

I ran yum clean metadata and then was able to get the errata, ufff.

Sorry I meant to comment on this yesterday with everything that was going out the last few days we saw longer then normal back up. Things should be updated now.

This update just seems to be missing its errata altogether!

libvirt.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-client.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-devel.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-python.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6

The updateinfo metadata has nothing, the portal has nothing, RHN classic says I have 0 errata and 4 packages out of date. Huh?

The errata push failed and the team tried to repush it this afternoon and it was not succesful. They are still looking into it and will get it taken care of soon.

@Shea, does the monitoring show the current issue with mariadb errata not on the CDN?

# yum clean metadata

# yum list mariadb
mariadb.x86_64 1:5.5.40-2.el7_0 rhel-7-workstation-rpms

I reported this when I was alerted by the support case earlier today and have seen new errata notices go out but still do not see the packages. I have let engineering team know and continue to monitor.

Kind of difficult to update my box to RHEL 7.1 when this keeps happening when running 'yum update' [Errno 14] HTTPS Error 404 - Not Found

I've noticed that it appears the fastrack channels have update notices that conflict with the other channels after the 7.1 release came out. I gave it a few days to see it was corrected but apparently someone needs to look into it.

# yum check-update
Loaded plugins: langpacks, product-id, subscription-manager
Update notice RHBA-2014:1045 (from rhel-7-workstation-optional-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-optional-rpms repository.
Update notice RHBA-2014:1180 (from rhel-7-workstation-optional-rpms) is broken, or a bad duplicate, skipping.
# yum --disablerepo=\*fastrack\* check-update
Loaded plugins: langpacks, product-id, subscription-manager

John I filed this with our internal engineering team. I am not sure if its a tooling issue or if I need to follow up with RHEL engineering. I will update when i get information

John this is sorted now. Root cause was that the packages were in the fast track and regular repos but with different time stamps. They should have just been moved to the regular repo at GA and we are adding this to lessons learned for the release.

Thanks again.

Is this issue with the CDN not being up to date still ongoing ?

Looking at the errata for RHEL6, I can see that dracut and bind were updated recently (9th and 10th March) yet the latest update on our RHUA server (which claims to have synchronised today) is the 389-ds-base update released on the 5th March.

Thanks Shea. This "infected" our satellite server also and that did not self-correct after the next sync. Any suggestion on how I can go about fixing it there? I've also asked in a support ticket but no one seems to be talking to me there ...

I have asked that the support case get some attention since it may require more then I can do here.

What version of satellite was impaced?

Hi Bryan,

An up to date 5.6 satellite. Just has the same symptoms since the original 7.1 GA sync.