Does Red Hat bother to monitor its CDN for erratum delivery?
Rewording post:
We have repeatedly been told that Red Hat monitors its CDN for erratum delivery. In practice, this does not appear to be the case. The question is why does Red Hat not monitor its CDN and instead makes customers jump through all kinds of hoops attempting to get Red Hat to actually look at their CDN and see the obvious problem with old metadata or missing packages.
The other frustrating aspect is to see Red Hat erratum delivered by CentOS and even Oracle before we can get them via the RHN CDN. It is not a matter of slowness, but the erratum are not available!
You can see these on the RHN hosted console when looking at how auto-applied-errata failed to be applied. If I worked at Red Hat within RHN, I would immediately write this query and create a dashboard.
"How many systems on the CDN have auto-apply-errata checked and failed to install a given errata?" Seeing a very high error rate would raise some sort of internal red flags, one would think.
original rant text:
Just curious why Red Hat does not actively monitor that its CDN is delivering erratum. It is frustrating to consistently see Oracle and now Centos deliver errata faster than Red Hat RHEL RHN does.
Responses
It was just brought to my attention that tzdata is not available via the CDN (RHN Hosted this time) for 32-bit RHEL5 clients. Freshly downloaded metadata is dated Sept. 30 still.
New php packages (5.3.3-27.el6_5.2) was released in RHEL6 optional channel on tuesday october 7 (maybe monday), but no corresponding php packages (i.e. php, php-common) was released in the updates channel. Errata was from september 30. Not only late, but incomplete release in CDN. Am i the only one seeing this? This breaks dependencies.
Hi,
Which architecture and variant are you using? From what I can tell, the packages were available on cdn.redhat.com as of October 1st with the exception of one repo: Workstation-Optional (x86_64). That was detected on October 6th and the packages were pushed to that repo at that point. I'm not seeing any missing php content on CDN.
Thanks
-- Dennis
We use RHEL6 Server x86_64. We also use RHEL6 Client x86_64, but there the packages all came in at the same time. In the client variant all (i think) php packages are in the optional channel. I will have to check again for errors on our side, I guess.
Well, I can't see anything on our end making this not work. We use lftp mirror against redhat cdn and we still have no updated php packages from updates channel. Could 'our' cdn mirror have some problems?
Here's how you can check:
curl -s --cacert /etc/rhsm/ca/redhat-uep.pem --cert /etc/pki/entitlement/917147791860476371.pem --key /etc/pki/entitlement/917147791860476371-key.pem https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os/repodata/repomd.xml | grep timestamp | head -1
date -d '@1412693341.53'
Tue Oct 7 10:49:01 EDT 2014
You can compare that timestamp to what you have locally. The URL I used is for RHEL 6 Server x86_64. Adjust accordingly for other architectures or variants.
Thanks for the tip - i now see that I can download the missing packages with curl so they are definitely available in the cdn. So this is a completely off topic I guess, but any guess why mrepo (using lftp mirror) does not download the packages?
Pro
433 points
Wow, somebody edited the subject/title of my posting. Conspiracy!
Red Hat
25369 points
A "conspiracy" would require two or more participants. This was just me modifying the post title, so it's a "coverup". ;)
In all seriousness though, I removed the "Answer: No" portion of the title because it was not accurate to the content in the thread and potentially misleading to people skimming the discussions.
Pro
433 points
Not accurate? Up until even two weeks ago, we pointed out a CDN issue that was not captured by Red Hat's monitoring? Are you saying Red Hat now monitors its CDN?
You could have asked first, before censoring.
Guru
6863 points
This response is not intended to simply be argumentative - but I have some perspective.
In all fairness - I'm not sure it is accurate to claim that Red Hat does not monitor their CDN. It is obvious that portions of the CDN is not updated occasionally, and they are aware (as indicated by the responses on this thread - and some others) and they have stated they are working on correcting the issue and trying to avoid the issue in the future. Since this thread has changed it's "spirit" and has become something different than when it was first started and the subject (which is what I see when I glance at my inbox) is no longer representative of what this thread is now discussing.
Again the RHEL 7 Workstation channels are wedged on the CDN. rhel-7-workstation-rpms is still giving out metadata from 10/28 and rhel-7-workstation-fastrack-rpms is giving out metadata from 10/23 both of which should have been updated yesterday, today, or both.
Is there any word on when yesterday's security updates for wget and php will be available from Red Hat? They still don't show up on the CDN or on RHN hosted but have been available from Oracle and CentOS for a long time now. I've tried various versions and flavors of RHEL6 and RHEL7 on both the CDN and on RHN hosted so I'm assuming they just aren't there at all this time.
Community Member
91 points
I have waited and waited for the updated wget package to become available (for RHEL 6 server).
Today, after trying (yet again) a "rm -fr /var/cache/yum/*; yum check-update" sequence without success, I resorted to downloading the source package file and then building the package set within "mock".
wget-1.12-5.el6_6.1.i386.rpm
wget-1.12-5.el6_6.1.src.rpm
wget-1.12-5.el6_6.1.x86_64.rpm
wget-debuginfo-1.12-5.el6_6.1.i386.rpm
wget-debuginfo-1.12-5.el6_6.1.x86_64.rpm
Guru
6863 points
Alan - I am unable to respond to your recent post.
I'm not sure when it was published, but it is there now. Do you use Satellite or RHN?
[root@apoc html]# yum list wget
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Installed Packages
wget.x86_64 1.12-5.el6_6.1 @rhel-x86_64-server-6
[root@apoc html]# rpm -qi wget
Name : wget Relocations: (not relocatable)
Version : 1.12 Vendor: Red Hat, Inc.
Release : 5.el6_6.1 Build Date: Fri 24 Oct 2014 10:53:14 AM EDT
... ...
I'm not sure which channel wget-debuginfo is included in.
It has been a while where updates seemed to be working well, at least I didn't notice any not working at the times I checked them. But this morning I still see none of the updates from yesterday (2014-12-18) including RHSA-2014:2010-1 in the 7 workstation channels. Could someone kindly check if the CDN needs poked to get this security advisory pushed through to us? Thanks in advance.
RHBA-2015:0007 (mariadb) and RHSA-2015:0008 (libvirt) from yesterday are not available on RHEL7 Workstation via the SM CDN.
Pro
433 points
Anybody from Red Hat wish to comment if their current monitoring of the CDN shows any issues? I suspect it should be....
The jasper errata
https://rhn.redhat.com/errata/RHSA-2015-0074.html
released on 2015-01-22 is missing from the base RHEL6 server x86_64 channel on the hosted side which is still giving out metadata for that channel dated 2015-01-20.
Pro
433 points
(wonders if the forum software is going to work today...)
Bryan, jasper was finally available this Saturday morning, a day and a half late. Was it showing up as an issue with your monitoring?
As of Sunday. 25 Jan 2015 at 11:42 AM PDT, the jasper update for my 32-bit RHEL 6 self-support Client has still not appeared via yum. Even after 'yum clean metadata', the newly downloaded metadata files are still dated 2015-01-20.
I do get an alert for the update under "System Status" in Classic Subscription Management, and everything else appears to be in order. I finally just downloaded and applied the RPM manually.
We have been reporting these problems now for over 6 months and I'm wondering if any progress is being made on resolving the underlying cause of the failures to update some repo metadata periodically?
I have unhappy people who gave up today and also fixed things by downloading the rpms and installing them locally. We should not ever need to do this for routine updates. I don't know what to tell them at this point, saying I've been trying to work with Red Hat for the past 6 months to get this resolved isn't making either of us look good at this point.
Red Hat
Active Contributor
286 points
Hello John,
We have been monitoring our systems and have started testing for availability on every errata. We still have some challenges to overcome but are making progress. In this particular case our monitoring and review have not been able to determine an issue on the Red Hat side.
We are currently investigating if there may be issues outside of Red Hat that may be contributing to what you are seeing. I would like to help troubleshoot this with you through a support case in order to make sure we capture everything and you get timely responses. If you will open a case and reference this discussion I will coordinate with the various teams at Red Hat.
Shea DeAntonio
Pro
433 points
Unbelievable, GSS asks for a spacewalk report? Seriously, this is crazy. Please humor us, what exact repo metadata time does your monitoring show for the RHEL6 channels available on the CDN? I doubt you'll tell us.
Pro
433 points
Crickets... So I sit here today watching the various channels on the CDN hoping for delivery of critical errata and not seeing much yet. Kind of lame to see CDN slowness noted in bugzilla as well. So frustrated with Red Hat right now.
Pro
433 points
Still awaiting the RHEL7 kernel to appear on the CDN for workstation 64bit (RHSA-2015:0102)....
This update just seems to be missing its errata altogether!
libvirt.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-client.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-devel.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
libvirt-python.x86_64 0.10.2-46.el6_6.3 rhel-x86_64-workstation-6
The updateinfo metadata has nothing, the portal has nothing, RHN classic says I have 0 errata and 4 packages out of date. Huh?
Pro
433 points
@Shea, does the monitoring show the current issue with mariadb errata not on the CDN?
# yum clean metadata
...
# yum list mariadb
mariadb.x86_64 1:5.5.40-2.el7_0 rhel-7-workstation-rpms
Pro
433 points
Kind of difficult to update my box to RHEL 7.1 when this keeps happening when running 'yum update'
https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/extras/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
I've noticed that it appears the fastrack channels have update notices that conflict with the other channels after the 7.1 release came out. I gave it a few days to see it was corrected but apparently someone needs to look into it.
# yum check-update
Loaded plugins: langpacks, product-id, subscription-manager
Update notice RHBA-2014:1045 (from rhel-7-workstation-optional-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-optional-rpms repository.
Update notice RHBA-2014:1180 (from rhel-7-workstation-optional-rpms) is broken, or a bad duplicate, skipping.
...
#
# yum --disablerepo=\*fastrack\* check-update
Loaded plugins: langpacks, product-id, subscription-manager
#
John I filed this with our internal engineering team. I am not sure if its a tooling issue or if I need to follow up with RHEL engineering. I will update when i get information
Red Hat
Active Contributor
286 points
John this is sorted now. Root cause was that the packages were in the fast track and regular repos but with different time stamps. They should have just been moved to the regular repo at GA and we are adding this to lessons learned for the release.
Thanks again.
Shea
Is this issue with the CDN not being up to date still ongoing ?
Looking at the errata for RHEL6, I can see that dracut and bind were updated recently (9th and 10th March) yet the latest update on our RHUA server (which claims to have synchronised today) is the 389-ds-base update released on the 5th March.
Thanks Shea. This "infected" our satellite server also and that did not self-correct after the next sync. Any suggestion on how I can go about fixing it there? I've also asked in a support ticket but no one seems to be talking to me there ...
Pages
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.