Does Red Hat bother to monitor its CDN for erratum delivery?
Rewording post:
We have repeatedly been told that Red Hat monitors its CDN for erratum delivery. In practice, this does not appear to be the case. The question is why does Red Hat not monitor its CDN and instead makes customers jump through all kinds of hoops attempting to get Red Hat to actually look at their CDN and see the obvious problem with old metadata or missing packages.
The other frustrating aspect is to see Red Hat erratum delivered by CentOS and even Oracle before we can get them via the RHN CDN. It is not a matter of slowness, but the erratum are not available!
You can see these on the RHN hosted console when looking at how auto-applied-errata failed to be applied. If I worked at Red Hat within RHN, I would immediately write this query and create a dashboard.
"How many systems on the CDN have auto-apply-errata checked and failed to install a given errata?" Seeing a very high error rate would raise some sort of internal red flags, one would think.
original rant text:
Just curious why Red Hat does not actively monitor that its CDN is delivering erratum. It is frustrating to consistently see Oracle and now Centos deliver errata faster than Red Hat RHEL RHN does.
Responses
Seems the broken update notice problem is back again today on the CDN. I don't believe it is related to fastrack channels this time. Disabling optional seems to clear the messages so probably the optional/base pair this time aren't in sync.
Bump. Updates to RHEL5 Server seems to have broken again.
The setroubleshoot security update from 2015-03-26 is missing as is everything subsequent to that:
https://rhn.redhat.com/errata/RHSA-2015-0729.html
Still no updates are coming through on RHN for RHEL5-Server. I'm not seeing anything after the last Firefox update on 2015-03-24
https://rhn.redhat.com/errata/rhel-server-errata.html
Tried 'yum clean all' but to no avail.
Is there anyone active here who can help or do I need to open yet another support case for this?
Hi Phillip,
I have looked into this with a few people and we are not seeing a systematic issue and are able to get updates. There may be something going on with the RHEL 5 channels that we did not uncover but have not seen anything yet and are not getting wide spread reports from others.
Would you mind opening a support case so that we can troubleshoot this further. You can ask to have me added to it for follow up.
Thomas Shea DeAntonio
We've had a good run here but all day (at least since the release of the new RHEL5 kernel today) I've been getting
# yum clean all; yum check-update
...
rhel-x86_64-server-5 | 1.4 kB 00:00
Error: failed to retrieve repodata/f27294a21e93212a8887b30c1177f6d32065210a-primary.xml.gz from rhel-x86_64-server-5
error was [Errno 14] HTTP Error 404: Not Found
on every machine talking directly to RHN hosted.
This CDN problem has returned.
# cat /etc/redhat-release
Red Hat Enterprise Linux Workstation release 7.1 (Maipo)
# yum clean all; yum check-update
Loaded plugins: langpacks, product-id, subscription-manager
Cleaning repos: rhel-7-workstation-fastrack-rpms
: rhel-7-workstation-optional-fastrack-rpms
: rhel-7-workstation-optional-rpms rhel-7-workstation-rpms
Cleaning up everything
Loaded plugins: langpacks, product-id, subscription-manager
rhel-7-workstation-fastrack-rpms | 2.9 kB 00:00
rhel-7-workstation-optional-fastrack-rpms | 2.9 kB 00:00
rhel-7-workstation-optional-rpms | 2.9 kB 00:00
rhel-7-workstation-rpms | 3.7 kB 00:00
(1/4): rhel-7-workstation-optional-fastrack-rpms/x86_64/pr | 52 kB 00:01
(2/4): rhel-7-workstation-fastrack-rpms/x86_64/primary_db | 87 kB 00:01
(3/4): rhel-7-workstation-optional-rpms/7Workstation/x86_6 | 2.2 MB 00:10
(4/4): rhel-7-workstation-rpms/7Workstation/x86_64/primary | 12 MB 00:37
(1/5): rhel-7-workstation-optional-fastrack-rpms/x86_64/up | 2.5 kB 00:01
(2/5): rhel-7-workstation-fastrack-rpms/x86_64/updateinfo | 3.2 kB 00:01
(3/5): rhel-7-workstation-rpms/7Workstation/x86_64/group_g | 137 kB 00:01
(4/5): rhel-7-workstation-optional-rpms/7Workstation/x86_6 | 388 kB 00:03
(5/5): rhel-7-workstation-rpms/7Workstation/x86_64/updatei | 569 kB 00:03
Update notice RHSA-2014:0679 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-rpms repository.
Update notice RHSA-2014:1327 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0372 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0335 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHEA-2015:0371 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0416 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0303 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0556 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0290 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0596 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:0578 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:0716 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
Update notice RHSA-2015:1115 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
#
# yum --disablerepo=rhel-7-workstation-optional-rpms check-update
Loaded plugins: langpacks, product-id, subscription-manager
#
I have been seeing similar for many months and, to be quite honest, have given up hoping that the system will ever be fixed. Here follows an example of what I see with the hundred or so (I haven't bothered to check the exact number) advisory lines grep-ed out --
[Duo1 ~]# yum check-update | grep -v ^Up
Loaded plugins: fastestmirror, product-id, subscription-manager
Loading mirror speeds from cached hostfile
You should report this problem to the owner of the rhel-7-server-fastrack-source-rpms repository.
You should report this problem to the owner of the rhel-7-server-optional-source-rpms repository.
You should report this problem to the owner of the rhel-7-server-rpms repository.
You should report this problem to the owner of the rhel-7-server-source-rpms repository.
[Duo1 ~]#
Any chance someone could look at this? It has been broken for days. The errata released today showed up hours ago for servers but the workstation repos are still broken and do not contain any of today's errata.
I know it is release day and sometimes we need to be patient for things to sort themselves out but I think there is a problem with the RHEL6 64-bit server primary channel on hosted. I can download around 125 rpms but then the update fails telling me that the download for libxml2 and openssl do not match the intended download. It suggested cleaning the metadata and trying again which I've down a few times with no change. Can someone take a look please.
Currently, all of my RHEL5 i386 Workstations are doing this:
[root@kettle ~]# yum clean all
Loaded plugins: rhnplugin
Cleaning up Everything
[root@kettle ~]# yum update
Loaded plugins: rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.
rhel-i386-client-5 | 1.4 kB 00:00
rhel-i386-client-5/primary | 4.9 MB 00:00
rhel-i386-client-5 10353/10353
rhel-i386-client-supplementary-5 | 1.4 kB 00:00
rhel-i386-client-supplementary-5/primary | 277 kB 00:00
rhel-i386-client-supplementary-5 944/944
rhel-i386-client-workstation-5 | 1.4 kB 00:00
rhel-i386-client-workstation-5/primary | 825 kB 00:00
rhel-i386-client-workstation-5/primary | 825 kB 00:00
rhel-i386-client-workstation-5/primary | 825 kB 00:00
rhel-i386-client-workstation-5/primary | 825 kB 00:00
Error: failed to retrieve repodata/89e31cdcf3aef1da587850a365a8b014d40c003a-primary.xml.gz from rhel-i386-client-workstation-5
error was [Errno -1] Metadata file does not match checksum
[root@kettle ~]#
All RHEL5 i386/x86_64 Servers and RHEL5 x86_64 Workstations are updating happily.
Anyone else having issues?
All the computers have active subscriptions with RHN. One, which had lost its child channel subscriptions owing to temporarily having been downgraded to "Update" rather than "Management" entitlement, was prepared to update until I readded the "RHEL Desktop Workstation (v. 5 for 32-bit x86)" child channel to depsolve some packages, at which point I got this:
Error: failed to retrieve repodata/00c88bd13aefb00b56c228e899f418a0ace9a077-filelists.xml.gz from rhel-i386-client-workstation-5
error was [Errno 14] HTTP Error 404: Not Found
Which is subtly different, but still indicates some kind of issue with this child channel.
Looks like it was indeed the CDN issue. According to this post and this note, it's been fixed by flushing the CDN cache:
"I could not find any issues with the repodata files on RHN itself and suspect an issue with CDN caching. I've flushed the cache for the rhel-i386-client-workstation-5 repodata URLs."
We are back with the broken updateinfo again since the firefox update ...
Update notice RHSA-2014:0679 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-rpms repository.
Can someone please kick the rhel-7-workstation-rpms channel on the CDN? Metadata is about 20 hours old now and the pam security update is not getting through. Thanks.
I believe I got the openldap update yesterday but the updateinfo metadata is broken as well on both server and workstation.
Update notice RHSA-2014:0679 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-rpms repository.
Is there anything more we can do to help? We have been reporting these problems over and over in this ticket for more than a year now and while at times it has seemed better it still keeps breaking and Red Hat continues to not seem to notice until someone from outside points it out. I know I would be willing to help if I could so I didn't have to worry so much about updates not being available.
updateinfo is still a mess on the CDN for rhel-7 --- maybe yum just shouldn't print out all the broken or a bad duplicate messages so we don't know it is broken? Not ideal but at least we don't have to wade through all of them to find actual updates. Or I suppose it could be fixed on the CDN so it isn't broken too.
updateinfo still broken on rhel-7-server. tzdata still missing on rhel-5-server and rhel-6-server.
And the most amazing of all things today is that following yum clean all; yum check-update I was subscribed to the rhel-sap-for-rhel-{5,6,7}-server-rpms channel! Seriously? WTF?
Another week and we still have broken updateinfo. I'm giving up on this ever working now. Really I am. We'll just pretend it isn't broken like our upstream does from now on.
# cat /etc/redhat-release
Red Hat Enterprise Linux Workstation release 7.1 (Maipo)
# yum clean all; yum check-update
Loaded plugins: langpacks, product-id, subscription-manager
...
Update notice RHSA-2014:0675 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-rpms repository.
Update notice RHSA-2014:0686 (from rhel-7-workstation-rpms) is broken, or a bad duplicate, skipping.
followed by 440 more broken errata.
# yum updateinfo list
... broken update messages deleted again ...
RHSA-2015:1920 Critical/Sec. java-1.7.0-openjdk-1:1.7.0.91-2.6.2.2.el6_7.x86_64
RHEA-2015:0518 enhancement linux-firmware-1:20140911-0.1.git365e80c.el7.noarch
RHSA-2015:1840 Important/Sec. openldap-2.4.40-6.el6_7.x86_64
Ah, three errata, two of them security errata for packages missing from the repo.
Would someone please help fix this?
Thanks David. I looked into the stray errata a bit more tonight. Here is how things look now.
# yum clean all; yum --disablerepo=epel updateinfo list
Loaded plugins: langpacks, product-id, subscription-manager
Cleaning repos: epel rhel-7-workstation-fastrack-rpms
: rhel-7-workstation-optional-fastrack-rpms
: rhel-7-workstation-optional-rpms rhel-7-workstation-rpms
Cleaning up everything
Loaded plugins: langpacks, product-id, subscription-manager
rhel-7-workstation-fastrack-rpms/x86_64 | 3.8 kB 00:00
rhel-7-workstation-fastrack-rpms/x86_64/group | 104 B 00:00
rhel-7-workstation-fastrack-rpms/x86_64/updateinfo | 31 kB 00:00
rhel-7-workstation-fastrack-rpms/x86_64/primary_db | 137 kB 00:00
rhel-7-workstation-optional-fastrack-rpms/x86_64 | 2.9 kB 00:00
rhel-7-workstation-optional-fastrack-rpms/x86_64/updatei | 8.2 kB 00:00
rhel-7-workstation-optional-fastrack-rpms/x86_64/primary | 67 kB 00:00
Update notice RHBA-2015:1014 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-workstation-optional-fastrack-rpms repository.
Update notice RHBA-2015:1079 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1116 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1131 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1506 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1504 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1573 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1598 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1702 (from rhel-7-workstation-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
rhel-7-workstation-optional-rpms/7Workstation/x86_64 | 3.8 kB 00:00
rhel-7-workstation-optional-rpms/7Workstation/x86_64/gro | 104 B 00:00
https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/optional/os/repodata/88ee5746561d100f57ad06106bebafb5261cfc8c839f8f256ea047cdd9a48715-updateinfo.xml.gz: [Errno 14] curl#56 - "SSL received a record with an incorrect Message Authentication Code."
Trying other mirror.
rhel-7-workstation-optional-rpms/7Workstation/x86_64/upd | 1.4 MB 00:00
rhel-7-workstation-rpms/7Workstation/x86_64 | 4.0 kB 00:00
rhel-7-workstation-rpms/7Workstation/x86_64/group | 898 kB 00:00
rhel-7-workstation-rpms/7Workstation/x86_64/updateinfo | 1.7 MB 00:00
rhel-7-workstation-rpms/7Workstation/x86_64/primary_db | 18 MB 00:08
RHSA-2015:1920 Critical/Sec. java-1.7.0-openjdk-1:1.7.0.91-2.6.2.2.el6_7.x86_64
RHEA-2015:0518 enhancement linux-firmware-1:20140911-0.1.git365e80c.el7.noarch
RHSA-2015:1840 Important/Sec. openldap-2.4.40-6.el6_7.x86_64
RHEA-2015:1863 enhancement tzdata-2015g-2.el6.noarch
RHEA-2015:1863 enhancement tzdata-java-2015g-2.el6.noarch
updateinfo list done
# rpm -q java-1.7.0-openjdk linux-firmware openldap tzdata tzdata-java
java-1.7.0-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64
linux-firmware-20140911-0.1.git365e80c.el7.noarch
openldap-2.4.39-7.el7_1.x86_64
tzdata-2015g-1.el7.noarch
tzdata-java-2015g-1.el7.noarch
So curiously I actually have the current versions of all of those installed. For java-1.7.0-openjdk, openldap, tzdata, and tzdata-java strangely the errata lists packages for RHEL 6 rather than for RHEL 7. I have no explanation currently for why linux-firmware shows up since it appears to me I have the exact version listed in the errata installed.
Daryl there has actually been some things identified and progress made on the issue you reported on that case but there is still more work to be done. It also appears the updates from the bug were not making it into the case. I have asked for the information to be included in your case and will try to have the bug open if possible.
email me directly if you all need to shea@redhat.com
I've noticed this issue many times before, as well; and, you're right, a lot of it has improved. However, at the moment I'm seeing issues with rhel-7-server-optional-fastrack-rpms. The pertinent repomd.xml is dated Sep 2 and the following warnings/errors:
Skipping filters plugin, no data
Update notice RHBA-2015:1014 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-server-optional-fastrack-rpms repository.
Update notice RHBA-2015:1079 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1116 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1131 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1506 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1504 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1573 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1598 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1702 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
I, too, have been suffering from this problem with the CDN ever since the initial release of RHEL-7.
Here follows the latest example --
[~]# yum check-update
Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager
elrepo
elrepo-kernel
rhel-7-server-fastrack-rpms
rhel-7-server-fastrack-source-rpms
rhel-7-server-optional-fastrack-rpms
rhel-7-server-optional-fastrack-source-rpms
rhel-7-server-optional-rpms
rhel-7-server-optional-source-rpms
rhel-7-server-rpms
rhel-7-server-source-rpms
Determining fastest mirrors
* elrepo: mirrors.coreix.net
* elrepo-kernel: mirrors.coreix.net
Update notice RHBA-2015:1014 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-server-optional-fastrack-rpms repository.
Update notice RHBA-2015:1079 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1116 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1131 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1506 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1504 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1573 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1598 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1702 (from rhel-7-server-optional-fastrack-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1595 (from rhel-7-server-optional-fastrack-source-rpms) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the rhel-7-server-optional-fastrack-source-rpms repository.
Update notice RHBA-2015:1601 (from rhel-7-server-optional-fastrack-source-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1547 (from rhel-7-server-optional-fastrack-source-rpms) is broken, or a bad duplicate, skipping.
Update notice RHBA-2015:1690 (from rhel-7-server-optional-fastrack-source-rpms) is broken, or a bad duplicate, skipping.
[~]#
For at least the last few days I've been getting a "No more mirrors to try" error when trying to download repodata/productid from the RHEL 7Server RHUI channel - a colleague is getting the same with the normal RHEL 7Server channel, and doesn't appear to have downloaded any updates since Nov 19.
Caught exception when trying to fetch metadata file productid from [https://cdn.redhat.com/content/dist/rhel/rhui/server/7/7Server/x86_64/os]: failure: repodata/productid from : [Errno 256] No more mirrors to try.
I've just logged another support ticket, but hopefully someone from Red Hat monitoring this thread might be able to kick things into life again...
The libxml2 security update still hasn't reached the CDN for at least some repos. Just checked el7 Workstation, as of last night it wasn't available in the el7 or el6 Server repos. It is available from both CentOS and Oracle in public repos, sigh.
John this is being addressed and hope to have resolution soon. I also want to let you and others know that we are building a dedicated cross functional task force that will be focused on performance, reliability and consistency with our delivery of updates.
I will be able to share more information on this program and determine the best way to keep you and others updated on our progress in the New Year.
Shea
Critical: samba and samba4 updates (badlock fixes) were announced/released more than 24 hours ago. I still do not see them on my RHEL-6 systems. The rhel-6-server-rpms/repomd.xml file is dated Mar 25. yum cache has been cleaned a few times by now. Could someone please look into the issue?
Hello Daryl. We created a team to focus on the problem and identified numerous areas of focus.
This is the current status.
--Upgrading systems to improve the infrastructure and help with reducing push times. This upgrade was completed at the beginning of the month.
-- Reviewed the push process end to end and identified a number of steps that could be improved on for specific use cases. This is ongoing, some of it has been completed and the rest of it needs to be tested and validated before being released.
-- Increased repodata validation by performing multiple checks against systems to verify correctness. This is under development.
-- Remove coupling we have between RHN and RHSM to improve time to delivery and help with consistency. This is ongoing and hope to have a lot of it completed by the end of the month.
I am hopeful that when we get more of the things currently in development released we will see less problems, but this is not the end and we will continue to identify and improve on things until we hit goals set for speed, accuracy and consistency.
Hi everyone.
Is something going to change with "Update notice is broken?" I have each and every RHEL7 system reporting issues with rhel-* repositories nightly via yum-cron.
Could someone take a look at this one. RHEL 7 Workstation broken again on the CDN.
Downloading packages:
No Presto metadata available for rhel-7-workstation-rpms
firefox-45.6.0-1.el7_3.x86_64. FAILED
https://cdn.redhat.com/content/dist/rhel/workstation/7/7Workstation/x86_64/os/Packages/firefox-45.6.0-1.el7_3.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
...
Error downloading packages:
firefox-45.6.0-1.el7_3.x86_64: [Errno 256] No more mirrors to try.
I'm seeing this on multiple machines for hours now. Thanks.
Pages
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
