Troubleshooting '[Errno 14] HTTPS Error 404 - Not Found' error observed on Red Hat Satellite 6 clients

Updated -

Table of Contents

Issue

  • When trying to install/update packages using yum getting below error on Client registered to Red Hat Satellite 6 server:

    https://satellite.example.com/pulp/repos/test/Library/content/dist/rhel/server/6/6Server/x86_64/os/repodata /repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

    OR

    # yum update
...
Trying other mirror.

mpfr-3.1.1-4.el7_9.3.x86_64.rp FAILED 
https://satellite.example.com/pulp/content/satellite/Library/RHEL7_Content-view/content/els/rhel/server/7/7Server/x86_64/os/Packages/m/mpfr-3.1.1-4.el7_9.3.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Overview

Generally, the [Errno 14] HTTPS Error 404 - Not Found error is observed due to inconsistent metadata of the repositories or if the client could not find or access the requested package on the server. The troubleshooting steps which need to be followed to resolve this issue vary in different scenarios and also there are multiple resolutions available for it. This article lists different Solutions describing when this issue occurs and how to get it resolved.

There are multiple parameters that you need to check before following any Knowledge Solution like whether there is a Content View associated with the client or Default Organization View, whether the client is registered through Capsule or directly to Satellite, release version set on the client and many more.

Known Issues and Resolution

Verify if correct subscription-manager release is set on clients.

  • If the client has been limited to a specific major or minor release, verify that it is using the correct one.

    • What release is the client using?

      # cat /etc/redhat-release

    • What release is the client subscribed/limited to?

      # subscription-manager release --show

  • If these do not match, set the correct release version following the steps outlined in article 1421903.

Check if /var utilization is full on Red Hat Satellite/Capsule.

  • In some situations, the /var utilization gets full on the Satellite server as per the below output due to which the repository synchronization doesn't get completed resulting in 404 not found errors on clients.

    [root@satellite ~]# df -h /var/lib/pulp
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg00-pulp        1.1T  1.1T   22K  100% /var/lib/pulp

  • To extend a logical volume and its filesystem in Red Hat Enterprise Linux refer to article : /var utilization is full on Satellite/Capsule 6.

Verify if the required subscription is removed from the Red Hat Satellite 6 manifest file.

  • Ensure that the subscription required to fulfil the Red Hat Enterprise Linux Server product is present in the Red Hat Satellite 6 manifest file.

    In some scenarios, the subscription gets expired and removed from the manifest file, which causes the product and its belonging repository orphan and failure of yum transaction with error [Errno 14] HTTPS Error 404 - Not Found.

    • The following command can be executed in Red Hat Satellite to validate the attached subscriptions:

      <br><br>\# hammer organization list                          /// Note down the organization ID
# hammer subscription list --organization-id=X               /// replace X with the organization ID

    • Check if the Sync Status page shows that the affected product repository is orphan, refer to the resolution steps from article 3423341.

      **Satellite WebUI →  Content → Sync Status

If a "%24releasever" character is observed in repomd.xml URL when performing yum repolist then set the correct release version in yum.

  • Check if you see "%24releasever" character in the error observed as per the following example:

    https://satellite.example.com/pulp/repos/XYZ/Library/rhel-cv/content/dist/rhel/server/7/%24releasever/x86_64/os/repodata/repomd.xml: [Errno 14]   HTTPS Error 404 - Not Found Trying other mirror.

    • If yes, check if there is any pre-defined variable set in /etc/yum/vars/releasever file:

      # cat /etc/yum/vars/releasever
7.5

  • The unwanted "%24releasever" character set in URL can be removed by configuring the correct release in yum as mentioned in the article.

For clients associated with Default Organization View.

  • Verify if the client is associated with a Default Organization View be executing below command:

    # subscription-manager identity
environment name: Library               <--- here no CV name is reflected with the Library environment which means this host is associated with Default Organization View

  • This issue may occur due to corrupt metadata and can be resolved by following the steps mentioned in article.

For clients associated with Content View.

  • Verify if the client is associated with a Content View be executing below command:

    # subscription-manager identity
environment name: Prod/RHEL7-CV                                 <---     here CV name will be reflected after the Environment name

  • To resolve this issue, it is required to publish/promote the Content View with a new version. Please refer to the article for detailed steps.

For clients associated with newly created Content View.

  • Verify if the Organization label is changed.

    • Run below command on Satellite to check the current organization label:

      <br><br>\# hammer organization list

    • Execute the below command on affected client with newly created content view to check the relative path of content view:

       <br><br>\# yum repolist -v

    • Organization label on the relative path of newly created content view will be different from the current organization label of Red Hat Satellite in above command output.

  • To resolve this issue, Rollback the organization name and re-create the new content view.

  • Alternatively, create a new organization and delete the current one if possible.

    • Create new organization on the Satellite by following the steps mentioned here.
    • For deleting the organization on the Satellite refer the steps mentioned here.

Facing 404 error even if the package is in the metadata

  • In some scenarios you may see that the Metadata contains proper information about the package however the filename is a broken symlink:

    # file gofer-2.7.6-1.el7sat.noarch.rpm 
gofer-2.7.6-1.el7sat.noarch.rpm: broken symbolic link to `/var/lib/pulp/content/units/rpm/01        /d91ab1958314d44ce81038015cb83909aa7d968eb26e82c2791da9e21e5d7c/gofer-2.7.6-1.el7sat.noarch.rpm

  • To fix the broken symlinks issue or re-generate the repository metadata on Satellite/Capsule please refer to the article

While using Load Balancing Capsule setup:

  • Run below command on the client to check if stickiness is configured on port 443 or not.

    # while true ; do echo "$(date) $(curl -vv https://loadbalancer.example.com  2>&1 | grep "subject:")" ; sleep 5 ; done

  • Sticky session has to be configured on TCP port 443 to request yum metadata for RPM repositories from different Capsule Servers that are configured for Load Balancing. Please check article for more details.

The baseurl value in RHSM configuration is altered and not correct

  • Ensure the baseurl value in /etc/rhsm/rhsm.conf on content host registered with Red Hat Satellite is correctly set.

    # grep -e baseurl /etc/rhsm/rhsm.conf
baseurl = https://satellite.example.com/pulp/repos

    Note: The satellite.example.com above should be the FQDN of Satellite or Capsule server to which the content host is registered.

Comments