Troubleshooting '[Errno 14] HTTPS Error 404 - Not Found' error observed on Red Hat Satellite 6 clients

Updated -

Issue

When trying to install/update packages using yum getting below error on Client registered to Red Hat Satellite 6 server:

https://satellite.example.com/pulp/repos/test/Library/content/dist/rhel/server/6/6Server/x86_64/os/repodata /repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

Overview

Generally, the [Errno 14] HTTPS Error 404 - Not Found error is observed due to inconsistent metadata of the repositories or if the client could not find or access the requested package on the server. The troubleshooting steps which need to be followed to resolve this issue vary in different scenarios and also there are multiple resolutions available for it. This article lists different Solutions describing when this issue occurs and how to get it resolved.

There are multiple parameters that you need to check before following any Knowledge Solution like whether there is a Content View associated with the client or Default Organization View, whether the client is registered through Capsule or directly to Satellite, release version set on the client and many more.

Known Issues and Resolution

Verify if correct subscription-manager release is set on clients.

  • If the client has been limited to a specific major or minor release, verify that it is using the correct one.

    • What release is the client using?

      # cat /etc/redhat-release
      
    • What release is the client subscribed/limited to?

      # subscription-manager release --show 
      
  • If these do not match, you will need to correct the mismatch using the subscription-manager release command and passing it the appropriate release identifier. This process is described in article.

Check if /var utilization is full on Satellite/Capsule.

  • In some situations the /var utilization gets full on the Satellite server as per the below output due to which the repository synchronization doesn't get completed resulting in 404 not found errors on clients.

    [root@satellite ~]# df -h /var/lib/pulp
    Filesystem                   Size  Used Avail Use% Mounted on
    /dev/mapper/vg00-pulp        1.1T  1.1T   22K  100% /var/lib/pulp
    
  • To extend a logical volume and its filesystem in Red Hat Enterprise Linux refer to article : /var utilization is full on Satellite/Capsule 6.

If a "%24releasever" character is observed in repomd.xml URL when performing yum repolist then set the correct release version in yum.

  • Check if you see "%24releasever" character in the error observed as per the following example:

    https://satellite.example.com/pulp/repos/XYZ/Library/rhel-cv/content/dist/rhel/server/7/%24releasever/x86_64/os/repodata/repomd.xml: [Errno 14]   HTTPS Error 404 - Not Found Trying other mirror.
    
    • If yes, check if there is any pre-defined variable set in /etc/yum/vars/releasever file:

      # cat /etc/yum/vars/releasever
      7.5
      
  • The unwanted "%24releasever" character set in URL can be removed by configuring the correct release in yum as mentioned in the article.

For clients associated with Default Organization View.

  • Verify if the client is associated with a Default Organization View be executing below command:

    # subscription-manager identity
    environment name: Library               <--- here no CV name is reflected with the Library environment which means this host is associated with Default Organization View
    
  • This issue may occur due to corrupt metadata and can be resolved by following the steps mentioned in article.

For clients associated with Content View.

  • Verify if the client is associated with a Content View be executing below command:

    # subscription-manager identity
    environment name: Prod/RHEL7-CV                                 <---     here CV name will be reflected after the Environment name 
    
  • To resolve this issue, it is required to publish/promote the Content View with a new version. Please refer to the article for detailed steps.

Facing 404 error even if the package is in the metadata

  • In some scenarios you may see that the Metadata contains proper information about the package however the filename is a broken symlink:

    # file gofer-2.7.6-1.el7sat.noarch.rpm 
    gofer-2.7.6-1.el7sat.noarch.rpm: broken symbolic link to `/var/lib/pulp/content/units/rpm/01        /d91ab1958314d44ce81038015cb83909aa7d968eb26e82c2791da9e21e5d7c/gofer-2.7.6-1.el7sat.noarch.rpm
    
  • To fix the broken symlinks issue or re-generate the repository metadata on Satellite/Capsule please refer to the article

While using Load Balancing Capsule setup:

  • Run below command on the client to check the stickiness is configured on port 443 or not.

    # while true ; do echo "$(date) $(curl -vv https://loadbalancer.example.com  2>&1 | grep "subject:")" ; sleep 5 ; done
    
  • Sticky session has to be configured on TCP port 443 to request yum metadata for RPM repositories from different Capsule Servers that are configured for Load Balancing. Please check article for more details.