Troubleshooting '[Errno 14] HTTPS Error 404 - Not Found' error observed on Red Hat Satellite 6 clients

Issue

When trying to install/update packages using yum getting below error on Client registered to Red Hat Satellite 6 server:

https://satellite.example.com/pulp/repos/test/Library/content/dist/rhel/server/6/6Server/x86_64/os/repodata /repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

Overview

Generally, the [Errno 14] HTTPS Error 404 - Not Found error is observed due to inconsistent metadata of the repositories or if the client could not find or access the requested package on the server. The troubleshooting steps which need to be followed to resolve this issue vary in different scenarios and also there are multiple resolutions available for it. This article lists different Solutions describing when this issue occurs and how to get it resolved.

There are multiple parameters that you need to check before following any Knowledge Solution like whether there is a Content View associated with the client or Default Organization View, whether the client is registered through Capsule or directly to Satellite, release version set on the client and many more.

Known Issues and Resolution

Verify if correct subscription-manager release is set on clients.

• If the client has been limited to a specific major or minor release, verify that it is using the correct one.

  • What release is the client using?

    # cat /etc/redhat-release
    

  • What release is the client subscribed/limited to?

    # subscription-manager release --show 
    

• If these do not match, you will need to correct the mismatch using the subscription-manager release command and passing it the appropriate release identifier. This process is described in article.

Check if /var utilization is full on Satellite/Capsule.

• In some situations the /var utilization gets full on the Satellite server as per the below output due to which the repository synchronization doesn't get completed resulting in 404 not found errors on clients.

  [root@satellite ~]# df -h /var/lib/pulp
  Filesystem                   Size  Used Avail Use% Mounted on
  /dev/mapper/vg00-pulp        1.1T  1.1T   22K  100% /var/lib/pulp
  

• To extend a logical volume and its filesystem in Red Hat Enterprise Linux refer to article : /var utilization is full on Satellite/Capsule 6.

If a "%24releasever" character is observed in repomd.xml URL when performing `yum repolist` then set the correct release version in yum.

• Check if you see "%24releasever" character in the error observed as per the following example:

  https://satellite.example.com/pulp/repos/XYZ/Library/rhel-cv/content/dist/rhel/server/7/%24releasever/x86_64/os/repodata/repomd.xml: [Errno 14]   HTTPS Error 404 - Not Found Trying other mirror.
  

  • If yes, check if there is any pre-defined variable set in /etc/yum/vars/releasever file:

    # cat /etc/yum/vars/releasever
    7.5
    

• The unwanted "%24releasever" character set in URL can be removed by configuring the correct release in yum as mentioned in the article.

For clients associated with Default Organization View.

• Verify if the client is associated with a Default Organization View be executing below command:

  # subscription-manager identity
  environment name: Library               <--- here no CV name is reflected with the Library environment which means this host is associated with Default Organization View
  

• This issue may occur due to corrupt metadata and can be resolved by following the steps mentioned in article.

For clients associated with Content View.

• Verify if the client is associated with a Content View be executing below command:

  # subscription-manager identity
  environment name: Prod/RHEL7-CV                                 <---     here CV name will be reflected after the Environment name 
  

• To resolve this issue, it is required to publish/promote the Content View with a new version. Please refer to the article for detailed steps.

For clients associated with newly created Content View.

• Verify if the Organization label is changed.

  • Run below command on Satellite to check the current organization label:

    # hammer organization list
    

  • Run below command on affected client with newly created content view to check relative path of content view:

    # yum repolist -v
    

  • Organization label on the relative path of newly created content view will be different from the current organization label of Satellite in above command output.

• To resolve this issue, Rollback the organization name and re-create the new content view.

• Alternatively create a new organization and delete the current one if possible.

  • Create new organization on the Satellite by following the steps mentioned here.
  • For deleting the organization on the Satellite refer the steps mentioned here.

Facing 404 error even if the package is in the metadata

• In some scenarios you may see that the Metadata contains proper information about the package however the filename is a broken symlink:

  # file gofer-2.7.6-1.el7sat.noarch.rpm 
  gofer-2.7.6-1.el7sat.noarch.rpm: broken symbolic link to `/var/lib/pulp/content/units/rpm/01        /d91ab1958314d44ce81038015cb83909aa7d968eb26e82c2791da9e21e5d7c/gofer-2.7.6-1.el7sat.noarch.rpm
  

• To fix the broken symlinks issue or re-generate the repository metadata on Satellite/Capsule please refer to the article

While using Load Balancing Capsule setup:

• Run below command on the client to check the stickiness is configured on port 443 or not.

  # while true ; do echo "$(date) $(curl -vv https://loadbalancer.example.com  2>&1 | grep "subject:")" ; sleep 5 ; done
  

• Sticky session has to be configured on TCP port 443 to request yum metadata for RPM repositories from different Capsule Servers that are configured for Load Balancing. Please check article for more details.

The baseurl value in RHSM configuration is altered and not correct

• Ensure the baseurl value in /etc/rhsm/rhsm.conf on content host registered with Red Hat Satellite is correctly set.

  # grep -e baseurl /etc/rhsm/rhsm.conf
  baseurl = https://satellite.example.com/pulp/repos
  

  Note: The satellite.example.com above should be the FQDN of Satellite or Capsule server to which the content host is registered.