Satellite 6.6 on-demand packages failing to download

Latest response

hi all,

satellite 6.6.3
test server rhel 8.1

I am testing RH Satellite 6.6 and configured 3x repositories:
rhel 8 baseos
rhel 8 appstream
satellite 6.6. tools

I am trying to install a katello-agent on a rhel8.1 test server to be managed by satellite; but I am getting 404 code. (in fact any package I try I get the same error message

My Satellite 'Default Repository download policy is set to "on-demand"' therefore it will pull file from rh cdn 'upon request'

On my Satellite server, therefore, I can not see the dir /var/lib/pulp/content/units/rpm being created ....

I would like to understand how to tshoot this issue, which seems to be between my Satellite server and rh cdn.

Comments, suggestions appreciated.

rg
Log in to join the conversation

Responses

Pavel Moravec's picture Red Hat Guru 5128 points
4 October 2020 2:38 PM

Hello,

/var/lib/pulp/content/units/rpm would get the package even after it is downloaded "on demand". Once the request comes from a client, you should see in /var/log/httpd/foreman-ssl_access_ssl.log logs like:

1.2.3.4 - - [04/Oct/2020:16:26:30 +0200] "GET /streamer/var/lib/pulp/content/units/rpm/2a/f09784333edd6f37518190677ee2e76854d2d8d96826f9cd048742a2818fe0/sos-3.8-6.el7.noarch.rpm?policy=........%3D%3D HTTP/1.1" 503 299 "-" "urlgrabber/3.10 yum/3.4.3"
1.2.3.4 - - [04/Oct/2020:16:26:30 +0200] "GET /pulp/repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/os/Packages/s/sos-3.8-6.el7.noarch.rpm HTTP/1.1" 302 - "-" "urlgrabber/3.10 yum/3.4.3"

where the 302 return code means redirect to squid. I suspect there is some issue with this service (though it usually means 503 response, not 404..), so worth checking squid and e.g. /var/log/squid/access.log.

The 404 might suggest wrongly published pulp repository with repo metadata pointing the package to a wrong location. Here it is worth checking:

cd /var/lib/pulp/published/yum/https/
# copy below argument from the above httpd log entry
file repos/RedHat/Library/content/dist/rhel/server/7/7Server/x86_64/os/Packages/s/sos-3.8-6.el7.noarch.rpm

The file should be broken symbolic link to /var/lib/pulp/content/units/rpm/ - which is fine. The destination will be filled by squid on the first package request.

If the file does not exist, then the client attempts to fetch something outside the repo - worth checking what repo it is, where it is published and if the published dir has proper content.

rg Community Member 70 points
5 October 2020 8:53 AM

hi pavel,

thank you for this comprehensive reply. I followed your steps and everything seems to return as "expected" (not working I mean):

/var/log/httpd/foreman-ssl_access_ssl.log seems to produce 302 (re-directing to squid??) then 404

192.168.X.X - - [05/Oct/2020:09:11:14 +0100] "GET /rhsm/consumers/5ca6d402-64f0-45b6-b7c6-704d8d95abb5/certificates/serials HTTP/1.1" 200 32 "-" "RHSM/1.0 (cmd=yum)"
192.168.X.X - - [05/Oct/2020:09:11:15 +0100] "GET /rhsm/status HTTP/1.1" 200 377 "-" "RHSM/1.0 (cmd=yum)"

192.168.X.X - - [05/Oct/2020:09:11:15 +0100] "GET /pulp/repos/Operations/Development/Base/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml HTTP/1.1" 200 2425 "-" "libdnf"
192.168.X.X - - [05/Oct/2020:09:11:15 +0100] "GET /pulp/repos/Operations/Development/Base/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml HTTP/1.1" 200 2851 "-" "libdnf"
192.168.X.X - - [05/Oct/2020:09:11:15 +0100] "GET /pulp/repos/Operations/Development/Base/content/dist/layered/rhel8/x86_64/sat-tools/6.6/os/repodata/repomd.xml HTTP/1.1" 200 2157 "-" "libdnf"
192.168.X.X - - [05/Oct/2020:09:11:19 +0100] "GET /pulp/repos/Operations/Development/Base/content/dist/rhel8/8/x86_64/appstream/os/Packages/p/python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm HTTP/1.1" 302 - "-" "libdnf"
192.168.X.X - - [05/Oct/2020:09:11:19 +0100] "GET

192.168.X.X - - [05/Oct/2020:09:11:19 +0100] "GET /streamer/var/lib/pulp/content/units/rpm/9d/27b16e22fbecea6d759bd14708bf2bb8597b72d7d515c5450c8d507c862102/python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm?policy=eyJleHRlbnNpb25zIjogeyJyZW1vdGVfaXAiOiAiMTkyLjE2OC4xMjIuMTc4In0sICJyZXNvdXJjZSI6ICIvc3RyZWFtZXIvdmFyL2xpYi9wdWxwL2NvbnRlbnQvdW5pdHMvcnBtLzlkLzI3YjE2ZTIyZmJlY2VhNmQ3NTliZDE0NzA4YmYyYmI4NTk3YjcyZDdkNTE1YzU0NTBjOGQ1MDdjODYyMTAyL3B5dGhvbjM2LTMuNi44LTIubW9kdWxlK2VsOC4xLjArMzMzNCs1Y2I2MjNkNy54ODZfNjQucnBtIiwgImV4cGlyYXRpb24iOiAxNjAxODg1NTY5fQ%3D%3D;signature=o5MC8PaVocACoVOkFIuZv5GQL5mPuAv_Y0uIJq7HXHJIIl6k6mh_tG34z0-4TXdYGRdyF5GaNMVW-utXgr3hKxvG2Ii_p_kaQpwm9GTYVdUvIZsNApcmpwpzjvfkM8zaSyKZZb83Sv0wAInIqZrKLdBEjiaLLoMVOgqikaB9kC_MQEHJkKNF1XNZf7t1ySkUFUVmm7x66yoN66CbnZbPIFAI2Oo1WXb7HBYBu6pect6bRLcIgf_MCOH6HsQ_HWAZqOAZ0Tz9raxJjz3_b215K1r6L4gNeaS8UcDHI0ZTA-yhZ-FuUbzVyzusBCsJFQkQwXkt7YoOELumWkvZNtdUKw%3D%3D HTTP/1.1" 404 - "-" "libdnf"

Then within /var/log.squid/access.log

1601737320.920    358 127.0.0.1 TCP_MISS/404 563 GET http://127.0.0.1:3128/var/lib/pulp/content/units/rpm/9d/27b16e22fbecea6d759bd14708bf2bb8597b72d7d515c5450c8d507c862102/python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm - FIRSTUP_PARENT/127.0.0.1 text/html

Finally looking at /var/lib/pulp/published/yum/https/ I can indeed see the file with a broken symbolic link:

file python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm 
python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm: broken symbolic link to `/var/lib/pulp/content/units/rpm/9d/27b16e22fbecea6d759bd14708bf2bb8597b72d7d515c5450c8d507c862102/python36-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64.rpm'

I will continue to tshoot this, but if you think there is anything you could add, that would be appreciated.

Thank you.

*** Fixed by re-applying a manifest file ***

Pavel Moravec's picture Red Hat Guru 5128 points
23 October 2020 7:45 PM

Ah right, if manifest is outdated / subscription authorizing you to fetch that content from CDN is not valid, then this can happen. Good catch.