Repositories in Satellite are 48% complete, then saying 'finished' but nothing downloaded

Latest response

I have setup RHEL7.7+yum update+latest Satellite to the date.
We have an http proxy in our environment, Satellite is setup to operate through it.

In Satellite web GUI whatever repo is selected for syncing, it stucks at 48% complete on 'New packages [number][xGB]', then 'Syncing complete', 100% complete, Result:success... but nothing in /var/lib/pulp/content/units/ in reality, only yum_repo_metadata_file

Attachments

Responses

More than that, I observe full folders structure created in /var/lib/pulp/published/yum/, even metadata is there, but in the end of the tree for *.rpm there are only broken links to /var/lib/pulp/content/units/rpm/....rpm, because there is no such folder /var/lib/pulp/content/units/rpm

Here is some logs. It downloads metadata but for some reason cannot download the rpms.. and still reports false-positive success!

Sep 11 21:15:10 hostname.domain.com pulp[24689]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 11 21:15:11 hostname.domain.com pulp[24689]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 11 21:15:11 hostname.domain.com pulp[21757]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[3e81fec6-94c6-4f73-9e34-660da74b86f4]
Sep 11 21:15:11 hostname.domain.com pulp[21608]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[7d827b7c-ef6e-4d16-b4a3-e018172c4ed5]
Sep 11 21:15:11 hostname.domain.com pulp[22178]: celery.app.trace:INFO: [3e81fec6] Task pulp.server.async.tasks._queue_reserved_task[3e81fec6-94c6-4f73-9e34-660da74b86f4] succeeded in 0.0192936480003s: None
Sep 11 21:15:11 hostname.domain.com pulp[21608]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[29758f63-cd89-4b41-9fcb-23fd63d9efa0]
Sep 11 21:15:12 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/.
Sep 11 21:15:13 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/repomd.xml.
Sep 11 21:15:14 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Parsing metadata.
Sep 11 21:15:14 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/.
Sep 11 21:15:15 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/repomd.xml.
Sep 11 21:15:16 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Parsing metadata.
Sep 11 21:15:16 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/.
Sep 11 21:15:17 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/repomd.xml.
Sep 11 21:15:18 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Parsing metadata.
Sep 11 21:15:18 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading metadata from https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/.
Sep 11 21:15:19 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/repomd.xml.
Sep 11 21:15:20 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Parsing metadata.
Sep 11 21:15:20 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading metadata files.
Sep 11 21:15:27 hostname.domain.com pulp[21505]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Sep 11 21:15:27 hostname.domain.com pulp[21620]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[43a9a0b4-c8bf-42e8-a8b6-1bdf84a578f7]
Sep 11 21:15:27 hostname.domain.com pulp[22151]: kombu.transport.qpid:INFO: [43a9a0b4] Connected to qpid with SASL mechanism ANONYMOUS
Sep 11 21:15:27 hostname.domain.com pulp[21614]: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.download_deferred[06322b44-097f-493a-acbe-65cadd9fe497]
Sep 11 21:15:27 hostname.domain.com pulp[22151]: py.warnings:WARNING: [43a9a0b4] (22151-24640) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=Fa
Sep 11 21:15:27 hostname.domain.com pulp[22151]: py.warnings:WARNING: [43a9a0b4] (22151-24640)   "MongoClient opened before fork. Create MongoClient "
Sep 11 21:15:27 hostname.domain.com pulp[22151]: py.warnings:WARNING: [43a9a0b4] (22151-24640)
Sep 11 21:15:27 hostname.domain.com pulp[22158]: py.warnings:WARNING: [06322b44] (22158-98400) /usr/lib64/python2.7/site-packages/pymongo/topology.py:74: UserWarning: MongoClient opened before fork. Create MongoClient with connect=Fa
Sep 11 21:15:27 hostname.domain.com pulp[22158]: py.warnings:WARNING: [06322b44] (22158-98400)   "MongoClient opened before fork. Create MongoClient "
Sep 11 21:15:27 hostname.domain.com pulp[22158]: py.warnings:WARNING: [06322b44] (22158-98400)
Sep 11 21:15:27 hostname.domain.com pulp[22151]: celery.app.trace:INFO: [43a9a0b4] Task pulp.server.controllers.repository.queue_download_deferred[43a9a0b4-c8bf-42e8-a8b6-1bdf84a578f7] succeeded in 0.0502251859998s: None
Sep 11 21:15:28 hostname.domain.com pulp[22158]: celery.app.trace:INFO: [06322b44] Task pulp.server.controllers.repository.download_deferred[06322b44-097f-493a-acbe-65cadd9fe497] succeeded in 1.02176455s: None
Sep 11 21:15:45 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6b
Sep 11 21:15:47 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/05bdff5d420ef6463df57bf17fda18c153efc98ef
Sep 11 21:15:47 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/eb6cb5ebcd095094b88bec99ab6557daa2c0dc6a3
Sep 11 21:15:48 hostname.domain.com pulp[22134]: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4, connect=5, read=4, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnection
Sep 11 21:15:48 hostname.domain.com pulp[22134]: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4, connect=5, read=4, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnection
Sep 11 21:15:53 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/904bb0487876f57165e08f8c36656d9fc7a59d099
Sep 11 21:16:06 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/f23f5b2777ab41d7576275984f80fc4e7f3d40b13
Sep 11 21:16:10 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/productid.
Sep 11 21:16:11 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Generating metadata databases.
Sep 11 21:16:12 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Determining which units need to be downloaded.
Sep 11 21:16:13 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.purge:INFO: [7d827b7c] Removing missing units.
Sep 11 21:16:14 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Downloading additional units.
Sep 11 21:16:16 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download failed: Download of https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/.treeinfo failed with code 404: Not Found
Sep 11 21:16:45 hostname.domain.com pulp[22134]: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4, connect=5, read=4, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnection
Sep 11 21:16:50 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download failed: Download of https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/treeinfo failed with code 404: Not Found
Sep 11 21:16:51 hostname.domain.com pulp[22134]: pulp_rpm.plugins.importers.yum.sync:INFO: [7d827b7c] Sync complete.
Sep 11 21:16:51 hostname.domain.com pulp[22134]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Sep 11 21:16:51 hostname.domain.com pulp[21757]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[1a54bbd1-eb58-4091-9e79-b66529bed46d]
Sep 11 21:16:51 hostname.domain.com pulp[21608]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[64509a63-1e72-47dc-ac07-f2a93d7fa239]
Sep 11 21:16:51 hostname.domain.com pulp[22134]: celery.app.trace:INFO: [7d827b7c] Task pulp.server.managers.repo.sync.sync[7d827b7c-ef6e-4d16-b4a3-e018172c4ed5] succeeded in 99.401586624s: <pulp.server.async.tasks.TaskResult object
Sep 11 21:16:51 hostname.domain.com pulp[22134]: celery.app.trace:INFO: [29758f63] Task pulp.server.async.tasks._release_resource[29758f63-cd89-4b41-9fcb-23fd63d9efa0] succeeded in 0.00130301099989s: None
Sep 11 21:16:51 hostname.domain.com pulp[22178]: celery.app.trace:INFO: [1a54bbd1] Task pulp.server.async.tasks._queue_reserved_task[1a54bbd1-eb58-4091-9e79-b66529bed46d] succeeded in 0.0181889970004s: None
Sep 11 21:16:51 hostname.domain.com pulp[21608]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[e9e21148-a44d-4d0f-a897-9cd0d8480ca8]
Sep 11 21:16:52 hostname.domain.com pulp[22134]: celery.app.trace:INFO: [64509a63] Task pulp.server.managers.repo.publish.publish[64509a63-1e72-47dc-ac07-f2a93d7fa239] succeeded in 1.431943844s: {'exception': None, 'repo_id': '2a742c
Sep 11 21:16:52 hostname.domain.com pulp[22134]: celery.app.trace:INFO: [e9e21148] Task pulp.server.async.tasks._release_resource[e9e21148-a44d-4d0f-a897-9cd0d8480ca8] succeeded in 0.00124579300018s: None

The bottom line of my findings:

Sep 11 22:41:22 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download succeeded: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/repodata/f23f5b2777ab41d7576275984f80fc4e7f3d40b1394184e1b1aa1956f7c8a93e-filelists.xml.gz.

whereas

Sep 11 22:42:02 hostname.domain.com pulp[22134]: nectar.downloaders.threaded:INFO: Download failed: Download of https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ansible/2.4/os/treeinfo failed with code 404: Not Found

Resolved by RHEL Support Case, Navigate to Satellite WebUI -> Administer -> Settings -> Content -> ensure that 'Default Repository download policy' is set to 'Immediate'. If not, please set it to Immediate and then re-synchronize the repositories again.

But I wonder why 'Immediate' was not the default setting on Satellite, and why there is nothing mentioned about it in 6.5 setup instructions?

I faced this exact issue, and the above and my Red Hat Technical Account Manager told me to execute the same thing you cited Kirill https://access.redhat.com/solutions/3481621

Been busy here, else I'd have shared with you earlier

Regards

RJ