How to change syncing policy for Repositories on Satellite 6 from "Immediate" to "On-Demand".

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6
    • 6.3 - 6.9 (pulp-2)
    • 6.11+ (pulp-3)
    • no procedure exists for 6.10

Issue

  • Need to change syncing policy from "Immediate" to "On-Demand" in order to free disk space.

Resolution

Please note the below procedure(s) can work on either Satellite 6.9 or older (with pulp-2), or on 6.11 or newer (with pulp-3 AND "Reclaim Space" feature implemented). There is no procedure for Satellite 6.10.

WARNING: This will remove all RPMs that were uploaded via hammer or the UI for repositories that are not synchronized from a remote source. These packages will need to be re-uploaded after running the below steps.

WARNING: This action is irreversible and data-loss may occur as you are forcibly removing all previously downloaded and uploaded RPMs. Creating a VM snapshot or backing up your Satellite before performing this operation is highly recommended

1) Backup your Satellite 6 Server using foreman-maintain. (including pulp content)

2) During the below steps, prevent any repository synchronization and ideally also client yum/dnf activity.

3) Change policy for each repository to "On-Demand" using the rake command below:
Note: This will also change the kickstart repositories to on demand which might be something you do not want - manual change for those repos e.g. in WebUI must follow

# foreman-rake katello:change_download_policy DOWNLOAD_POLICY=on_demand

Now, follow steps depending on Satellite / pulp version:

Subsequent step for Sat 6.11+ (pulp-3)

For any repository that changed Download Policy to On Demand, Reclaim its space. Either via WebUI -> the repository page -> Select Action -> Reclaim Space, or e.g. via hammer repository reclaim-space command.

Subsequent steps for Sat 6.9 or older (pulp-2)

4) Delete all the files on the file system which are stored at:

# rm -rf /var/lib/pulp/content/units/{rpm,srpm,drpm,distribution}

5) Update the database with the following script:

#!/usr/bin/env python
try:
    from pymongo import MongoClient
except ImportError:
    print 'Error: PyMongo module is not installed'
    exit()

client = MongoClient()
db = client.pulp_database

collections_list = [db.units_rpm, db.units_srpm, db.units_drpm, db.units_distribution]


def mongoupdate(collection):
    print "Updating collection " + collection.name
    results = collection.update_many(
        {"downloaded": True},
        {"$set": {"downloaded": False}}
    )
    print 'Done, updated ' + str(results.modified_count) + " records \n"

for i in collections_list:
    mongoupdate(i)

count = 0
for i in collections_list:
    print '\nChecking if there are any downloaded packages left at ' + i.name + '\n'
    cur = i.find({"downloaded": True})
    for k in cur:
        count += 1
    print '  Amount of downloaded packages  ' + str(count) + '\n'
    count = 0

6) Reload pulp:

# for i in pulp_resource_manager pulp_workers pulp_celerybeat pulp_streamer; do service $i restart; done

7) Resync all the repos with hammer using the --skip-metadata-check true flag:

USER=username
PASSWORD=password
for id in $(hammer -u $USER -p $PASSWORD repository list | grep yum | awk '{ print $1 }'); do
  echo "repository synchronize --id $id --skip-metadata-check=yes --async"
done | hammer -u $USER -p $PASSWORD shell

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

8 Comments

Just to confirm, I've tested this procedure on Satellite 6.3.0 and seems to work fine, too.

Two comments:

Step 2 can be triggered with a single command via :

DOWNLOAD_POLICY=on_demand foreman-rake  katello:change_download_policy

Step 5 depends on the number of enabled repos, and can be parallelized according to the number of CPU cores your Satellite server has by substituting "-P 4" below with the number of cores:

hammer repository list | grep -P '^\d' | cut -f 1 -d '|' |xargs -n 1 -P 4 hammer repository synchronize --skip-metadata-check true --id 

In my 4-core Satellite system the metadata synchronization took about 3h for ~70 repos with -P4, YMMV.

I tried to run this on a Satellite 6.2.13, but got an error in step 3:

[root@salcto-sat01 ~]# foreman-rake katello:change_download_policy DOWNLOAD_POLICY=on_demand
rake aborted!
Don't know how to build task 'katello:change_download_policy'

(See full trace by running task with --trace)
[root@salcto-sat01 ~]# rpm -q satellite
satellite-6.2.13-4.0.el7sat.noarch
[root@salcto-sat01 ~]# 

Any pointers are appreciated.

Thanks!

foreman-rake katello:change_download_policy DOWNLOAD_POLICY=on_demand

Is available only from Sat 6.3, so for 6.2 you should rely on

# for id in `hammer repository list --organization-id 1 | awk '{print $1}' | grep -v "-" | grep -v "ID" `; do  hammer repository update --id=$id --download-policy=on_demand ; done   

For more references see the article Satellite 6.2 Feature Overview: Lazy Sync

Tried it on Sat 6.2.15, it worked, but the repository update should only be done on RPMs repositories, not Kickstart ones.
Having a Kickstart repo in on-demand mode gives me problems with PXE provisioning until I revert it to Immediate and fully sync the repo.

Just had an issue with pulp_streamer service being down while doing yum-install(s) on client machines.
Better be sure to restart it along with the other pulp services, like

# for i in pulp_resource_manager pulp_workers pulp_celerybeat pulp_streamer; do service $i restart; done

Hi There,

Can anyone confirm if this procedure will work for Satellite 6.7.3? The size of /var/lib/pulp is quite up there and want to reduce it.

Thanks Albert

It works fine on 6.9.2.1 so I'm sure it will work on 6.7.3 as well.

Just make sure you have a backup / snapshot and you should be fine.

Better taxonomy , have a pulp2 and pulp3 , ... pulp4 tags vs just pulp ?

As we were to be on sat7 by now , but the powers at be were forced to keep sat6 version.

As RH need to do a sweep of the kbase to clean up all the pulp2ism. and re-taging.