How to create a local mirror of the latest update for Red Hat Enterprise Linux 5, 6 and 7 without using Satellite server?
Environment
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 5
- Red Hat Network (RHN)
- Red Hat Subscription Management (RHSM)
- reposync
Issue
- What is
reposyncutility and how to use it? - How to create a local mirror of the latest update for Red Hat Enterprise Linux 5, 6 or 7 without using Satellite server
- Need to download all packages / rpms from specific channel locally
- How to make a local repository
Resolution
For RHEL 8 and 9 refer How to create a local mirror of the latest update for Red Hat Enterprise Linux 8 and 9 without using Satellite server?
RHEL 5
# mkdir /var/repo
# reposync --gpgcheck -l --repoid=rhel-5-server-els-rpms --download_path=/var/repo --downloadcomps
# cd /var/repo/rhel-5-server-els-rpms
# createrepo -v /var/repo/rhel-5-server-els-rpms
# yum clean all
# yum list-sec
# find /var/cache/yum/ -name *updateinfo.xml*
# mv /var/cache/yum/rhel-5-server-els-rpms/365ae03ca85bb9d3bc509ea9129d1d3fb9a18381-updateinfo.xml.gz /tmp
# cd /tmp
# gzip -d 365ae03ca85bb9d3bc509ea9129d1d3fb9a18381-updateinfo.xml.gz
# mv 365ae03ca85bb9d3bc509ea9129d1d3fb9a18381-updateinfo.xml updateinfo.xml
# cp updateinfo.xml /var/repo/rhel-5-server-els-rpms/repodata/
# modifyrepo /var/repo/rhel-5-server-els-rpms/repodata/updateinfo.xml /var/repo/rhel-5-server-els-rpms/repodata/
RHEL 6,7
- Create a basic repository
- Create a repository that allows clients to install groups
- Modify repodata to identify which packages are security related
Install the required packages
-
Install the
yum-utilsandcreaterepopackages on the registered system.# yum install yum-utils createrepo
Create a basic local repository
Note: Change <repo-id> to the repository you intend to sync
-
Sync all the packages from a specified repository to a specified directory
# reposync --gpgcheck -l --repoid=<repo-id> for example: # reposync --gpgcheck -l --repoid=rhel-6-server-rpms --download_path=/var/www/html -
In the targeted directory, there will be a new directory named after the Repository ID. All the downloaded packages will be inside this directory.
# cd /var/www/html/<repo-id> # createrepo -v /var/www/html/<repo-id>
Create a local repository that allows clients to use groups
How to download all the metadata for the repository that is being synced which will allow the use of various plugins such as 'yum groupinstall'
-
On RHEL6 and later, reposync supports the
--download-metadataand--downloadcompsoptions. For example:# reposync --gpgcheck -l --repoid=repo-id --downloadcomps --download-metadata for example: # reposync --gpgcheck -l --repoid=rhel-6-server-rpms --download_path=/var/www/html --downloadcomps --download-metadata -
To have access to the group data for the newly synced repo, please run the
createrepocommand as follows:# cd /var/www/html/<repo-id> # createrepo -v /var/www/html/<repo-id>/ -g comps.xml
Modify the repodata to define which packages are security related.
-
These steps require that the createrepo command has already been run.
# yum clean all # yum list-sec # find /var/cache/yum/ -name updateinfo.xml ##For RHEL 5 use '-name *updateinfo.xml*' -
From the find command above, identify the
updateinfo.xmlthat matches the<repo-id>that you ranreposyncagainst and move that file into your repodata directory.# mv updateinfo.xml /var/www/html/<repo-id>/repodata/updateinfo.xml # modifyrepo /var/www/html/<repo-id>/repodata/updateinfo.xml /var/www/html/<repo-id>/repodata -
How to update security Erratas on system which is not connected to internet ?
NOTE:
- To keep the sync current, for example, cronjobs can be used. The
createrepocommand supports--updateto efficiently update existing repositories. - The locally created repository is typically used by other RHEL clients via LAN, for example via HTTP/HTTPS (for example provided by the apache webserver which is part of RHEL), via FTP (i.e. vsftpd) or NFS (nfs-utils package). Share this local repository with the offline systems to update the offline systems.
reposyncutility can only mirror repositories which the system is entitled to.- Related information How do I delete old packages in local repository server?
- For RHEL 7.7+, The Red Hat Customer Portal assumes that if all Red Hat provided CDN repositories are disabled the system lacks access to the latest content, and will not show errata information. If you need/want to see a systems errata information in the Red Hat Customer Portal, subscription manager must have at least one official redhat repository enabled. Note that only the enabled repositories are considered as part of the errata applicability calculation.
- You can create a copy of a directory which is prepared as YUM repository, and then without adding further packages to that repo have clients use the repo as a "frozen" set of packages. Creating such copies needs extra storage, but as the files are duplicates of existing packages, as long as the involved files are on the same file system,
hardlinkscan be used to then reduce the requirements.
-
If there are any issues after following above steps, open a Red Hat Technical Support Case and share below details.
- Output of the following command :
# tree /directory_path_of_the_repository- Generate and provide SOS report from Repo server and client system.
Root Cause
Red Hat provides a utility called reposync which can be used to download the packages from the CDN. In order to download all packages from a specific channel, the system should be subscribed to that channel. If the system is not subscribed to the required channel then reposync will not be able to download and sync those packages on local system.
Diagnostic Steps
createrepo-0.9.9-26.el6.noarchwhich is part of RHEL6.9GA has an issue regarding--update, refer to bz1434369 for details.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
79 Comments
How can one do this for multiple software channels and architectures ? I am trying to maintain servers and workstations in both 32 and 64 bit and I need to try to do it without Satellite.
Hi
,
Try another repoid:
Could just change this to something like (but this is a guess..):
Createrepo in both directories, configure a service to serve your rpm's and you're ready to go.
The below command not work on rhel5 32 bit.
reposync --gpgcheck -l --repoid=rhel-x86_64-server-5 --download_path=/test/path
pls let me know y it won't work
Thanks
Correction: sudo reposync --gpgcheck -l --repoid=rhel-x86_64-server-5 --download_path=/test/path
reposync would pull the contents from RHN only when the system needs to be subscribed to the channels. As you are trying to sync x86_64 channel on a i386 (32bit) system, this will not work because system is not subscribed to rhel-x86_64-server-5 channel.
I have updated the article accordingly.
Thanks !
OK, but is it possible to get both a 32 bit and 64 bit channel repo sync'ed onto a 64 bit machine ?
For the purposes of a repo-server, can one subscribe one physical box to multiple channels ?
No, system cannot subscribe to multiple arch base channels (example RHEL 5 32bit and 64bit).
As reposync has the ability to sync packages only from the channels/repositories to which it has access, it is not possible sync multiple arch channels from RHN on the same system.
You can refer https://access.redhat.com/kb/docs/DOC-9744 which has the details on setting up repository from RHEL DVD iso. This could help you setup multiple channel repositories on a single system but it won't have latest errata's after the recent RHEL update release. (example RHEL 5.7)
What about subscribing to multiple version channels (RHEL 5 and RHEL 6) ? This is for a spacewalk server.
can one separate the errata's per minor release, e.g. 5.9 errates, 5.10 errates, 5.11 errates, ..etc?
Did anyone knows a solution for the different minor releases?
FYI, you can use reposync to pull from content sets the machine you are running from is not directly subscribed to.
you need to extract the URL's from your entitlement cert (rct works decebnt for this). then make a custom yum.conf with the URL, your client cert and such and finally run reposync giving it a path to download to
was able to run reposync just fine to slurp down rhels7 content sets while running on a rhels6 install util box.
where can I find other channel-ids or repoids that are valid names?
Under el6 you can see all the available channels using rhn-channel -L which is part of the rhn-setup package. It probably exists on el5 too.
So this is a total fail. I have to install a separate system for each Architecture just to be able to sync updates.
Worthles.... Even M$ doesn't require this nonsense.
I'm trying this and it mostly works, but now I am getting a series of errors on some packages (but not most of them) on the different channels, such as this example. I tried doing a yum clean and running reposync again, but it continues to get the same errors on the same set of packages.
The reposync command in the script I use looks like (I noticed that you use -g here. I did not have that, but I added it and it made no difference as one would expect. Also removing -q does not add anything useful to the output messages. Adding the -t option also makes no difference)
~~~
reposync -q -n -a x86_64 -d -m -l -p ${base_dir} \
-r rhel-x86_64-server-6 \
-r rhel-x86_64-server-fastrack-6 \
-r rhel-x86_64-server-optional-6 \
-r rhel-x86_64-server-optional-fastrack-6
~~~
For each of a number of certain packages, consistent which ones, but most packages do not produce these messages, I get
~~~
Could not retrieve package batik-1.7-6.3.el6.x86_64. Error was failed to retrieve getPackage/batik-1.7-6.3.el6.x86_64.rpm from rhel-x86_64-server-6
error was [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 416 Requested Range Not Satisfiable"
~~~
Any advice about this?
I don't know whether this cleared it or if the rhn server was busy cleaning up a problem while I was doing this, but when I tried yumdownloader of a package that had this error it downloaded ok, and then running reposync again has no more errors on the channel that contained that package. Then I did yumdownloader of another package that still had the error, and running reposync again had no error from that package and some others, but still had the error on some other packages in the same channel, i.e., the use of yumdownloader did not fix one entire channel as it appeared to from the first time I did it. However each time I ran yumdownloader to get a package that had this error it downloaded ok and then the error did not repeat when I ran reposync again, repeating until reposync runs with no errors at all.
Is there anything additional required to sync security errata? Can reposync/createrepo manage this? We would like to use the local repo for security updates as well.
Hi,
Is it possible to subscribe and mirror 2 different releases (on the same architecture - x86_64).
So I can have one repo for rhel 5.x and another for 6.x; both on the same machine though.
Thanks much,
Dumitru
It looks like 'createrepo' no longer comes with yum-utils (if it ever did). It's now in a separate package named createrepo. You may want to update this solution with the additional step of running 'yum install createrepo'.
To add one , along with createrepo two more packages will be required i.e deltarpm & python-deltarpm.
Hi,
Is it possible to follow this with RHEL 7?
Thanks!
For better performance, one may want to give createrepo_c a try instead of createrepo. It is available from the EPEL repository (therefore not officially supported by Red Hat).
Is there a way to download the original repodata folder, or the groups xml file?
What is the recommended way to disable the RHN base channel of systems that are using a local repo created in this manner so that package updates only come from the local repo?
Hi Matthew,
You can disable it by enable=0 in /etc/yum/pluginconf.d/rhnplugin.conf
The reposync option --download-metadata is not available on RHEL5's version of reposync. This is needed on RHEL6 (and above?) to pull security metadata.
Is there an equivalent on Redhat 5?
I would like to know the same. I am in the process of mirroring "rhel-5-server-rpms" which I suppose is the relative equivalent of "rhel-x86_64-server-5" in these instructions.
I get an error that --download-metadata is an invalid option. --downloadcomps works, but if I understand correctly that only contains the package groupings.
How do I download the security metadata so that I can use the yum-security plugin with this mirror?
I didn't see any channels for rhel-x86_64-server-5 in the output of
subscription-manager repos --list | grep "Repo ID"How to view available repositories:
https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/entitlements-and-yum.html#viewing-repos
Here's what I've been doing to keep the metadata. It's not elegant but it works by just copying the metadata from the local cache every time.
Let me know if it works for you :)
duplicate response - please delete
How run sync on server ?
what and how to configure the client need?
How we can download the RPM's when we are nor exposed to internet? The solution says when you don't have satellite server.
Pls provide the solution if we don't have both the alternatives(rhn connectitivty and satellite server).
Indeed, this solutions requirement is always that somewhere there is internet connectivity. I am not aware that we are offering services like "shipping USB sticks" or "dvd-media" with these updates. Please open a case with Red Hat support to discuss possible alternatives. Seeing you posting this comment makes be confident there might be some level of internet connectivity.
Hi Christian. What are the least privileges that successfully download rpms and metadata without flipping bits? (I would like to download rpms and metadata from a limited user account. )
Regarding alternatives: Does Red Hat supply any (metadata) distribution schemes or interfaces which are (loosely) analogous to https://support.microsoft.com/en-ca/kb/913086?
Thanks!
Hi,
As for interfaces to access data about security fixes in an automated way, "yum security" on an installed system or https://access.redhat.com/solutions/4161 (describing our OVAL data) are a good start.
Hi Christian, Attached please find the logic to refresh repository metadata.
Can it be simplified?
Best Regards, Brent
Hi, did you verify if createrepo option
--updatecould help you here?There is nothing to update.
Hello,
If I'm following this correctly, I believe you are trying to include the security data in your local repository yes? If you haven't seen this it may help. https://access.redhat.com/solutions/55654
You want the updatinfo.xml.gz to be in the repodata folder created by running createrepo after the reposync.
Then you will want to use modifyrepo as shown below to apply the security data to the repo.
the updateinfo.xml.gz file should be put in //repodata/
Once all this is done, remember that clients connecting to this should run "yum clean all" to ensure they remove the cached metadata and get the new metadata.
Hi John. Thanks for the pointer to https://access.redhat.com/solutions/55654. 189533 helps as well.
Question: Under what circumstances do the *-updateinfo file(s) below differ?
For the local repository server, when it connects it will store the updatinfo.xml.gz file in: /var/cache/yum////*-updateinfo.xml.gz
That file needs to be taken from there, and put into the directory where you ran createrepo. Then modify repo will put the data in place. So if I understand the question correctly, These should be the same files. The main difference I saw is that "updatinfo.xml" needs to be put in the repodata folder of the local repo. //repodata/updatinfo.xml
Hi John. You are correct, updateinfo.xml should be in the repodata folder.
Question. Where is the XML schema definition file for updateinfo?
Thanks! Brent
Hi John and Christian
I see that comps.xml is verified by comps.rng. Where can I find the other schema definitions?
https://bugzilla.redhat.com/show_bug.cgi?id=1296282
https://rhn.redhat.com/errata/RHBA-2016-2187.html
I can not answer that from the top of my head. If it is possible for you to open a case at the customer center, with the details around the issue, then this would probably the best way to answer this question.
Hi Christian.
Good tip! Thanks
Cheers, could you maybe add rough numbers for the expected sizes of the respective repositories?
Would depend on whether you're doing a full sync or using the "only latest" option. Having done a reposync, today, of the 6-Server channel, using the "only latest" option cut the download from 17,000+ RPMs to a little over 4,000.
Not easy to do this reliably, I would do a full fetch, take note of the size, and then look at the growth over some months. Atleast one minor release should also be part of the observation, as these are also introducing new features (for releases in production phase 1), and bring many new packages. From that one could draw estimations for the consumtion. In production phase 2 and later, the size will grow much slower.
Is it possible to use a single subscribed Centos 7(preferably) or RHEL 7 server to host base and update package repos for RHEL5-7? We have no interest in Satellite server just want a simple network local yum repo for RHEL OS updates like we have for CentOS.
"hosting" repos for all of these via protocols like http or nfs could be done on a single system like a Centos7 box. Yet, fetching does to my knowledge require one "RHEL" per architecture and per major line. So for 32bit/64bit for rhel5/6/7, one can setup KVM guests with these releases for fetching, and configure the host to offer the fetched packages. In that config the host could also be RHEL7/64bit directly for fetching this release, and 4 VMs could be used to cover the rhel5/6 flavours.
Thanks, that is what i ended up doing building a sync server using RHEL 6 and 7 and subscribing them in order to provide RHEL repos internally.
Is there an easy way to grab just the comps.xml?
Not aware of a way in style of "wget ...comps.xml" to get the latest version. Maybe the versions which are on the RHEL media are ok for you. The ordering of packages into groups is not changing very often.
The note about security repositories requiring a different process is unclear (at least to me).
Doesn't the rhel-7-server-rpms channel include security updates for the packages in that channel when they're released?
Also, the link to that separate process goes to a page that only references RHEL5 and RHEL6 -- does it still apply to RHEL7? It would appear NOT, because the updateinfo.xml.gz doesn't exist in my RHEL7 system, although I do have files with suspiciously similar names, like this one:
/var/cache/yum/x86_64/7Server/rhel-7-server-thirdparty-oracle-java-rpms/0afde39e01bc126f3d3335a6adddc87cc50e46c6ebfdd6574e8193e2aeedf14c-updateinfo.xml.gz
It is my understanding that this only does a one-time copy of the repo data from the source repo and that if you want to have it automatically update the repository data on a regular basis you need to set up cron/acron/... with a script or individual commands that perform reposync and "createrepo --update" for each of the repos.
It could be useful to others to suggest that automated tasks/a script be set up and an example of the commands to run for each repo. I did not do the initial set up's of our repo server and we did not use the path recommended in this article, however it also appears that a web server nor an ftp server are a requirements for either yum-utils or createrepo to be installed. It might be useful to mention something like a webserver or ftpserver need to be installed and configured; and that if you install apache with an http based repos and put the repos in /var/www/html as instructed no additional configuration should be needed. Thank you.
Thank you, hints on this have been incorporated. We did this not until now, because there are so many options for distribution. Besides HTTP(S)/FTP/NFS, distributing the repos with rsync/ssh or pulp is also not uncommon.
In reviewing the article I think that waiting to mention --update as a note under Diagnostic Steps where it could easily be missed by a rushing sysadmin will cause the sysadmins to rely on this for days to possibly months before they realize that somethings is wrong, and then having them open an after the fact support ticket or wasting a bunch of time trying to figure out why the local repo is not providing update content. Maybe it should be moved up as an optional step without specific details as it will vary greatly by preference and number of repositories that are mirrored. Also this may be my over interpreting things but to me "local mirror," as contained in the Issue section, makes me think of something that can be accessed by more than just the box itself, using most likely http, https, ftp or sftp even though other protocols will work. I would find it would be much more end user friendly if the note "Above commands create a local copy ... or NFS (nfs-utils package)" were inline before the "How to create a .repo file" link; along with some wording about "For other boxes to access the repo you may need to;" and that "choosing which distribution method to use and how to configure them is outside the scope of this document." Mention something like "Though an apache http install and it's configuration should be relatively easy due to the repositories being placed in /var/www/html/” could be especially useful since the linked repo configuration page is a very simple page with minimal information and the only example of a repo file has http for the base url. With comments like these inline or maybe in the Resolution section like the fact that the box has to be subscribed to the channel it would be much clearer that additional steps are needed. Thank you for the changes you have already made and for considering the following comments ideas for the main article.
Thanks.
Hi, I have subscription for Red Hat Enterprise Linux for SAP Applications, how can I sync it locally?
This local repository is not minor version specific.
When you use reposync against a repository, it will sync all available packages. The only way to version lock the local repo is to version lock the server running "reposync". Then it will only have access to the minor release specified.
You will need to keep this is in a separate folder from the main repo
Thanks
Dear Experts,
When we create and offline repo from a server that have valid subscription does the client that are using these offline repo will need a valid subscription too? does it legal to use the the offline repo for a production machine that does not have a subscription from terms and condition point of view? (self support)
I could not found anything of the lisense agreement
Regards Thanks!
I would like to know if there is a possibility of creating repositories of version 6.x and 7.x on a single server with version 6.9.
You will need one registered system for every Major Version or Variant repository you need to sync. You can sync the RHEL 7 packages then move them to the RHEL 6 machine in order to host them on one system.
..or have the rhel6 and rhel7 guests fetch onto a common filesystem, for example an NFS share. One could also run a rhel7 on bare metal, have the rhel6 run as KVM guest, and consolidate the files on an NFS file system hosted by the rhel7 hypervisor.
Just in case anyone was wondering how to find out what repo-ids are available for their system, you can reference this page:
Enabling or disabling a repository using Red Hat Subscription Management
getting below error
reposync --gpgcheck -l --repoid=rhel-x86_64-server-6 --download_path=/repodata/rhel68 --downloadcomps --download-metadata Loaded plugins: product-id, rhnplugin, subscription-manager This system is receiving updates from RHN Classic or RHN Satellite. Traceback (most recent call last): File "/usr/bin/reposync", line 386, in main() File "/usr/bin/reposync", line 290, in main resultfile = repo.retrieveMD(ftype) File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1571, in retrieveMD return self._retrieveMD(mdtype) File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1654, in _retrieveMD size=thisdata.size) File "/usr/share/yum-plugins/rhnplugin.py", line 397, in _getFile self.id, e) yum.Errors.RepoError: failed to retrieve repodata/comps.xml from rhel-x86_64-server-6 error was [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 500 Internal Server Error"
Hello everyone,
I would like to draw your attention to a small project of mine which provides some useful scripts to setup a local mirror for RHEL repos without using Satellite server.
The project is called Poor Man's RHEL Mirror and it is hosted on GitHub.com. Information about what it does and for what it could be used you will find in the README.md.
Please feel free to use it and adapt it to your own needs. Feedback is welcome.
Best regards,
Joerg
Could this article be updated to include RHEL 8? I'm afraid introduction of molularity adds some complexity to the process.
RHEL8 looks quite a bit different, we decided to not make this here more complex and created https://access.redhat.com/solutions/4163651 . Still state 'solution in progress'. As for modularity, I think that this is dealed with on yum/dnf level, once they just plainly see the repo. For the 2 repos (baseos/appstream) from the rhel8 media, it works like this. rhel8 clients get these 2 configured as repos, and the modularity parts are then done on yum layer.
shouldn't it be /var/www/html//repodata/updateinfo.xml ??
Yes it should. I have updated the article now to correctly reflect that. Thank you for letting me know.
Forgive me if I missed it in previous comments, but how does a local repo handle RH subscriptions and validate that I'm not standing up unlicensed servers? Just trying to get a full understanding of this setup and don't want to run into any unexpected difficulties. Thanks!
At this time, local repositories do not validate if the clients pulling content from it are entitled or not. For that purpose we have Red Hat Satellite which can sync and host repositories for all of your clients which register to the satellite itself. The local repo is a single registered system that is downloading all of the content of a repository it has access to, and then hosting it via http (most commonly).
Yes. We currently have Satellite 5 and are planning to move to a different platform and I was looking at this as a possible solution for that, but it sounds like that will not work for what we're looking for. Thanks!
Hi, Got my RHEL 7 local repo server running with no problem, however, how do I restrict client servers to use only minor release, e.g. 7.6 updates? Thanks.
Hm.. can't think of an easy way, sorry. Most client systems probably use 'subscription-manager release --set=..', but as these clients here just use a pure yum repo, that does not apply. You could by some way track for example the rhel-7.6 stream receiving updates, and on the repo server manually maintain a repo where you copy in the rpm packages released in that stream. I see no easy way to automate this. This might be one of the advanced features for which you should consider RH Satellite.
When we release lock a registered system, it actually changes the baseurl and looks at a different location. You would need to do the same thing by release locking the Repo Server, then syncing to a folder for specifically only that release.
Then the clients would need a repo configuration to look at that 7.6 locked repo.
Pages