How to create a local mirror of the latest update for Red Hat Enterprise Linux 5, 6, 7 without using Satellite server?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5
  • Red Hat Network (RHN)
  • Red Hat Subscription Management (RHSM)
  • reposync

Issue

  • What is reposync utility and how to use it?
  • How to create a local mirror of the latest update for Red Hat Enterprise Linux 5, 6, or 7 without using Satellite server
  • Need to download all packages / rpms from specific channel locally

Resolution

Red Hat provides a utility called reposync which can be used to download the packages from RHN. In order to download all packages from a specific channel, the system should be subscribed to that channel. If the system is not subscribed to the required channel then reposync will not be able to download and sync those packages on local system.

Note: below steps will sync a repository locally just one time, please read this knowledge base completely for details on setting up a mirror which frequently pulls down the newest packages.

Example: If a Red Hat Enterprise Linux 6 system is subscribed to rhel-6-server-rpms and rhel-6-server-eus-rpms, then reposync can sync the contents only from these 2 channels. If you try to sync contents from rhel-6-server-optional-rpms or any other channels to which system is not subscribed then reposync will not work.

Install the required packages

  • First install the "yum-utils" and "createrepo" packages on the system which is registered:

    # yum install yum-utils createrepo
    

Create a basic local repository

NOTE: Please change the --repoid=<repo-id> according to what repo id the system is currently using.

  • Sync all the packages from a specified repository to a specified directory

    # reposync --gpgcheck -l --repoid=repo-id
    for example:
    # reposync --gpgcheck -l --repoid=rhel-6-server-rpms --download_path=/var/www/html
    
  • In the targeted directory, there will be a new directory named after the Repository ID. Inside there is all the packages.

    # cd /var/www/html/<repo-id>
    # createrepo -v /var/www/html/<repo-id>
    

Create a local repository that allows clients to install groups and use the security plugin

How to download all the metadata for the repository that is being synced which will allow the use of various plugins such as 'yum groupinstall'

  • On RHEL6 and later, reposync supports the --download-metadata and --downloadcomps options. For example:

    # reposync --gpgcheck -l --repoid=channel-id --downloadcomps --download-metadata
    for example:
    # reposync --gpgcheck -l --repoid=rhel-6-server-rpms --download_path=/var/www/html \
        --downloadcomps --download-metadata
    
  • To have access to the group data for the newly synced repo, please run the createrepo command as follows:

    # cd /var/www/html/<channel-id>
    # createrepo -v  /var/www/html/<channel-id>/ -g comps.xml
    
  • To enable the security feature of the repo you will need to run these additional commands AFTER running the createrepo command above:

    # cd /var/www/html/<channel-id>
    # gunzip *updateinfo.xml.gz
    # mv *updateinfo.xml ./repodata/updateinfo.xml
    # modifyrepo /var/www/html/<channel-id>/repodata/updateinfo.xml /var/www/html/<channel-id>/repodata
    
  • Share this local repository with the offline systems to update the offline systems.

NOTE:

  • Above commands sync one or multiple current repositories. To keep the sync current, for example, cronjobs can be used. The createrepo command supports --update to efficiently update existing repositories. Please note that createrepo-0.9.9-26.el6.noarch which is part of RHEL6.9GA has an issue regarding --update, refer to bz1434369 for details.
  • Above commands create a local copy of the repository. This is typically used by other RHEL clients via LAN, for example via HTTP/HTTPS (for example provided by the apache webserver which is part of RHEL), via FTP (i.e. vsftpd) or NFS (nfs-utils package).
  • reposync utility will create the local mirror for the channel to which the system is subscribed.
  • For creating local security repositories, the procedure to be followed is different as the metadata for security rpms differs from that for regular rpms . createrepo will not create the updateinfo.xml file. Please visit this link for more information on how to create a local security repository. Note that the server which will act as a local security repository should be of the same RHEL version as the clients. (Example: if clients' version is RHEL6.6, server should also be RHEL 6.6).
  • Accessing multiple repositories without the appropriate, active, subscription of RHEL version on 'single' system is not supported and will be considered as abuse of subscription.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

65 Comments

How can one do this for multiple software channels and architectures ?  I am trying to maintain servers and workstations in both 32 and 64 bit and I need to try to do it without Satellite.

Hi

,

Try another repoid:

reposync --gpgcheck -l --repoid=rhel-x86_64-server-5 --download_path=/test/path  

Could just change this to something like (but this is a guess..):

reposync --gpgcheck -l --repoid=rhel-i386-server-5 --download_path=/test/path2

Createrepo in both directories, configure a service to serve your rpm's and you're ready to go.

The below command not work on rhel5 32 bit.

reposync --gpgcheck -l --repoid=rhel-x86_64-server-5 --download_path=/test/path

pls let me know y it won't work

Thanks

Correction: sudo reposync --gpgcheck -l --repoid=rhel-x86_64-server-5 --download_path=/test/path

reposync would pull the contents from RHN only when the system needs to be subscribed to the channels.  As you are trying to sync x86_64 channel on a i386 (32bit) system, this will not work because system is not subscribed to rhel-x86_64-server-5 channel.

I have updated the article accordingly.

Thanks !

OK, but is it possible to get both a 32 bit and 64 bit channel repo sync'ed onto a 64 bit machine ?

For the purposes of a repo-server, can one subscribe one physical box to multiple channels ?

No, system cannot subscribe to multiple arch base channels (example RHEL 5 32bit and 64bit).

As reposync has the ability to sync packages only from the channels/repositories to which it has access, it is not possible sync multiple arch channels from RHN on the same system.

You can refer https://access.redhat.com/kb/docs/DOC-9744 which has the details on setting up repository from RHEL DVD iso. This could help you setup multiple channel repositories on a single system but it won't have latest errata's after the recent RHEL update release. (example RHEL 5.7)

What about subscribing to multiple version channels (RHEL 5 and RHEL 6) ? This is for a spacewalk server.

can one separate the errata's per minor release, e.g. 5.9 errates, 5.10 errates, 5.11 errates, ..etc?

Did anyone knows a solution for the different minor releases?

FYI, you can use reposync to pull from content sets the machine you are running from is not directly subscribed to.

you need to extract the URL's from your entitlement cert (rct works decebnt for this). then make a custom yum.conf with the URL, your client cert and such and finally run reposync giving it a path to download to

was able to run reposync just fine to slurp down rhels7 content sets while running on a rhels6 install util box.

where can I find other channel-ids or repoids that are valid names?

Under el6 you can see all the available channels using rhn-channel -L which is part of the rhn-setup package. It probably exists on el5 too.

So this is a total fail. I have to install a separate system for each Architecture just to be able to sync updates.

Worthles.... Even M$ doesn't require this nonsense.

I'm trying this and it mostly works, but now I am getting a series of errors on some packages (but not most of them) on the different channels, such as this example. I tried doing a yum clean and running reposync again, but it continues to get the same errors on the same set of packages.

The reposync command in the script I use looks like (I noticed that you use -g here. I did not have that, but I added it and it made no difference as one would expect. Also removing -q does not add anything useful to the output messages. Adding the -t option also makes no difference)

~~~

reposync -q -n -a x86_64 -d -m -l -p ${base_dir} \
-r rhel-x86_64-server-6 \
-r rhel-x86_64-server-fastrack-6 \
-r rhel-x86_64-server-optional-6 \
-r rhel-x86_64-server-optional-fastrack-6

~~~

For each of a number of certain packages, consistent which ones, but most packages do not produce these messages, I get

~~~

Could not retrieve package batik-1.7-6.3.el6.x86_64. Error was failed to retrieve getPackage/batik-1.7-6.3.el6.x86_64.rpm from rhel-x86_64-server-6
error was [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 416 Requested Range Not Satisfiable"

~~~

Any advice about this?

I don't know whether this cleared it or if the rhn server was busy cleaning up a problem while I was doing this, but when I tried yumdownloader of a package that had this error it downloaded ok, and then running reposync again has no more errors on the channel that contained that package. Then I did yumdownloader of another package that still had the error, and running reposync again had no error from that package and some others, but still had the error on some other packages in the same channel, i.e., the use of yumdownloader did not fix one entire channel as it appeared to from the first time I did it. However each time I ran yumdownloader to get a package that had this error it downloaded ok and then the error did not repeat when I ran reposync again, repeating until reposync runs with no errors at all.

Is there anything additional required to sync security errata? Can reposync/createrepo manage this? We would like to use the local repo for security updates as well.

Hi,

Is it possible to subscribe and mirror 2 different releases (on the same architecture - x86_64).
So I can have one repo for rhel 5.x and another for 6.x; both on the same machine though.

Thanks much,
Dumitru

It looks like 'createrepo' no longer comes with yum-utils (if it ever did). It's now in a separate package named createrepo. You may want to update this solution with the additional step of running 'yum install createrepo'.

To add one , along with createrepo two more packages will be required i.e deltarpm & python-deltarpm.

Hi,
Is it possible to follow this with RHEL 7?

Thanks!

For better performance, one may want to give createrepo_c a try instead of createrepo. It is available from the EPEL repository (therefore not officially supported by Red Hat).

Is there a way to download the original repodata folder, or the groups xml file?

What is the recommended way to disable the RHN base channel of systems that are using a local repo created in this manner so that package updates only come from the local repo?

Hi Matthew,

You can disable it by enable=0 in /etc/yum/pluginconf.d/rhnplugin.conf

The reposync option --download-metadata is not available on RHEL5's version of reposync. This is needed on RHEL6 (and above?) to pull security metadata.

Is there an equivalent on Redhat 5?

I would like to know the same. I am in the process of mirroring "rhel-5-server-rpms" which I suppose is the relative equivalent of "rhel-x86_64-server-5" in these instructions.

I get an error that --download-metadata is an invalid option. --downloadcomps works, but if I understand correctly that only contains the package groupings.

How do I download the security metadata so that I can use the yum-security plugin with this mirror?

I didn't see any channels for rhel-x86_64-server-5 in the output of subscription-manager repos --list | grep "Repo ID"

How to view available repositories:
https://access.redhat.com/documentation/en-US/Red_Hat_Subscription_Management/1/html/RHSM/entitlements-and-yum.html#viewing-repos

Here's what I've been doing to keep the metadata. It's not elegant but it works by just copying the metadata from the local cache every time.

Let me know if it works for you :)

#!/bin/bash
#==============================================================================
# Title        : syncRepo.sh
#
# Description  : Synchronizes the local RHEL repository to RHN
#
# Author       : Firas AlShafei
#
# Version      : v1.1
#
#==============================================================================
RELEASE=rhel-x86_64-server-5

# Clear YUM cache
yum clean all

# Create directory structure
mkdir -p /MYPATH/RHEL5_REPO/$RELEASE

# Remove previous metadata from our repository
rm -f /MYPATH/RHEL5_REPO/$RELEASE/repodata/*

# Sync RHN repo to our local folder 
reposync -p /MYPATH/RHEL5_REPO/ --repoid=$RELEASE -l

# Create metadata
yum --disablerepo=* --enablerepo=$RELEASE makecache

# Copy metadata from cache to our local repo
cp -f /var/cache/yum/$RELEASE/*.xml* /MYPATH/RHEL5_REPO/$RELEASE/repodata/

duplicate response - please delete

How run sync on server ?

what and how to configure the client need?

How we can download the RPM's when we are nor exposed to internet? The solution says when you don't have satellite server.

Pls provide the solution if we don't have both the alternatives(rhn connectitivty and satellite server).

Indeed, this solutions requirement is always that somewhere there is internet connectivity. I am not aware that we are offering services like "shipping USB sticks" or "dvd-media" with these updates. Please open a case with Red Hat support to discuss possible alternatives. Seeing you posting this comment makes be confident there might be some level of internet connectivity.

Hi Christian. What are the least privileges that successfully download rpms and metadata without flipping bits? (I would like to download rpms and metadata from a limited user account. )

Regarding alternatives: Does Red Hat supply any (metadata) distribution schemes or interfaces which are (loosely) analogous to https://support.microsoft.com/en-ca/kb/913086?

Thanks!

Hi,

What are the least privileges that successfully download rpms and metadata without flipping bits? (I would like to download rpms and metadata from a limited user account. ) I think the user needs to be able to register a system under the account and be able to receive packages. The latter part can be verified in having the user download a package from the customer portal download area. The metadata pieces are included in this. I think the required permissions are at this point not documented more verbosely. One could either try this out (restrict an account as far as possible) or have us investigate and document this with a customer center case.

Regarding alternatives: Does Red Hat supply any (metadata) distribution schemes or interfaces which are (loosely) analogous to https://support.microsoft.com/en-ca/kb/913086? Our media of minor releases could be seen as "iso containing collections of the known fixes". These contain stability and security fixes, and can additionally be used to install new systems. So, for example when a RHEL6.5 was installed, the RHEL6.6GA, 6.7GA and 6.8GA media can serve as iso containing fixes.

As for interfaces to access data about security fixes in an automated way, "yum security" on an installed system or https://access.redhat.com/solutions/4161 (describing our OVAL data) are a good start.

Hi Christian, Attached please find the logic to refresh repository metadata.

Can it be simplified?

Best Regards, Brent

#if [ "$(getfacl -n /etc/pki/entitlement/*-key.pem |grep group:####:r-- |wc -l)" != "1" ]; then exit 1; fi
#yum clean all
#rm -rf upstreamrepo
#reposync --gpgcheck -l -u --repoid=upstreamrepo -d --downloadcomps --download-metadata --download_path `pwd` | gzip -9 -c > upstreamrepo.url.txt.gz
#createrepo -v  `pwd`/<upstreamrepo> -g comps.xml 
#pushd upstreamrepo
#cp *updateinfo.xml.gz repodata/updateinfo.xml.gz
#gunzip repodata/updateinfo.xml.gz
#modifyrepo `pwd`/repodata/updateinfo.xml `pwd`/repodata/
#popd 

Hi, did you verify if createrepo option --update could help you here?

There is nothing to update.

#createrepo --update -v  `pwd`/<upstreamrepo> -g comps.xml 

Scanning old repo data
Could not find valid repo at: /upstreamrepo/

Hello,

If I'm following this correctly, I believe you are trying to include the security data in your local repository yes? If you haven't seen this it may help. https://access.redhat.com/solutions/55654

You want the updatinfo.xml.gz to be in the repodata folder created by running createrepo after the reposync.

Then you will want to use modifyrepo as shown below to apply the security data to the repo.

the updateinfo.xml.gz file should be put in //repodata/

   # cd /var/cache/yum/<arch>/<release>/<upstreamrepo>/
   # cp *updateinfo.xml.gz /<upstreamrepo>/repodata/updateinfo.xml.gz
   # gzip -d /<upstreamrepo>/repodata/updateinfo.xml.gz
   # modifyrepo /<upstreamrepo>/repodata/updateinfo.xml /<upstreamrepo>/repodata/

Once all this is done, remember that clients connecting to this should run "yum clean all" to ensure they remove the cached metadata and get the new metadata.

Hi John. Thanks for the pointer to https://access.redhat.com/solutions/55654. 189533 helps as well.

Question: Under what circumstances do the *-updateinfo file(s) below differ?

/upstreamrepo/*-updateinfo.xml.gz 
/var/cache/yum/<arch>/<release>/<upstreamrepo>/*-updateinfo.xml.gz 

For the local repository server, when it connects it will store the updatinfo.xml.gz file in: /var/cache/yum////*-updateinfo.xml.gz

That file needs to be taken from there, and put into the directory where you ran createrepo. Then modify repo will put the data in place. So if I understand the question correctly, These should be the same files. The main difference I saw is that "updatinfo.xml" needs to be put in the repodata folder of the local repo. //repodata/updatinfo.xml

Hi John. You are correct, updateinfo.xml should be in the repodata folder.

Question. Where is the XML schema definition file for updateinfo?

Thanks! Brent

Hi John and Christian

I see that comps.xml is verified by comps.rng. Where can I find the other schema definitions?

https://bugzilla.redhat.com/show_bug.cgi?id=1296282

https://rhn.redhat.com/errata/RHBA-2016-2187.html

I can not answer that from the top of my head. If it is possible for you to open a case at the customer center, with the details around the issue, then this would probably the best way to answer this question.

Hi Christian.

Hi, did you verify if createrepo option --update could help you here?

Good tip! Thanks

Cheers, could you maybe add rough numbers for the expected sizes of the respective repositories?

Would depend on whether you're doing a full sync or using the "only latest" option. Having done a reposync, today, of the 6-Server channel, using the "only latest" option cut the download from 17,000+ RPMs to a little over 4,000.

Not easy to do this reliably, I would do a full fetch, take note of the size, and then look at the growth over some months. Atleast one minor release should also be part of the observation, as these are also introducing new features (for releases in production phase 1), and bring many new packages. From that one could draw estimations for the consumtion. In production phase 2 and later, the size will grow much slower.

Is it possible to use a single subscribed Centos 7(preferably) or RHEL 7 server to host base and update package repos for RHEL5-7? We have no interest in Satellite server just want a simple network local yum repo for RHEL OS updates like we have for CentOS.

"hosting" repos for all of these via protocols like http or nfs could be done on a single system like a Centos7 box. Yet, fetching does to my knowledge require one "RHEL" per architecture and per major line. So for 32bit/64bit for rhel5/6/7, one can setup KVM guests with these releases for fetching, and configure the host to offer the fetched packages. In that config the host could also be RHEL7/64bit directly for fetching this release, and 4 VMs could be used to cover the rhel5/6 flavours.

Thanks, that is what i ended up doing building a sync server using RHEL 6 and 7 and subscribing them in order to provide RHEL repos internally.

Is there an easy way to grab just the comps.xml?

Not aware of a way in style of "wget ...comps.xml" to get the latest version. Maybe the versions which are on the RHEL media are ok for you. The ordering of packages into groups is not changing very often.

The note about security repositories requiring a different process is unclear (at least to me).

Doesn't the rhel-7-server-rpms channel include security updates for the packages in that channel when they're released?

Also, the link to that separate process goes to a page that only references RHEL5 and RHEL6 -- does it still apply to RHEL7? It would appear NOT, because the updateinfo.xml.gz doesn't exist in my RHEL7 system, although I do have files with suspiciously similar names, like this one:

/var/cache/yum/x86_64/7Server/rhel-7-server-thirdparty-oracle-java-rpms/0afde39e01bc126f3d3335a6adddc87cc50e46c6ebfdd6574e8193e2aeedf14c-updateinfo.xml.gz

It is my understanding that this only does a one-time copy of the repo data from the source repo and that if you want to have it automatically update the repository data on a regular basis you need to set up cron/acron/... with a script or individual commands that perform reposync and "createrepo --update" for each of the repos.
It could be useful to others to suggest that automated tasks/a script be set up and an example of the commands to run for each repo. I did not do the initial set up's of our repo server and we did not use the path recommended in this article, however it also appears that a web server nor an ftp server are a requirements for either yum-utils or createrepo to be installed. It might be useful to mention something like a webserver or ftpserver need to be installed and configured; and that if you install apache with an http based repos and put the repos in /var/www/html as instructed no additional configuration should be needed. Thank you.

Thank you, hints on this have been incorporated. We did this not until now, because there are so many options for distribution. Besides HTTP(S)/FTP/NFS, distributing the repos with rsync/ssh or pulp is also not uncommon.

In reviewing the article I think that waiting to mention --update as a note under Diagnostic Steps where it could easily be missed by a rushing sysadmin will cause the sysadmins to rely on this for days to possibly months before they realize that somethings is wrong, and then having them open an after the fact support ticket or wasting a bunch of time trying to figure out why the local repo is not providing update content. Maybe it should be moved up as an optional step without specific details as it will vary greatly by preference and number of repositories that are mirrored. Also this may be my over interpreting things but to me "local mirror," as contained in the Issue section, makes me think of something that can be accessed by more than just the box itself, using most likely http, https, ftp or sftp even though other protocols will work. I would find it would be much more end user friendly if the note "Above commands create a local copy ... or NFS (nfs-utils package)" were inline before the "How to create a .repo file" link; along with some wording about "For other boxes to access the repo you may need to;" and that "choosing which distribution method to use and how to configure them is outside the scope of this document." Mention something like "Though an apache http install and it's configuration should be relatively easy due to the repositories being placed in /var/www/html/” could be especially useful since the linked repo configuration page is a very simple page with minimal information and the only example of a repo file has http for the base url. With comments like these inline or maybe in the Resolution section like the fact that the box has to be subscribed to the channel it would be much clearer that additional steps are needed. Thank you for the changes you have already made and for considering the following comments ideas for the main article.

Thanks.

Hi, I have subscription for Red Hat Enterprise Linux for SAP Applications, how can I sync it locally?

This local repository is not minor version specific.

When you use reposync against a repository, it will sync all available packages. The only way to version lock the local repo is to version lock the server running "reposync". Then it will only have access to the minor release specified.

You will need to keep this is in a separate folder from the main repo

Dear Experts,

When we create and offline repo from a server that have valid subscription does the client that are using these offline repo will need a valid subscription too? does it legal to use the the offline repo for a production machine that does not have a subscription from terms and condition point of view? (self support)

I could not found anything of the lisense agreement

Regards Thanks!

I would like to know if there is a possibility of creating repositories of version 6.x and 7.x on a single server with version 6.9.

You will need one registered system for every Major Version or Variant repository you need to sync. You can sync the RHEL 7 packages then move them to the RHEL 6 machine in order to host them on one system.

..or have the rhel6 and rhel7 guests fetch onto a common filesystem, for example an NFS share. One could also run a rhel7 on bare metal, have the rhel6 run as KVM guest, and consolidate the files on an NFS file system hosted by the rhel7 hypervisor.