Subscription-manager for the former Red Hat Network User: Part 7 - understanding the Red Hat Content Delivery Network
alternate title: disconnected customers like nice things too.
Overview
The Red Hat Content Delivery Network (henceforth known as the CDN) is the source of content for Satellite 6. Understanding
This document aims to document
- What the Red Hat CDN is
- How to mirror it.
- How to leverage many of the tools in the Satellite 6 product to easily mirror or copy it for disconnected usage
What is the Red Hat CDN?
The Red Hat Content Delivery Network, nominally accessed via cdn.redhat.com is a geographically distributed series of static webservers, which contain content and errata that is designed to be consumed by systems. This content can be consumed directly (such as via a system registered via Red Hat Subscription Management) OR mirrored via on premise solution, such as Red Hat Satellite 6. The Red Hat Content Delivery network is protected by x.509 certificate authentication, to
ensure that only valid users can access it.
In the case of a system registered to Red Hat Subscription Management, the attached subscriptions govern which subset of the CDN the system can access. In the case of Satellite 6, the subscriptions that are attached to the subscription manifest govern which subset of the CDN the system can access.
Directory Structure of the CDN.
Now that you understand what the CDN is, let's take a look at a CDN mirror to see what the directory structure looks like:
$ tree -d -L 11
└── content
├── beta
│ └── rhel
│ └── server
│ └── 7
│ └── x86_64
│ └── sat-tools
│ └── 6
└── dist
└── rhel
└── server
└── 7
├── 7.2
│ └── x86_64
│ └── kickstart
└── 7Server
└── x86_64
└── os
This directory structure is important and has the following meaning
- Top-level directory (always named content)
- Second Level Directory (What is the lifecycle of this content? Common directories include beta (for Beta code), dist (for Production Bits) and eus (For Extended Update Support bits))
- Third Level Directory (which product. Usually rhel for Red Hat Enterprise Linux)
- Fourth Level Directory (Which Variant of the product. For Red Hat Enterprise Linux this includes server, workstation, and computenode )
- Fifth Level Directory (Major version, such as 5,6, or 7)
- Sixth Level Directory (Release version such as 7.0, 7.1, and 7Server)
- Seventh Level Directory (Base architecture, such as i386 or x86_64 )
- Eighth Level Directory (repository name such as kickstart, optional, rhscl, etc). Some components have additional subdirectories, and those may vary.
This directory structure is also used in the subscription manifest. We can look at a subscription manifest to determine which directories of the CDN each subscription has access to.
NOTE: The output below has been shortened for brevity.
rct cat-manifest export.zip
+-------------------------------------------+
Manifest
+-------------------------------------------+
General:
Server:
Server Version: 0.9.51.15-1
Date Created: 2016-09-14T14:27:26.081+0000
Creator: [REDACTED]
Consumer:
Name: [REDACTED]
UUID: [REDACTED]
Type: satellite
Subscription:
Name: Red Hat Enterprise Linux Server, Standard (Physical or Virtual Nodes)
Quantity: 100
Created: 2016-07-06T00:42:43.000+0000
Start Date: 2016-07-04T04:00:00.000+0000
End Date: 2017-07-04T03:59:59.000+0000
Service Level: Standard
Service Type: L1-L3
Architectures: x86_64,ppc64le,ppc64,ia64,ppc,s390,x86,s390x
SKU: RH00004
Contract: [REDACTED]
Order: [REDACTED]
Account: [REDACTED]
Virt Limit:
Requires Virt-who: False
Entitlement File: export/entitlements/8a99f98355af32300155bda82c721c90.json
Certificate File: export/entitlement_certificates/4019380680377493250.pem
Certificate Version: 3.2
Provided Products:
69: Red Hat Enterprise Linux Server
176: Red Hat Developer Toolset (for RHEL Server)
180: Red Hat Beta
201: Red Hat Software Collections (for RHEL Server)
205: Red Hat Software Collections Beta (for RHEL Server)
240: Oracle Java (for RHEL Server)
271: Red Hat Enterprise Linux Atomic Host
272: Red Hat Enterprise Linux Atomic Host Beta
273: Red Hat Container Images
274: Red Hat Container Images Beta
317: dotNET on RHEL (for RHEL Server)
318: dotNET on RHEL Beta (for RHEL Server)
Content Sets:
/content/dist/rhel/server/5/$releasever/$basearch/cf-tools/1/os
/content/dist/rhel/server/5/$releasever/$basearch/cf-tools/1/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/debug
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/debug
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/os
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/debug
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/os
/content/dist/rhel/server/5/$releasever/$basearch/devtoolset/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/iso
/content/dist/rhel/server/5/$releasever/$basearch/kickstart
/content/dist/rhel/server/5/$releasever/$basearch/oracle-java/iso
/content/dist/rhel/server/5/$releasever/$basearch/oracle-java/os
/content/dist/rhel/server/5/$releasever/$basearch/oracle-java/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/os
/content/dist/rhel/server/5/$releasever/$basearch/productivity/debug
/content/dist/rhel/server/5/$releasever/$basearch/productivity/os
/content/dist/rhel/server/5/$releasever/$basearch/productivity/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/rh-common/debug
/content/dist/rhel/server/5/$releasever/$basearch/rh-common/iso
/content/dist/rhel/server/5/$releasever/$basearch/rh-common/os
/content/dist/rhel/server/5/$releasever/$basearch/rh-common/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/debug
/content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/os
/content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/source/SRPMS
/content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/debug
/content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/iso
/content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/os
/content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/source/SRPMS
/content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/debug
/content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/os
/content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/source/SRPMS
/content/dist/rhel/server/6/$releasever/$basearch/debug
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/debug
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/os
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/source/SRPMS
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/debug
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/os
/content/dist/rhel/server/6/$releasever/$basearch/devtoolset/source/SRPMS
/content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/debug
/content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/os
/content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/source/SRPMS
/content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/debug
/content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/iso
/content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/os
/content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/source/SRPMS
/content/dist/rhel/server/7/$releasever/$basearch/rhs-client/debug
/content/dist/rhel/server/7/$releasever/$basearch/rhs-client/os
/content/dist/rhel/server/7/$releasever/$basearch/rhs-client/source/SRPMS
/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/containers
/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/debug
/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/iso
/content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/os
Listing files
Every directory from the Top-Level Directory (content) to the Eighth Level Directory (optional, rhscl , etc) MUST contain a plain-text file, named listing, which contains, one per line, a listing of the subdirectories that the current directory contains. These are used by Red Hat Satellite 6 to determine which content does the CDN contain. See Below:
[root@cdn content]# ls
beta dist eus listing
[root@cdn content]# cat listing
beta
dist
eus
[root@cdn content]# cd dist
[root@cdn dist]# ls
cf-me listing rhel
[root@cdn dist]# cat listing
cf-me
rhel
If the listing files are incorrect, you will NOT see the proper repositories as being available for synchronization.
Connected versus Disconnected Satellites.
Many Satellite users run Red Hat Satellite 6 in disconnected or Airgapped environments, which cannot reach cdn.redhat.com. The process of synchronizing a Satellite is exactly the same for these users. The only difference between a connected Satellite (which can reach cdn.redhat.com) and a disconnected Satellite (which cannot) is:
- The disconnected satellite has a CDN URL defined which is NOT cdn.redhat.com
- The administrator has to create a local mirror of cdn.redhat.com that the Satellite can synchronize from.
Ok, so how do we build a mirror of cdn.redhat.com?
Tools for building a CDN mirror.
Effectively supporting a disconnected Satellite depends on being able to effectively build a mirror of cdn.redhat.com locally.
There are a number of tools available to mirror cdn.redhat.com, which can be used for the task.
- Satellite 6.2 Inter-Satellite Sync (ISS).
- Content ISOs.
- katello-disconnected.
- reposync
Satellite 6.2 Inter-Satellite Sync
Included in Red Hat Satellite 6.2 is the Inter-Satellite Sync feature, described in its Feature Overview, which allows an administrator to export a repository, a content-view (including the special 'Default Organization View' which represents all of the Red Hat content in the Library lifecycle environment), or a repository within a content view. Satellite 6.2 allows this export to disk or ISO, and it can be done on a full or incremental basis.
Pros:
- incremental export allows the administrator the ability to quickly respond to high priority errata quickly.
- Built into Satellite's workflows and tooling such as Hammer, allowing easy automation
- Easy setup & maintenance by not having to 'roll your own' tooling
Cons:
- Additional expense of managing an additional internet connected Satellite.
Content ISOs
Content ISOs, available for download on the Customer Portal are effectively exports of a selected subset of repositories from cdn.redhat.com and packaged in ISO format for download.
Pros:
- Second Satellite not required to download and export content. Can run a completely disconnected Satellite that doesn't retrieve content from the CDN.
Cons:
- ISOs are only released ~ 6 weeks, making dealing with high priority security or bugfix errata difficult.
- Not every Red Hat product is released via content ISOs.
katello-disconnected
Katello-disconnected was a tool shipped with Satellite 6.0 & 6.1, which was a special purpose server, which could mirror and export content from the CDN to populate a disconnected Satellite. While still supported, it is deprecated, will receive no new features, and is superceded by Satellite 6.2's ISS feature.
reposync
You can use reposync with a system registered to Red Hat Subscription management to mirror repositories to make a CDN mirror. We generally don't recommend this as it is labor intensive (you have to mirror each repo one by one AND create the required directory structure), but it works in a pinch.
Buiding a CDN mirror - A practical example using Content ISOs
In this example, we'd like to setup our CDN mirror using content ISOs. This example, I've downloaded both the RHEL5 & RHEL6 Server ISOs and have copied them to a server.
Make a directory to hold the CDN content
#mkdir -p /srv/www/html/pub/cdnroot/
Make a directory to be a temporary mountpoint while the content ISOs are loopback mounted
#mkdir /mnt/iso
Change to the directory that has the RHEL6 ISOs
# cd /tmp/ISO/RHEL6
# ls
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-01.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-02.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-03.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-04.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-05.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-06.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-07.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-08.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-09.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-10.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-11.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-12.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-13.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-14.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-15.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-16.iso*
Now let's extract the content ISOs to our directory
# for i in *.iso; do mount -o loop "$i" /mnt/iso; /bin/cp -av /mnt/iso/* /srv/www/html/pub/cdnroot/; umount /mnt/iso; done
And now let's look at the cdnroot directory
# cd /srv/www/html/pub/cdnroot
# tree -d
.
└── content
└── dist
└── rhel
└── server
└── 6
└── 6Server
└── x86_64
├── devtoolset
│ ├── 2
│ │ └── os
│ │ └── repodata
│ └── os
│ └── repodata
├── extras
│ └── os
│ └── repodata
├── optional
│ └── os
│ └── repodata
├── oracle-java
│ └── os
│ └── repodata
├── os
│ └── repodata
├── rh-common
│ └── os
│ └── repodata
├── rhscl
│ └── 1
│ └── os
│ └── repodata
├── sat-tools
│ └── 6.1
│ └── os
│ └── repodata
└── supplementary
└── os
└── repodata
As you can see, this content ISO contains the correct directory structure, and can be used to synchronize content into a Satellite 6 installation. Now let's import our RHEL5 content ISOs:
# cd /tmp/ISO/RHEL5
# ls
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-01.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-02.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-03.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-04.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-05.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-06.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-07.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-08.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-09.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-10.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-11.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-12.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-13.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-14.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-15.iso*
And extract the RHEL5 ISOs now:
for i in *.iso; do mount -o loop "$i" /mnt/iso; /bin/cp -av /mnt/iso/* /srv/www/html/pub/cdnroot/; umount /mnt/iso; done
Repeat the tree command.
tree -d
.
└── content
└── dist
└── rhel
└── server
├── 5
│ └── 5Server
│ └── x86_64
│ ├── devtoolset
│ │ ├── 2
│ │ │ └── os
│ │ │ └── repodata
│ │ └── os
│ │ └── repodata
│ ├── oracle-java
│ │ └── os
│ │ └── repodata
│ ├── os
│ │ └── repodata
│ ├── sat-tools
│ │ └── 6.1
│ │ └── os
│ │ └── repodata
│ └── supplementary
│ └── os
│ └── repodata
└── 6
└── 6Server
└── x86_64
├── devtoolset
│ ├── 2
│ │ └── os
│ │ └── repodata
│ └── os
│ └── repodata
├── extras
│ └── os
│ └── repodata
├── optional
│ └── os
│ └── repodata
├── oracle-java
│ └── os
│ └── repodata
├── os
│ └── repodata
├── rh-common
│ └── os
│ └── repodata
├── rhscl
│ └── 1
│ └── os
│ └── repodata
├── sat-tools
│ └── 6.1
│ └── os
│ └── repodata
└── supplementary
└── os
└── repodata
While the content is properly copied to the cdnroot directory, there is a small issue. Each content ISO provides its own listing files, which only contain the metadata for its portion of the CDN. This means that the listing files will only reflect the last content ISO that was copied. In this case, these are the listing files from the RHEL5 ISOs as shown below:
# cd /srv/www/html/pub/cdnroot/content/dist/rhel/server/
# ls
5/ 6/ listing
# cat listing
5
We'll need to either:
- figure out a means to merge the listing files while importing.
- create new listing files, after the import is completed.
We are going to do the latter.
Updating the listing files.
Firstly, we need to delete the existing listing files.
# find /srv/www/html/pub/cdnroot/ -iname listing -delete
Next, to create new listing files, we can create them by hand, or leveraging automation to do it. I've created the makeCDNListingFiles.py script to create a new set of listing files. Let's download and take a look at the file.
# wget -q https://raw.githubusercontent.com/sideangleside/makeCDNListingFiles/master/makeCDNListingFiles.py
# chmod +x makeCDNListingFiles.py
# ./makeCDNListingFiles.py
Must specify directory of expanded CDN content: see usage
Usage: makeCDNListingFiles.py [options]
Options:
-h, --help show this help message and exit
-c CDNEXPORTDIR, --cdn-export-dir=CDNEXPORTDIR
Directory of expanded CDN content
-v, --verbose Verbose output
Example usage: ./makeCDNListingFiles.py -c /var/www/html/pub/sat-import/
And run the script against our export.
#./makeCDNListingFiles.py -c /srv/www/html/pub/cdnroot
Now we can host this export via a webserver of our choosing, and configure a Satellite 6 installation to use it as its CDN URL.
Configuring your Satellite to use your local CDN
In my environment, I've configured my web server (webserver.example.com) to export /srv/www/html/pub/ as its DocumentRoot, meaning that I can access the exported CDN content via http://webserver.example.com/cdnroot/.
TIP: Satellite servers export /var/www/html/pub/ as a place to store public content. You can copy your CDN export there and not need an additional web server to host the content.
Now, we need to switch our CDN URL so that our Satellite can synchronize from webserver.example.com. This can be done in one of two ways:
- In the UI via Content -> Red Hat Subscriptions -> Manage Manifest, selecting (and changing the 'Red Hat CDN URL')
- Via the hammer CLI:
hammer subscription upload \
--organization Example \
--file /root/export.zip \
--repository-url http://webserver.example.com/cdnroot/
Now, we can synchronize repositories, leveraging our CDN export.
Summary
Knowing how to mirror the Red Hat CDN is a required technique for disconnected Satellite users. It can also be used to quickly populate lab/testing Satellites.
Further Reading
- Subscription-manager for the former Red Hat Network User: Part 1
- Subscription-manager for the former Red Hat Network User: Part 2 - Subscription-manager learns grep
- Subscription-manager for the former Red Hat Network User: Part 3 - Understanding virt-who
- Subscription-manager for the former Red Hat Network User: Part 4 - Understanding Subscription Manifests
- Subscription-manager for the former Red Hat Network User: Part 5 - Working with subscriptions that require virt-who
- Subscription-manager for the former Red Hat Network User: Part 6 - understanding and improving the renewal experience
About The Author
Red Hat
Guru
3050 points
Comments