Warning message

Log in to add comments.

Subscription-manager for the former Red Hat Network User: Part 7 - understanding the Red Hat Content Delivery Network

Rich Jerrido published on 2016-09-20T03:09:42+00:00, last updated 2016-09-20T15:25:26+00:00

alternate title: disconnected customers like nice things too.

Overview

The Red Hat Content Delivery Network (henceforth known as the CDN) is the source of content for Satellite 6. Understanding
This document aims to document

  • What the Red Hat CDN is
  • How to mirror it.
  • How to leverage many of the tools in the Satellite 6 product to easily mirror or copy it for disconnected usage

What is the Red Hat CDN?

The Red Hat Content Delivery Network, nominally accessed via cdn.redhat.com is a geographically distributed series of static webservers, which contain content and errata that is designed to be consumed by systems. This content can be consumed directly (such as via a system registered via Red Hat Subscription Management) OR mirrored via on premise solution, such as Red Hat Satellite 6. The Red Hat Content Delivery network is protected by x.509 certificate authentication, to
ensure that only valid users can access it.

In the case of a system registered to Red Hat Subscription Management, the attached subscriptions govern which subset of the CDN the system can access. In the case of Satellite 6, the subscriptions that are attached to the subscription manifest govern which subset of the CDN the system can access.

Directory Structure of the CDN.

Now that you understand what the CDN is, let's take a look at a CDN mirror to see what the directory structure looks like:

$ tree -d -L 11
└── content
    ├── beta
    │   └── rhel
    │       └── server
    │           └── 7
    │               └── x86_64
    │                   └── sat-tools
    │                       └── 6
    └── dist
        └── rhel
            └── server
                └── 7
                ├── 7.2
                │   └── x86_64
                │       └── kickstart
                └── 7Server
                    └── x86_64
                        └── os

This directory structure is important and has the following meaning

  • Top-level directory (always named content)
    • Second Level Directory (What is the lifecycle of this content? Common directories include beta (for Beta code), dist (for Production Bits) and eus (For Extended Update Support bits))
    • Third Level Directory (which product. Usually rhel for Red Hat Enterprise Linux)
      • Fourth Level Directory (Which Variant of the product. For Red Hat Enterprise Linux this includes server, workstation, and computenode )
      • Fifth Level Directory (Major version, such as 5,6, or 7)
        • Sixth Level Directory (Release version such as 7.0, 7.1, and 7Server)
        • Seventh Level Directory (Base architecture, such as i386 or x86_64 )
          • Eighth Level Directory (repository name such as kickstart, optional, rhscl, etc). Some components have additional subdirectories, and those may vary.

This directory structure is also used in the subscription manifest. We can look at a subscription manifest to determine which directories of the CDN each subscription has access to.

NOTE: The output below has been shortened for brevity.

rct cat-manifest export.zip
+-------------------------------------------+
    Manifest
+-------------------------------------------+

General:
    Server:
    Server Version: 0.9.51.15-1
    Date Created: 2016-09-14T14:27:26.081+0000
    Creator: [REDACTED]

Consumer:
    Name: [REDACTED]
    UUID: [REDACTED]
    Type: satellite
Subscription:
    Name: Red Hat Enterprise Linux Server, Standard (Physical or Virtual Nodes)
    Quantity: 100
    Created: 2016-07-06T00:42:43.000+0000
    Start Date: 2016-07-04T04:00:00.000+0000
    End Date: 2017-07-04T03:59:59.000+0000
    Service Level: Standard
    Service Type: L1-L3
    Architectures: x86_64,ppc64le,ppc64,ia64,ppc,s390,x86,s390x
    SKU: RH00004
    Contract: [REDACTED]
    Order: [REDACTED]
    Account: [REDACTED]
    Virt Limit:
    Requires Virt-who: False
    Entitlement File: export/entitlements/8a99f98355af32300155bda82c721c90.json
    Certificate File: export/entitlement_certificates/4019380680377493250.pem
    Certificate Version: 3.2
    Provided Products:
        69: Red Hat Enterprise Linux Server
        176: Red Hat Developer Toolset (for RHEL Server)
        180: Red Hat Beta
        201: Red Hat Software Collections (for RHEL Server)
        205: Red Hat Software Collections Beta (for RHEL Server)
        240: Oracle Java (for RHEL Server)
        271: Red Hat Enterprise Linux Atomic Host
        272: Red Hat Enterprise Linux Atomic Host Beta
        273: Red Hat Container Images
        274: Red Hat Container Images Beta
        317: dotNET on RHEL (for RHEL Server)
        318: dotNET on RHEL Beta (for RHEL Server)
    Content Sets:
        /content/dist/rhel/server/5/$releasever/$basearch/cf-tools/1/os
        /content/dist/rhel/server/5/$releasever/$basearch/cf-tools/1/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/debug
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/debug
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/os
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/2/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/debug
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/os
        /content/dist/rhel/server/5/$releasever/$basearch/devtoolset/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/iso
        /content/dist/rhel/server/5/$releasever/$basearch/kickstart
        /content/dist/rhel/server/5/$releasever/$basearch/oracle-java/iso
        /content/dist/rhel/server/5/$releasever/$basearch/oracle-java/os
        /content/dist/rhel/server/5/$releasever/$basearch/oracle-java/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/os
        /content/dist/rhel/server/5/$releasever/$basearch/productivity/debug
        /content/dist/rhel/server/5/$releasever/$basearch/productivity/os
        /content/dist/rhel/server/5/$releasever/$basearch/productivity/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/rh-common/debug
        /content/dist/rhel/server/5/$releasever/$basearch/rh-common/iso
        /content/dist/rhel/server/5/$releasever/$basearch/rh-common/os
        /content/dist/rhel/server/5/$releasever/$basearch/rh-common/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/debug
        /content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/os
        /content/dist/rhel/server/5/$releasever/$basearch/rhev-agent/3/source/SRPMS
        /content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/debug
        /content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/iso
        /content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/os
        /content/dist/rhel/server/5/$releasever/$basearch/rhn-tools/source/SRPMS
        /content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/debug
        /content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/os
        /content/dist/rhel/server/6/$releasever/$basearch/cf-tools/1/source/SRPMS
        /content/dist/rhel/server/6/$releasever/$basearch/debug
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/debug
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/os
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/2/source/SRPMS
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/debug
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/os
        /content/dist/rhel/server/6/$releasever/$basearch/devtoolset/source/SRPMS
        /content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/debug
        /content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/os
        /content/dist/rhel/server/6/$releasever/$basearch/insights-client/1/source/SRPMS
        /content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/debug
        /content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/iso
        /content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/os
        /content/dist/rhel/server/7/$releasever/$basearch/rhn-tools/source/SRPMS
        /content/dist/rhel/server/7/$releasever/$basearch/rhs-client/debug
        /content/dist/rhel/server/7/$releasever/$basearch/rhs-client/os
        /content/dist/rhel/server/7/$releasever/$basearch/rhs-client/source/SRPMS
        /content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/containers
        /content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/debug
        /content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/iso
        /content/dist/rhel/server/7/$releasever/$basearch/rhscl/1/os

Listing files

Every directory from the Top-Level Directory (content) to the Eighth Level Directory (optional, rhscl , etc) MUST contain a plain-text file, named listing, which contains, one per line, a listing of the subdirectories that the current directory contains. These are used by Red Hat Satellite 6 to determine which content does the CDN contain. See Below:

[root@cdn content]# ls
beta  dist  eus  listing
[root@cdn content]# cat listing
beta
dist
eus

[root@cdn content]# cd dist
[root@cdn dist]# ls
cf-me  listing  rhel
[root@cdn dist]# cat listing
cf-me
rhel

If the listing files are incorrect, you will NOT see the proper repositories as being available for synchronization.

Connected versus Disconnected Satellites.

Many Satellite users run Red Hat Satellite 6 in disconnected or Airgapped environments, which cannot reach cdn.redhat.com. The process of synchronizing a Satellite is exactly the same for these users. The only difference between a connected Satellite (which can reach cdn.redhat.com) and a disconnected Satellite (which cannot) is:

  • The disconnected satellite has a CDN URL defined which is NOT cdn.redhat.com
  • The administrator has to create a local mirror of cdn.redhat.com that the Satellite can synchronize from.

Ok, so how do we build a mirror of cdn.redhat.com?

Tools for building a CDN mirror.

Effectively supporting a disconnected Satellite depends on being able to effectively build a mirror of cdn.redhat.com locally.
There are a number of tools available to mirror cdn.redhat.com, which can be used for the task.

  • Satellite 6.2 Inter-Satellite Sync (ISS).
  • Content ISOs.
  • katello-disconnected.
  • reposync

Satellite 6.2 Inter-Satellite Sync

Included in Red Hat Satellite 6.2 is the Inter-Satellite Sync feature, described in its Feature Overview, which allows an administrator to export a repository, a content-view (including the special 'Default Organization View' which represents all of the Red Hat content in the Library lifecycle environment), or a repository within a content view. Satellite 6.2 allows this export to disk or ISO, and it can be done on a full or incremental basis.

Pros:

  • incremental export allows the administrator the ability to quickly respond to high priority errata quickly.
  • Built into Satellite's workflows and tooling such as Hammer, allowing easy automation
  • Easy setup & maintenance by not having to 'roll your own' tooling

Cons:
- Additional expense of managing an additional internet connected Satellite.

Content ISOs

Content ISOs, available for download on the Customer Portal are effectively exports of a selected subset of repositories from cdn.redhat.com and packaged in ISO format for download.

Pros:

  • Second Satellite not required to download and export content. Can run a completely disconnected Satellite that doesn't retrieve content from the CDN.

Cons:

  • ISOs are only released ~ 6 weeks, making dealing with high priority security or bugfix errata difficult.
  • Not every Red Hat product is released via content ISOs.

katello-disconnected

Katello-disconnected was a tool shipped with Satellite 6.0 & 6.1, which was a special purpose server, which could mirror and export content from the CDN to populate a disconnected Satellite. While still supported, it is deprecated, will receive no new features, and is superceded by Satellite 6.2's ISS feature.

reposync

You can use reposync with a system registered to Red Hat Subscription management to mirror repositories to make a CDN mirror. We generally don't recommend this as it is labor intensive (you have to mirror each repo one by one AND create the required directory structure), but it works in a pinch.

Buiding a CDN mirror - A practical example using Content ISOs

In this example, we'd like to setup our CDN mirror using content ISOs. This example, I've downloaded both the RHEL5 & RHEL6 Server ISOs and have copied them to a server.

Make a directory to hold the CDN content

#mkdir -p /srv/www/html/pub/cdnroot/

Make a directory to be a temporary mountpoint while the content ISOs are loopback mounted

#mkdir /mnt/iso

Change to the directory that has the RHEL6 ISOs

# cd /tmp/ISO/RHEL6
# ls
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-01.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-02.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-03.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-04.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-05.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-06.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-07.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-08.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-09.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-10.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-11.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-12.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-13.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-14.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-15.iso*
sat-6-isos--rhel-6-server-x86_64-2015-08-12T11.42-16.iso*

Now let's extract the content ISOs to our directory

# for i in *.iso; do mount -o loop "$i" /mnt/iso; /bin/cp -av /mnt/iso/* /srv/www/html/pub/cdnroot/; umount /mnt/iso; done

And now let's look at the cdnroot directory

# cd /srv/www/html/pub/cdnroot
# tree -d
.
└── content
    └── dist
        └── rhel
            └── server
                └── 6
                    └── 6Server
                        └── x86_64
                            ├── devtoolset
                            │   ├── 2
                            │   │   └── os
                            │   │       └── repodata
                            │   └── os
                            │       └── repodata
                            ├── extras
                            │   └── os
                            │       └── repodata
                            ├── optional
                            │   └── os
                            │       └── repodata
                            ├── oracle-java
                            │   └── os
                            │       └── repodata
                            ├── os
                            │   └── repodata
                            ├── rh-common
                            │   └── os
                            │       └── repodata
                            ├── rhscl
                            │   └── 1
                            │       └── os
                            │           └── repodata
                            ├── sat-tools
                            │   └── 6.1
                            │       └── os
                            │           └── repodata
                            └── supplementary
                                └── os
                                    └── repodata

As you can see, this content ISO contains the correct directory structure, and can be used to synchronize content into a Satellite 6 installation. Now let's import our RHEL5 content ISOs:

# cd /tmp/ISO/RHEL5
# ls
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-01.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-02.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-03.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-04.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-05.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-06.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-07.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-08.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-09.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-10.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-11.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-12.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-13.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-14.iso*
sat-6-isos--rhel-5-server-x86_64-2015-08-12T08.44-15.iso*

And extract the RHEL5 ISOs now:

for i in *.iso; do mount -o loop "$i" /mnt/iso; /bin/cp -av /mnt/iso/* /srv/www/html/pub/cdnroot/; umount /mnt/iso; done

Repeat the tree command.

tree -d
.
└── content
    └── dist
        └── rhel
            └── server
                ├── 5
                │   └── 5Server
                │       └── x86_64
                │           ├── devtoolset
                │           │   ├── 2
                │           │   │   └── os
                │           │   │       └── repodata
                │           │   └── os
                │           │       └── repodata
                │           ├── oracle-java
                │           │   └── os
                │           │       └── repodata
                │           ├── os
                │           │   └── repodata
                │           ├── sat-tools
                │           │   └── 6.1
                │           │       └── os
                │           │           └── repodata
                │           └── supplementary
                │               └── os
                │                   └── repodata
                └── 6
                    └── 6Server
                        └── x86_64
                            ├── devtoolset
                            │   ├── 2
                            │   │   └── os
                            │   │       └── repodata
                            │   └── os
                            │       └── repodata
                            ├── extras
                            │   └── os
                            │       └── repodata
                            ├── optional
                            │   └── os
                            │       └── repodata
                            ├── oracle-java
                            │   └── os
                            │       └── repodata
                            ├── os
                            │   └── repodata
                            ├── rh-common
                            │   └── os
                            │       └── repodata
                            ├── rhscl
                            │   └── 1
                            │       └── os
                            │           └── repodata
                            ├── sat-tools
                            │   └── 6.1
                            │       └── os
                            │           └── repodata
                            └── supplementary
                                └── os
                                    └── repodata

While the content is properly copied to the cdnroot directory, there is a small issue. Each content ISO provides its own listing files, which only contain the metadata for its portion of the CDN. This means that the listing files will only reflect the last content ISO that was copied. In this case, these are the listing files from the RHEL5 ISOs as shown below:

# cd /srv/www/html/pub/cdnroot/content/dist/rhel/server/
# ls
5/  6/  listing
# cat listing
5

We'll need to either:

  • figure out a means to merge the listing files while importing.
  • create new listing files, after the import is completed.

We are going to do the latter.

Updating the listing files.

Firstly, we need to delete the existing listing files.

# find /srv/www/html/pub/cdnroot/ -iname listing -delete

Next, to create new listing files, we can create them by hand, or leveraging automation to do it. I've created the makeCDNListingFiles.py script to create a new set of listing files. Let's download and take a look at the file.

# wget -q https://raw.githubusercontent.com/sideangleside/makeCDNListingFiles/master/makeCDNListingFiles.py

# chmod +x makeCDNListingFiles.py
# ./makeCDNListingFiles.py
Must specify directory of expanded CDN content: see usage
Usage: makeCDNListingFiles.py [options]

Options:
  -h, --help            show this help message and exit
  -c CDNEXPORTDIR, --cdn-export-dir=CDNEXPORTDIR
                        Directory of expanded CDN content
  -v, --verbose         Verbose output

Example usage: ./makeCDNListingFiles.py -c /var/www/html/pub/sat-import/

And run the script against our export.

#./makeCDNListingFiles.py -c /srv/www/html/pub/cdnroot

Now we can host this export via a webserver of our choosing, and configure a Satellite 6 installation to use it as its CDN URL.

Configuring your Satellite to use your local CDN

In my environment, I've configured my web server (webserver.example.com) to export /srv/www/html/pub/ as its DocumentRoot, meaning that I can access the exported CDN content via http://webserver.example.com/cdnroot/.

TIP: Satellite servers export /var/www/html/pub/ as a place to store public content. You can copy your CDN export there and not need an additional web server to host the content.

Now, we need to switch our CDN URL so that our Satellite can synchronize from webserver.example.com. This can be done in one of two ways:

  • In the UI via Content -> Red Hat Subscriptions -> Manage Manifest, selecting (and changing the 'Red Hat CDN URL')
  • Via the hammer CLI:
hammer subscription upload \
  --organization Example \
  --file /root/export.zip \
  --repository-url http://webserver.example.com/cdnroot/

Now, we can synchronize repositories, leveraging our CDN export.

Summary

Knowing how to mirror the Red Hat CDN is a required technique for disconnected Satellite users. It can also be used to quickly populate lab/testing Satellites.

Further Reading

English

About The Author

richjerrido's picture

Rich Jerrido

Rich Jerrido, Red Hat Product Manager, is a “doer-of-all-things Red Hat Satellite,” including training, integration, enablement, documentation, and helping to identify product requirements. He serves as a technology expert, frequently speaking in web seminars and at industry events. With mor...
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.