Chapter 4. Importing Red Hat Content

Our Satellite Server now has the necessary subscription information imported. We are ready to add content to our system. For this chapter, we explore the concept of a Definitive Media Library (DML) and how to synchronize content to create our DML.

4.1. Creating a Definitive Media Library

A DML is a repository that stores and protects the definitive, authorized versions of software and configurations. In other words, the DML acts as a master version of any content imported into Satellite. This includes Red Hat content, such as RPM files, kickstart trees, and ISO images. As mentioned in Section 1.1, “Overview of Red Hat Satellite 6 Content Management”, Red Hat Satellite 6 stores and manages content in a DML.

4.2. Using Products and Repositories in Satellite

In Satellite, we use the concept of a Product as an organizational unit to group multiple repositories together. Such repository collections are analogous to the concept of real life products. For example, if we view Red Hat Enterprise Linux Server as a Product in Satellite, the repositories for that product might consist of different versions (6.0, 6.1, 7.0), different architectures (i386, x86_64, s390x, arm), and different add-ons (Optional repositories, Supplementary repositories, Virt V2V tools). This unifies all related repositories within the DML. Using Products ensures repositories that depend on each other are synchronized together. For Red Hat repositories, products are created automatically after enabling the repository.

In this chapter, our aim is to start creating our DML using Red Hat content. To do this, we synchronize our DML with Red Hat’s Products and their repositories.

4.3. Synchronizing Content

When you select which repositories form the DML, Satellite Server synchronizes its own repositories with the repositories on the Red Hat CDN. This ensures that Satellite Server retains an exact copy of Red Hat’s repositories as a part of its DML. Satellite Server fetches this repository information and stores it on Satellite Server’s file system. After an initial synchronization, you can create a synchronization plan that checks to ensure the repositories in the DML is up to date with the CDN’s repositories.

It is possible to perform an initial synchronization using ISO images. See Appendix C, Importing Content ISOs into a Connected Satellite for more information on using Content ISOs. For locations with bandwidth limitations, using an On Demand or Background download policy as described below might be quicker than downloading and importing Content ISOs.

4.4. Using Download Policies

Red Hat Satellite provides multiple download policies for synchronizing RPM content. For example, you might aim to save time and only download the content metadata while deferring the actual content download for later.

Satellite Server offers the following policies:

  • Immediate — Satellite Server downloads all metadata and packages during synchronization.
  • On Demand — Satellite Server only downloads the metadata during synchronization. Satellite Server only fetches and stores packages on the file system when Capsules or directly connected clients request them. This setting has no effect if you set a corresponding repository on a Capsule to Immediate because Satellite Server is forced to download all the packages.
  • Background — Satellite Server creates a background task to download all packages after the initial synchronization.

The latter two policies act as a Lazy Synchronization feature because they save time synchronizing content. The lazy sync feature must only be used for yum repositories. The packages can be added to Content Views and promoted to life cycle environments as normal.

Capsule Server offers the following policies:

  • Immediate — Capsule Server downloads all metadata and packages during synchronization. Do not use this setting if the corresponding repository on Satellite Server is set to On Demand as Satellite Server is forced to download all the packages.
  • On Demand — Capsule Server only downloads the metadata during synchronization. The Capsule only fetches and stores packages on the file system when directly connected clients request them.

    Note

    When using an On Demand download policy, content is downloaded from Satellite Server if it is not available on the Capsule Server.

  • Background — Capsule Server creates a background task to download all packages after the initial synchronization.
  • Inherit — Capsule Server inherits the download policy for the repository from the corresponding repository on Satellite Server.

These policies are not available if a Capsule was installed or updated with --enable-foreman-proxy-plugin-pulp set to false.

4.5. Selecting Red Hat Repositories to Synchronize

The first step in selecting the repositories to synchronize is to identify the product that contains the repository, then enable that repository based on release version and base architecture.

Important

If using a Disconnected Satellite, you need to import the Content ISOs for Red Hat Satellite and change the CDN URL on Satellite Server before synchronizing content. For more information, see Appendix B, Importing Content ISOs into a Disconnected Satellite.

For Web UI Users

Navigate to Content > Red Hat Repositories. This displays a set of tabs for different content types. Each tab contains a list of products. The RPMs tab is the default tab upon loading this page. This tab contains list of all subscribed products that provide RPM content.

The relationship between products and specific repositories is connected through a cascading hierarchy. Select a product and this opens a list of repository sets for that product. Select a repository set and this opens a list of repositories that you can enable. For our scenario, select Red Hat Enterprise Linux Server, then Red Hat Enterprise Linux 7 Server (RPMs), then enable Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server. This enables the latest RPM files for Red Hat Enterprise Linux 7.

Note

The difference between associating Red Hat Enterprise Linux Operating System with either 7 Server repositories or 7.X repositories is that 7 Server repositories contain all the latest updates while Red Hat Enterprise Linux 7.X repositories stop getting updates after the next minor version release. Note that Kickstart repositories only have minor versions.

For CLI Users

The relationship between products and repositories is the same. To search for your product, enter the following command:

# hammer product list --organization "ACME"

List the repository set for the product:

# hammer repository-set list \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

This displays the repositories in the product’s repository set, including their name and ID number. Enable the repository using either the name or ID number. Also include the release version (7Server) and base architecture (x86_64). For example:

# hammer repository-set enable \
--name "Red Hat Enterprise Linux 7 Server (RPMs)" \
--releasever "7Server" \
--basearch "x86_64" \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

For this scenario, use either the Web UI or the CLI to enable the following repositories for ACME:

RepositoryTypeDescription

Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server

RPM

Repositories for the latest version of Red Hat Enterprise Linux 7. Use the 7 Server repository instead of the 7.2 repository so that you receive continuous package updates.

Red Hat Satellite Tools 6.3 for RHEL 7 Server RPMs x86_64

RPM

The Satellite Tools repository, which contains system management agents and tools for client systems. After provisioning a new system, Satellite installs tools such as katello-agent and Puppet to the client. Use the 7 Server repository instead of the 7.2 repository so that you receive continuous package updates.

Red Hat Enterprise Linux 7.2 Kickstart x86_64 7Server

Kickstart

The kickstart tree for Red Hat Enterprise Linux 7.2. Use this as installation media for provisioning new systems over PXE.

These repositories provide some initial content for our scenario’s DML. You can select more repositories based upon your needs.

Note

For this scenario, all repositories use x86_64 as the base architecture.

4.6. Synchronizing Red Hat Repositories

We have enabled specific repositories to form our initial DML. Now we synchronize the repositories with the Red Hat CDN’s repositories.

For Web UI Users

Navigate to Content > Products and select Red Hat Enterprise Linux Server. This displays all enabled repositories in our product. Select all repositories and click Sync Now. You can also view the progress of the synchronization in the Web UI. Navigate to Content > Sync Status and expanding the Product/Repository tree (or click Expand All).

For CLI Users

Synchronize the enabled repositories in the Red Hat Enterprise Linux Server product:

# hammer product synchronize \
--name "Red Hat Enterprise Linux Server" \
--organization "ACME"

You can also synchronize each repository individually. List all repositories in the product, then synchronize using the ID number for the corresponding repositories. For example:

# hammer repository list \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"
# hammer repository synchronize \
--name "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

The synchronization duration depends on the size of each repository and the speed of your network connection. The following table provides estimates of how long it would take to synchronize content, depending on the available Internet bandwidth:

 Single Package (10Mb)Minor Release (750Mb)Major Release (6Gb)

256 Kbps

5 Mins 27 Secs

6 Hrs 49 Mins 36 Secs

2 Days 7 Hrs 55 Mins

512 Kbps

2 Mins 43.84 Secs

3 Hrs 24 Mins 48 Secs

1 Day 3 Hrs 57 Mins

T1 (1.5 Mbps)

54.33 Secs

1 Hr 7 Mins 54.78 Secs

9 Hrs 16 Mins 20.57 Secs

10 Mbps

8.39 Secs

10 Mins 29.15 Secs

1 Hr 25 Mins 53.96 Secs

100 Mbps

0.84 Secs

1 Min 2.91 Secs

8 Mins 35.4 Secs

1000 Mbps

0.08 Secs

6.29 Secs

51.54 Secs

A manual synchronization is often required for the initial content import into the DML. However, it is recommended to create a synchronization plan to ensure that our DML updates on a regular basis.

Note

You can change the download policy for a Red Hat repository. Select a repository in the Red Hat Enterprise Linux Server product and navigate to the Download Policy field. If using the CLI, enter the hammer repository update command with the --download-policy option.

4.6.1. Recovering a Repository

In the case of repository corruption, you can recover it by using an advanced synchronization, which has three options:

  • Optimized Sync - it synchronizes the repository bypassing RPMs that have no detected differences from the upstream RPMs.
  • Complete Sync - it synchronizes all RPMs regardless of detected changes. Use this option if specific RPMs could not be downloaded to the local repository even though they exist in the upstream repository.
  • Validate Content Sync - it synchronizes all RPMs and then verifies the checksum of all RPMs locally. If the checksum of an RPM differs from the upstream, it re-downloads the RPM. This option is relevant for yum repositories only. Use this option if you have one of the following errors:

    • Specific RPMs cause a 404 error while synchronizing with yum.
    • Package does not match intended download error, which means that specific RPMs are corrupted.

To synchronize a specific repository with an advanced option:

For Web UI users

  1. Navigate to Content > Products.
  2. Select the product containing the corrupted repository.
  3. Click on the name of a repository you want to synchronize.
  4. Extend the Select Action menu and select Advanced Sync.
  5. Select the option and click Sync.

For CLI users

  1. Obtain a list of repository IDs:

    # hammer repository list --organization "Default Organization"
  2. Synchronize a corrupted repository using the necessary option:

    • For the optimized synchronization:

      # hammer repository synchronize --incremental true --id 1
    • For the complete synchronization:

      # hammer repository synchronize --skip-metadata-check true --id 1
    • For the validate content synchronization:

      # hammer repository synchronize --validate-contents true --id 1

4.6.2. Limiting Synchronization Speed

You can control the speed of synchronization to avoid exhaustion of available bandwidth and prevent other performance issues. This is done by configuring PULP_CONCURRENCY and max_speed parameters. Note that these settings are overwritten on an upgrade. It is recommended to backup changed files prior to an upgrade to be able to restore the configuration.

  1. To control the number of synchronization jobs that run in parallel, configure the PULP_CONCURRENCY parameter in the /etc/default/pulp_workers file. For example, to set the number of jobs that run in parallel to 1, change PULP_CONCURRENCY to 1:

    PULP_CONCURRENCY=1

    By default, on a system with less than 8 CPUs, PULP_CONCURRENCY is set to the number of CPUs. On a system with more than 8 CPUs, it is set to 8.

  2. To set the maximum network speed for synchronizing in bytes per second, configure the max_speed parameter. This parameter must be configured separately for each importer in the /etc/pulp/server/plugins.conf.d/ directory.

    1. For example, to set the maximum speed for synchronizing RPM content to 10 bytes per second, set the "max_speed" parameter in the /etc/pulp/server/plugins.conf.d/yum_importer.json file to 10:

      # cat /etc/pulp/server/plugins.conf.d/yum_importer.json
      {
          "proxy_host": null,
          "proxy_port": null,
          "proxy_username": null,
          "proxy_password": null,
          "max_speed": 10
      }
    2. Verify the syntax of the file after editing:

      # json_verify < /etc/pulp/server/plugins.conf.d/yum_importer.json
      JSON is valid
  3. Restart Satellite services to apply the changes:

    # katello-service restart

4.7. Creating a Synchronization Plan

A synchronization plan checks and updates the content in your DML at a regularly scheduled date and time. Red Hat Satellite 6 provides users with the ability to create a synchronization plan and assign products to it.

For Web UI Users

Navigate to Content > Sync Plans and click New Sync Plan. The UI provides a set of fields where you can input details about your synchronization plan:

  • Name - A plain text name for the plan. Enter Example Plan.
  • Description - A plain text description of the plan. Enter Example Plan for ACME’s repositories.
  • Interval - Defines when to run the synchronization. Select daily.
  • Start Date and Start Time - Defines when to run the synchronization. We already completed a synchronization today, so set the synchronization for tomorrow at 1:00 (1AM).

Click Save to create a plan. The plan details page displays along with two tabs for Details and Products.

Now add your products. Click the Products tab, then click Add. Select the Red Hat Enterprise Linux Server product and click Add Selected.

For CLI Users

To create the synchronization plan, enter the following command:

# hammer sync-plan create \
--name "Red Hat Products 2" \
--description "Example Plan for ACME's Red Hat Products" \
--interval daily \
--sync-date "2016-02-01 01:00:00" \
--enabled true \
--organization "ACME"

Then assign the Red Hat Enterprise Linux Server product to it:

# hammer product set-sync-plan \
--name "Red Hat Enterprise Linux Server" \
--sync-plan "Red Hat Products" \
--organization "ACME"

Now Satellite Server checks its DML content against the Red Hat CDN on a daily basis and keeps its Red Hat repositories up to date.

4.8. Chapter Summary

This chapter showed how to import Red Hat content into ACME’s Satellite Server and keep it up to date through synchronization plans.

The next chapter explores a importing custom content into Satellite Server’s DML. This process is similar to importing Red Hat content except we create and manage custom products.