Red Hat Training

A Red Hat training course is available for Red Hat Satellite

Chapter 5. Importing Red Hat Content

Our Satellite Server now has the necessary subscription information imported. We are ready to add content to our system. For this chapter, we explore the concept of a Definitive Media Library (DML) and how to synchronize content to create our DML.

5.1. Creating a Definitive Media Library

A DML is a repository that stores and protects the definitive, authorized versions of software and configurations. In other words, the DML acts as a master version of any content imported into Satellite. This includes Red Hat content, such as RPM files, kickstart trees, and ISO images. As mentioned in Section 1.1, “Overview of Red Hat Satellite 6 Content Management”, Red Hat Satellite 6 stores and manages content in a DML.

5.2. Using Products and Repositories in Satellite

In Satellite, we use the concept of a Product as an organizational unit to group multiple repositories together. Such repository collections are analogous to the concept of real life products. For example, if we view Red Hat Enterprise Linux Server as a Product in Satellite, the repositories for that product might consist of different versions (6.0, 6.1, 7.0), different architectures (i386, x86_64, s390x, arm), and different add-ons (Optional repositories, Supplementary repositories, Virt V2V tools). This unifies all related repositories within the DML. Using Products ensures repositories that depend on each other are synchronized together. For Red Hat repositories, products are created automatically after enabling the repository.

In this chapter, our aim is to start creating our DML using Red Hat content. To do this, we synchronize our DML with Red Hat’s Products and their repositories.

5.3. Synchronizing Content

When you select which repositories form the DML, Satellite Server synchronizes its own repositories with the repositories on the Red Hat CDN. This ensures that Satellite Server retains an exact copy of Red Hat’s repositories as a part of its DML. Satellite Server fetches this repository information and stores it on Satellite Server’s file system. After an initial synchronization, you can create a synchronization plan that checks to ensure the repositories in the DML is up to date with the CDN’s repositories.

It is possible to perform an initial synchronization using ISO images. See Appendix C, Importing Content ISOs into a Connected Satellite for more information on using Content ISOs. For locations with bandwidth limitations, using an On Demand or Background download policy as described below might be quicker than downloading and importing Content ISOs.

5.3.1. Disabling the Global HTTP Proxy

If you want to synchronize local repositories without using the configured proxy, you can ignore the global proxy that is set when you run the satellite-installer tool.

Procedure

To ignore the global HTTP proxy, complete the following steps:

  1. In the Satellite web UI, navigate to Content > Products and select the product that you want to change.
  2. Click the Repositories tab, and select the repository that you want to change.
  3. In the Sync Settings area, navigate to Ignore Global HTTP Capsule. Click the edit icon, select the check box, and then click Save.

For CLI users

# hammer repository update --id Repository_ID --ignore-global-proxy yes

5.4. Configuring Download Policies

Red Hat Satellite provides multiple download policies for synchronizing RPM content. For example, you might aim to save time and only download the content metadata while deferring the actual content download for later.

Satellite Server offers the following policies:

  • Immediate — Satellite Server downloads all metadata and packages during synchronization.
  • On Demand — Satellite Server only downloads the metadata during synchronization. Satellite Server only fetches and stores packages on the file system when Capsules or directly connected clients request them. This setting has no effect if you set a corresponding repository on a Capsule to Immediate because Satellite Server is forced to download all the packages.
  • Background — Satellite Server creates a background task to download all packages after the initial synchronization.

The latter two policies act as a Lazy Synchronization feature because they save time synchronizing content. The lazy sync feature must only be used for yum repositories. The packages can be added to Content Views and promoted to life cycle environments as normal.

Capsule Server offers the following policies:

  • Immediate — Capsule Server downloads all metadata and packages during synchronization. Do not use this setting if the corresponding repository on Satellite Server is set to On Demand as Satellite Server is forced to download all the packages.
  • On Demand — Capsule Server only downloads the metadata during synchronization. The Capsule only fetches and stores packages on the file system when directly connected clients request them.

    Note

    When using an On Demand download policy, content is downloaded from Satellite Server if it is not available on the Capsule Server.

  • Background — Capsule Server creates a background task to download all packages after the initial synchronization.
  • Inherit — Capsule Server inherits the download policy for the repository from the corresponding repository on Satellite Server.

These policies are not available if a Capsule was installed or updated with --enable-foreman-proxy-plugin-pulp set to false.

5.4.1. Changing the Default Download Policy

You can set the default download policy globally so that Satellite applies it to new repositories you create in all organizations. Changing the default value does not change existing settings.

For Web UI Users

  1. In the Satellite web UI, navigate to Administer > Settings.
  2. Click the Content tab, locate default_download_policy.
  3. In the Value field, click the edit icon.
  4. Select the required download policy, and then click Save.

For CLI Users

To change the default download policy to one of immediate, on_demand, or background, enter the following command:

# hammer settings set \
--name default_download_policy \
--value immediate

5.4.2. Changing the Download Policy for a Repository

Use this procedure to change the download policy for a specific repository.

For Web UI Users

  1. In the web UI, navigate to Content > Products, and click the required product name.
  2. Click the required repository name, locate the Download Policy field, and click the edit icon.
  3. From the list, select the required download policy and then click Save.

For CLI Users

  1. List the repositories for an organization:

    # hammer repository list \
    --organization-label organization-label
  2. Change the download policy for a repository to one of immediate, on_demand, or background:

    # hammer repository update \
    --organization-label organization-label  \
    --product "Red Hat Enterprise Linux Server" \
    --name "Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.5"  \
    --download-policy immediate

5.5. Selecting Red Hat Repositories to Synchronize

To select the repositories to synchronize you must first identify the product that contains the repository, and then enable that repository based on the relevant release version and base architecture.

Important

If using a Disconnected Satellite, you need to import the Content ISOs for Red Hat Satellite and change the CDN URL on Satellite Server before synchronizing content. For more information, see Appendix B, Importing Content ISOs into a Disconnected Satellite.

For Web UI Users

  1. Navigate to Content > Red Hat Repositories.

    The Red Hat Repositories page displays the following two panes: the Available Repositories and the Enabled Repositories panes. The Available Repositories pane contains a list of all subscribed products that provide RPM content. The Enabled Repositories pane contains a list of enabled repositories.

    The relationship between products and specific repositories is connected through a cascading hierarchy that is displayed in the Available Repositories pane.

  2. To find the Red Hat Enterprise Linux 7 Server (RPMs) repositories, use one of the following methods:

    • In the Search field, enter the following repository name: rhel-7-server-rpms.
    • Toggle the Recommended Repositories button to the ON position to view a list of repositories that are used in a typical setup.
  3. In the Available Repositories pane, click on the Red Hat Enterprise Linux 7 Server (RPMs) to expand the repository set.
  4. For the x86_64 7Server entry, click the Enable icon to enable the repository. This enables the latest RPM files for Red Hat Enterprise Linux 7.
Note

The difference between associating Red Hat Enterprise Linux Operating System with either 7 Server repositories or 7.X repositories is that 7 Server repositories contain all the latest updates while Red Hat Enterprise Linux 7.X repositories stop getting updates after the next minor version release. Note that Kickstart repositories only have minor versions.

For CLI Users

The relationship between products and repositories is the same. To search for your product, enter the following command:

# hammer product list --organization "ACME"

List the repository set for the product:

# hammer repository-set list \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

This displays the repositories in the product’s repository set, including their name and ID number. Enable the repository using either the name or ID number. Also include the release version (7Server) and base architecture (x86_64). For example:

# hammer repository-set enable \
--name "Red Hat Enterprise Linux 7 Server (RPMs)" \
--releasever "7Server" \
--basearch "x86_64" \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

For this scenario, use either the Web UI or the CLI to enable the following repositories for ACME:

RepositoryTypeDescription

Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server

RPM

Repositories for the latest version of Red Hat Enterprise Linux 7. Use the 7 Server repository instead of the 7.2 repository so that you receive continuous package updates.

Red Hat Satellite Tools 6.4 for RHEL 7 Server RPMs x86_64

RPM

The Satellite Tools repository, which contains system management agents and tools for client systems. After provisioning a new system, Satellite installs tools such as katello-agent and Puppet to the client. Use the 7 Server repository instead of the 7.2 repository so that you receive continuous package updates.

Red Hat Enterprise Linux 7.2 Kickstart x86_64 7Server

Kickstart

The kickstart tree for Red Hat Enterprise Linux 7.2. Use this as installation media for provisioning new systems over PXE.

These repositories provide some initial content for our scenario’s DML. You can select more repositories based upon your needs.

Note

For this scenario, all repositories use x86_64 as the base architecture.

5.6. Synchronizing Red Hat Repositories

We have enabled specific repositories to form our initial DML. Now we synchronize the repositories with the Red Hat CDN’s repositories.

For Web UI Users

Navigate to Content > Products and select Red Hat Enterprise Linux Server. This displays all enabled repositories in our product. Select all repositories and click Sync Now. You can also view the progress of the synchronization in the Web UI. Navigate to Content > Sync Status and expanding the Product/Repository tree (or click Expand All).

For CLI Users

Synchronize the enabled repositories in the Red Hat Enterprise Linux Server product:

# hammer product synchronize \
--name "Red Hat Enterprise Linux Server" \
--organization "ACME"

You can also synchronize each repository individually. List all repositories in the product, then synchronize using the ID number for the corresponding repositories. For example:

# hammer repository list \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"
# hammer repository synchronize \
--name "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" \
--product "Red Hat Enterprise Linux Server" \
--organization "ACME"

The synchronization duration depends on the size of each repository and the speed of your network connection. The following table provides estimates of how long it would take to synchronize content, depending on the available Internet bandwidth:

 Single Package (10Mb)Minor Release (750Mb)Major Release (6Gb)

256 Kbps

5 Mins 27 Secs

6 Hrs 49 Mins 36 Secs

2 Days 7 Hrs 55 Mins

512 Kbps

2 Mins 43.84 Secs

3 Hrs 24 Mins 48 Secs

1 Day 3 Hrs 57 Mins

T1 (1.5 Mbps)

54.33 Secs

1 Hr 7 Mins 54.78 Secs

9 Hrs 16 Mins 20.57 Secs

10 Mbps

8.39 Secs

10 Mins 29.15 Secs

1 Hr 25 Mins 53.96 Secs

100 Mbps

0.84 Secs

1 Min 2.91 Secs

8 Mins 35.4 Secs

1000 Mbps

0.08 Secs

6.29 Secs

51.54 Secs

A manual synchronization is often required for the initial content import into the DML. However, it is recommended to create a synchronization plan to ensure that our DML updates on a regular basis.

Note

You can change the download policy for a Red Hat repository. Select a repository in the Red Hat Enterprise Linux Server product and navigate to the Download Policy field. If using the CLI, enter the hammer repository update command with the --download-policy option.

5.6.1. Recovering a Repository

In the case of repository corruption, you can recover it by using an advanced synchronization, which has three options:

  • Optimized Sync - it synchronizes the repository bypassing RPMs that have no detected differences from the upstream RPMs.
  • Complete Sync - it synchronizes all RPMs regardless of detected changes. Use this option if specific RPMs could not be downloaded to the local repository even though they exist in the upstream repository.
  • Validate Content Sync - it synchronizes all RPMs and then verifies the checksum of all RPMs locally. If the checksum of an RPM differs from the upstream, it re-downloads the RPM. This option is relevant for yum repositories only. Use this option if you have one of the following errors:

    • Specific RPMs cause a 404 error while synchronizing with yum.
    • Package does not match intended download error, which means that specific RPMs are corrupted.

To synchronize a specific repository with an advanced option:

For Web UI users

  1. Navigate to Content > Products.
  2. Select the product containing the corrupted repository.
  3. Click on the name of a repository you want to synchronize.
  4. Extend the Select Action menu and select Advanced Sync.
  5. Select the option and click Sync.

For CLI users

  1. Obtain a list of repository IDs:

    # hammer repository list --organization "Default Organization"
  2. Synchronize a corrupted repository using the necessary option:

    • For the optimized synchronization:

      # hammer repository synchronize --incremental true --id 1
    • For the complete synchronization:

      # hammer repository synchronize --skip-metadata-check true --id 1
    • For the validate content synchronization:

      # hammer repository synchronize --validate-contents true --id 1

5.6.2. Limiting Synchronization Speed

You can control the speed of synchronization to avoid exhaustion of available bandwidth and prevent other performance issues. This is done by configuring PULP_CONCURRENCY and max_speed parameters. Note that these settings are overwritten on an upgrade. It is recommended to backup changed files prior to an upgrade to be able to restore the configuration.

  1. To control the number of synchronization jobs that run in parallel, configure the PULP_CONCURRENCY parameter in the /etc/default/pulp_workers file. For example, to set the number of jobs that run in parallel to 1, change PULP_CONCURRENCY to 1:

    PULP_CONCURRENCY=1

    By default, on a system with less than 8 CPUs, PULP_CONCURRENCY is set to the number of CPUs. On a system with more than 8 CPUs, it is set to 8.

  2. To set the maximum network speed for synchronizing in bytes per second, configure the max_speed parameter. This parameter must be configured separately for each importer in the /etc/pulp/server/plugins.conf.d/ directory.

    1. For example, to set the maximum speed for synchronizing RPM content to 10 bytes per second, set the "max_speed" parameter in the /etc/pulp/server/plugins.conf.d/yum_importer.json file to 10:

      # cat /etc/pulp/server/plugins.conf.d/yum_importer.json
      {
          "proxy_host": null,
          "proxy_port": null,
          "proxy_username": null,
          "proxy_password": null,
          "max_speed": 10
      }
    2. Verify the syntax of the file after editing:

      # json_verify < /etc/pulp/server/plugins.conf.d/yum_importer.json
      JSON is valid
  3. Restart Satellite services to apply the changes:

    # foreman-maintain service restart

5.7. Creating a Synchronization Plan

A synchronization plan checks and updates the content in your DML at a regularly scheduled date and time. Red Hat Satellite 6 provides users with the ability to create a synchronization plan and assign products to it.

For Web UI Users

Navigate to Content > Sync Plans and click New Sync Plan. The UI provides a set of fields where you can input details about your synchronization plan:

  • Name - A plain text name for the plan. Enter Example Plan.
  • Description - A plain text description of the plan. Enter Example Plan for ACME’s repositories.
  • Interval - Defines when to run the synchronization. Select daily.
  • Start Date and Start Time - Defines when to run the synchronization. We already completed a synchronization today, so set the synchronization for tomorrow at 1:00 (1AM).

Click Save to create a plan. The plan details page displays along with two tabs for Details and Products.

Now add your products. Click the Products tab, then click Add. Select the Red Hat Enterprise Linux Server product and click Add Selected.

For CLI Users

To create the synchronization plan, enter the following command:

# hammer sync-plan create \
--name "Red Hat Products 2" \
--description "Example Plan for ACME's Red Hat Products" \
--interval daily \
--sync-date "2016-02-01 01:00:00" \
--enabled true \
--organization "ACME"

Then assign the Red Hat Enterprise Linux Server product to it:

# hammer product set-sync-plan \
--name "Red Hat Enterprise Linux Server" \
--sync-plan "Red Hat Products" \
--organization "ACME"

Now Satellite Server checks its DML content against the Red Hat CDN on a daily basis and keeps its Red Hat repositories up to date.