Chapter 1. About Red Hat Update Infrastructure
Red Hat Update Infrastructure (RHUI) is a highly scalable, highly redundant framework that focuses on repository and content management and enables cloud providers to deliver content and updates to Red Hat Enterprise Linux (RHEL) instances. Based on the upstream Pulp project, RHUI allows cloud providers to locally mirror Red Hat-hosted repository content, create custom repositories with their own content, and make those repositories available to a large group of end users via a load-balanced content delivery system.
The Red Hat Update Infrastructure 3.0.Beta System Administrator Guide is intended to help system administrators prepare their infrastructure for participation in the Red Hat Certified Cloud and Service Provider program.
This guide documents the steps necessary to install and configure the Red Hat Update Appliance (RHUA), content delivery servers (CDSes), repositories, shared storage, and load balancing. Experienced RHEL system administrators are the target audience. Cloud providers whose staff do not have extensive Red Hat Enterprise Linux skills should consider engaging Red Hat Consulting to provide a Red Hat Certified Cloud Provider Architecture Service.
The Red Hat Update Infrastructure 3.0.Beta System Administrator Guide also provides guidance to system administrators when configuring, managing, and updating the Red Hat Update Infrastructure. This guide discusses the various components of Red Hat Update Infrastructure, content provider types, the command-line interface (CLI) used to manage the components, and utility commands. It addresses certificate management and content management.
See Appendix A, Red Hat Update Infrastructure Management Tool Menus and Commands for a list of all of the available menus and commands in the Red Hat Update Infrastructure Management Tool.
See Appendix B, Common Issues in Red Hat Update Infrastructure for some of the common known issues.
See RestAPI for a list of the Pulp APIs used in Red Hat Update Infrastructure.
See the following resources for more information on the various Red Hat Update Infrastructure components.
1.1. What’s New
Red Hat Update Infrastructure 3.0.Beta offers:
- easy installation using Puppet.
- code rebased to Pulp 2.8 to align with Red Hat Satellite 6.
- reworked CDS architecture for automated installations.
- shared storage (Red Hat Gluster Storage) to speed up content availability at the CDS and eliminate the need for synchronization.
- default use of Gluster Storage as the repository back end to reduce synchronization time.
- high-availability deployment to reduce the error of one CDS not being synchronized with another CDS.
- a load balancer/HA Proxy instance that is client-facing. (This functionality was integrated previously into the CDS logic.)
- certificates managed by the
rhui-installerandrhui-managercommands. - updates to yum.repos.d/*, certificates, and keys to use a new unified URL.
- removal of client-side load balancing functionality from rhui-lb.py.
- support for Docker and OSTree (atomic) content.
1.2. Organization
1.2.1. Installation Options
There are several scenarios for setting up your cloud environment. This guide documents Option 1.
Option 1: Full Installation
- A Red Hat Update Appliance (RHUA)
- Two or more CDS instances with Gluster Storage
- Two or more HAProxy servers
Option 2: "Bring your own Storage" and HAProxy
- A RHUA
- Two or more CDS instances with an existing storage solution
- Two or more HAProxy servers
Option 3: “Bring your own Load Balancer" and Gluster Storage
- A RHUA
- Two or more CDS instances with Gluster Storage
- Existing load balancer
Option 4: "Bring your own Storage" and "Bring your own Load Balancer"
- A RHUA
- Two or more CDS instances with an existing storage solution
- Existing load balancer
The following figure depicts a high-level view of how the various RHUI components interact with each other.
Figure 1.1. Red Hat Update Infrastructure Overview
The following subsections describe the various Red Hat Update Infrastructure components.
1.2.2. Red Hat Update Appliance
There is one RHUA per RHUI installation, though in many cloud environments there will be one RHUI installation per region or datacenter. For example, Amazon’s EC2 cloud comprises 11 regions. There is a separate RHUI installation in each region, each with its own RHUA instance.
The RHUA:
- downloads new packages from the Red Hat CDN. The RHUA is the only RHUI component that connects to Red Hat, and its synchronization schedule is configurable.
- copies new packages to each CDS instance.
- checks the RHUI installation’s health and writes the results to a file located on the RHUA. Monitoring solutions use this file to determine the RHUI installation’s health.
- provides a human-readable view of the RHUI installation’s health through a CLI tool.
Despite its name, the RHUA is not actually shipped as an appliance; it is an RPM installed on an instance in the cloud.
The RHUI uses two main configuration files: /etc/rhua/rhua.yml and /etc/rhua/config.yml.
The /etc/rhua/rhua.yml configuration file contains general options used by the RHUA, such as the location of log files, default file locations for certificates, and default configuration parameters for the Red Hat CDN synchronization. This file normally does not require editing.
The Red Hat Update Infrastructure Management Tool generates the /etc/rhua/config.yml configuration file based on user-inputted values. It contains all the information that drives the running of a RHUA in a particular region. An example configuration includes the destination on the RHUA to download packages and a list of CDS instances (host names) in the RHUI installation.
The RHUA employs several services to synchronize, organize, and distribute content for easy delivery:
- Pulp: The service that oversees management of the supporting services, providing a user interface for users to interact with
- MongoDB: A NoSQL database used to keep track of currently synchronized repositories, packages, and other crucial metadata. MongoDB stores values in BSON (Binary JSON) objects.
- Qpid: An Apache-based messaging broker system that allows the RHUA to interact securely with the CDSes to inform them of desired actions against the RHUA (synchronize, remove, adjust repositories, and so on). This allows for full control over the RHUI appliance from just the RHUA system.
Some considerations to keep in mind include:
- Qpid certificates expire after 1 year by default.
- MongoDB’s files appear to take up a large amount of room on the file system and are sometimes larger than the database content itself. This is normal behavior based on Mongo’s allocation method. For more information, see RHUIs mongodb files are larger than the actual database contents.
- If MongoDB fails to start, clearing database locks and performing a repair is often effective as outlined in Red Hat Update Infrastructure fails to start due to a MongDB startup error.
1.2.3. Content Delivery Server
The CDS instances provide the yum repositories that clients connect to for the updated packages. There can be as few as one CDS. Because the RHUI provides a load balancer with failover capabilities, we recommended that you use multiple CDS instances.
The CDSes host content to end-user RHEL systems. While there is no required number of systems, the CDS works in a round-robin style load-balanced fashion (A, B, C, A, B, C) to deliver content to end-user systems. The CDS uses HTTP to host content to end-user systems via httpd-based yum repositories.
During configuration, you specify the CDS’s directory where packages are synchronized. Similar to the RHUA, the only requirement is that you mount the directory on the CDS. It is up to the cloud provider to determine the best course of action when allocating the necessary devices. The Red Hat Update Infrastructure Management Tool configuration RPM takes care of linking the package directory with the Apache configuration to serve it.
If NFS is used, rhui-installer can configure an NFS share on the RHUA to store RPM packages and Yum repository data as well as a directory on the CDSes where the NFS share will be mounted. The following rhui-manager options control these settings:
- --remote-fs-mountpoint: the file system location where the remote file system share should be mounted (default: /var/lib/rhui/remote_share)
- --remote-fs-server: remote mountpoint for a shared file system to use; for example,
nfs.example.com:/path/to/share(default:nfs.example.com:/export)
If these default values are used, the /export directory on the RHUA and the /var/lib/rhui/remote_share directory on each CDS are identical. For example, the published subdirectory has the following structure if Yum, Docker, and OSTree repositories are already synchronized.
[root@rhua ~]# ls /export/published/ docker ostree yum [root@cds01 ~]# ls /var/lib/rhui/remote_share/published/ docker ostree yum
The expected usage is that each CDS will keep its own copy of the packages. It is possible the cloud provider will use some form of shared storage (such as Gluster Storage) that the RHUA writes packages to and each CDS reads from.
The storage solution needs to provide an NFS endpoint for mounting on the RHUA and CDSes. If local storage is implemented, shared storage is needed for the cluster to work. If you want to provide local storage to the RHUA, configure the RHUA to also act as the NFS server with a rhua.example.com:/path/to/nfs/share endpoint configured.
The only nonstandard logic that takes place on each CDS is the entitlement certificate checking. This checking is performed to ensure that the client making requests on the yum repositories is authorized by the cloud provider to access those packages. The check:
- ensures that the entitlement certificate was signed by the cloud provider’s Certificate Authority (CA) Certificate. The CA certificate is installed on the CDS as part of its configuration to facilitate this verification.
- ensures the requested URI matches an entitlement found in the client’s entitlement certificate.
If the CA verification fails, the client will see an SSL error. More information can be found in the CDS’s Apache logs under /var/log/httpd/.
[root@cds01 ~]# ls -1 /var/log/httpd/ access_log cds.example.com_access_ssl.log cds.example.com_error_ssl.log crane_access_ssl.log crane_error_ssl.log default_error.log error_log
The Apache configuration is handled through the /etc/httpd/conf.d/cds_ssl.conf file during the CDS installation.
If multiple clients experience issues updating against a repository, this may indicate a problem with the RHUI itself. See Yum generates 'Errno 14 HTTP Error 401: Authorization Required' while accessing RHUI CDS for more details.
1.2.4. Load Balancer
Clients are not configured to go directly to a CDS; their repository files are configured to point to HAProxy, the RHUI load balancer. HAProxy is a TCP/HTTP reverse proxy particularly suited for high-availability environments. HAProxy can:
- route HTTP requests depending on statically assigned cookies.
- spread the load among several servers while assuring server persistence through the use of HTTP cookies.
- switch to backup servers in the event a main server fails.
- accept connections to special ports dedicated to service monitoring.
- stop accepting connections without breaking existing ones.
- add, modify, and delete HTTP headers in both directions.
- block requests matching particular patterns.
- persist client connections to the correct application server depending on application cookies.
- report detailed status as HTML pages to authenticated users from a URI intercepted from the application.
With RHEL 7, the load balancer technology is included in the base operating system. The load balancer can either be installed on another instance (external load balancer) in the cloud or run on the RHUA (also known as the embedded load balancer). The configuration for both types of deployment is the same.
If you are using an existing load balancer, ensure ports 5000 and 443 are configured in the load balancer for the cds-lb-hostname forwarded to the pool and that all CDSes in the cluster are in the load balancer’s pool. You do not need to follow the steps in Chapter 9, Add and Configure an HAProxy Server.
Some considerations to keep in mind include:
- When clients fail to connect successfully, it is important to review the httpd logs on the CDS under /var/log/httpd/ to ensure that any requests reached the CDS. If not, issues such as DNS or general network connectivity may be at fault.
1.2.5. Repositories, Containers, and Content
A repository is a storage location for software packages (RPMs). RHEL uses yum commands to search a repository, download, install, and configure the RPMs. The RPMs contain all the dependencies needed to run an application. RPMs also download updates for software in your repositories.
RHEL 7 implements Linux containers using core technologies such as control groups (cgroups) for resource management, namespaces for process isolation, and SELinux for security, enabling secure multiple tenancy and reducing the potential for security exploits. Linux containers enable rapid application deployment, simpler testing, maintenance, and troubleshooting while improving security. Using RHEL 7 with Docker allows you to increase staff efficiency, deploy third-party applications faster, enable a more agile development environment, and manage resources more tightly.
There are two general scenarios for using Linux containers in RHEL 7. You can work with host containers as a tool for application sandboxing, or you can use the extended features of image-based containers. When you launch a container from an image, a writable layer is added on top of this image. Every time you commit a container (using the docker commit command), a new image layer is added to store your changes.
Content, as it relates to RHUI, is the software (RPMs) that you download from the Red Hat CDN for use on the RHUA and the CDS instances. The RPMs provide the files necessary to run specific applications and tools. Clients are granted access by a set of SSL content certificates and keys provided by an rpm package, which also provides a set of generated yum repository files.
See What Channels Can Be Delivered at Red Hat’s Certified Certified Cloud & Service Provider (CCSP) Partners? for a list of all of the channels available from CCSP partners.
1.2.6. Instances and Systems
In a cloud computing environment, an instance is normally a virtual machine or server. For example, within RHUI, the RHUA and the CDSes are considered instances. When you combine all the nodes (instances, storage, and load balancing), you create a system.
1.3. Content Provider Types
There are three types of cloud computing environments: public cloud, private cloud, and hybrid cloud. This guide focuses on public and private clouds.
A public cloud provides its framework and services remotely via the Internet. Its strength is how efficiently it can offer shared resources. A public cloud is a good option for collaborating and for creating a testing and development environment. Another benefit is the ability to scale up to handle peaks. One drawback is that a public cloud is not as secure as a private cloud. This concern should be taken seriously if you have critical applications you want to deploy to a cloud environment.
A private cloud provides more security and control than a public cloud because it is hosted on a private network, but those benefits come at a cost. You have to buy and maintain all the software and infrastructure. That might not be an issue if your company is large enough to run an efficient datacenter. You may also decide that you cannot afford to have your data breached, or you may be in an industry that has stringent security and privacy regulations. If any of these options are true, a private cloud may make sense.
1.4. Red Hat Update Infrastructure Command-Line Interface
The bulk of RHUI is in its installation. Once installed, it runs on its own, periodically getting updated packages from the Red Hat CDN and automatically making those packages available to clients.
A command-line interface called Red Hat Update Infrastructure Management Tool (run with the rhui-manager command) facilitates the installation. This tool provides interactive prompts for the necessary configuration elements for each RHUI component: RHUA, CDS, and load balancer. This tool also provides a means for taking the content certificate provided by Red Hat for use when connecting to the Red Hat CDN and generating internal, cloud-specific certificates that clients will use to connect to RHUI. The Red Hat Update Infrastructure Management Tool allows the cloud provider to generate a client configuration bundle that will be installed on client RHEL instances. This bundle allows them to get updates from the RHUI installation.
Red Hat Update Infrastructure Management Tool is designed to be run using the interactive shell; some functions can also be run from a standard shell prompt. The Red Hat Update Infrastructure Management Tool uses five main commands. For each command’s subcommand, a list of options is provided if the subcommand expects one or more options other than -h and --help.
View all options and commands by running the following command.
[root@rhua ~]# rhui-manager --help Usage: rhui-manager [options] OPTIONS -h/--help show this help message and exit --debug enables debug logging --config absolute path to the configuration file; defaults to /etc/rhui/rhui-tools.conf --server location of the RHUA server (overrides the config file) --username if specified, previously saved authentication credentials are ignored and this username is used to login --password used in conjunction with --username COMMANDS cert : Red Hat content certificate management packages : package manipulation on repositories repo : repository listing and manipulation status : RHUI status and health information client : Red Hat client management
The following subsections provide more details about the subcommands and options for their associated command.
1.4.1. cert
info: display information about the current content certificate
- (This subcommand has no options.)
upload: uploads a new content certificate
- --cert - full path to the new content certificate (required)
- --key - full path to the new content certificate’s key
Example
[root@rhua ~]# rhui-manager cert upload --cert /tmp/extra_rhui_files/rhcert.pem
Red Hat Entitlements
Valid
Beta RHEL RHUI Everything 7 Debug
Expiration: 05-17-2026 Certificate: rhcert.pem
Beta RHEL RHUI Everything 7 OS
Expiration: 05-17-2026 Certificate: rhcert.pem1.4.2. packages
list: lists all packages in a repository
- --repo_id - id of the repository to list packages for (required)
upload: uploads a package or directory of packages to a custom repository
- --repo_id - id of the custom repository where the packages will be uploaded (required)
- --packages - path to an .rpm file or directory of RPMs that will be uploaded (required)
Example
[root@rhua ~]# rhui-manager packages upload --repo_id my_custom_repo --packages /tmp/webalizer-2.21-6.2.x86_64.rpm Uploading /tmp/webalizer-2.21-6.2.x86_64.rpm... /tmp/webalizer-2.21-6.2.x86_64.rpm successfully uploaded
1.4.3. repo
sync: sync a repository
- --repo_id - identifies the repository to display (required)
add_by_repo: add Red Hat repositories to the RHUA via repo ID
- --repo_ids - repo IDs to add, comma-separated (required)
unused: list of products available but not synced to the RHUA
- -by_repo_id - List unused products by repo id
list: lists all repositories in the RHUI
- (this subcommand has no options)
add: add a Red Hat repository to the RHUA
- --product_name - product to add the RHUA (required)
info: displays information on an individual repo
- --repo_id - identifies the repository to display (required)
Example
[root@rhua ~]# rhui-manager repo list ID :: Repository Name Red Hat Repositories -------------------- rhel-rhui-server-7-rh-gluster-samba-3.1-os-7Server-x86_64 :: RHEL RHUI Server 7 Rh-gluster-samba 3.1 OS (7Server-x86_64) Custom Repositories -------------------- my_custom_repo :: My Custom Repo
1.4.4. status
(This command has no subcommands.)
- --code - if specified, only a numeric code for the result will be displayed
Example
[root@rhua ~]# rhui-manager status RHEL RHUI Server 7 Rh-gluster-samba 3.1 OS (7Server-x86_64) ........... Never Entitlement CA certificate expiration date = 2038-01-18T10:30:32Z .... [ OK ] 0
1.4.5. client
rpm: create a client config rpm
- --private_key - entitlement private key (required)
- --entitlement_cert - entitlement certificate (required)
- --rpm_version - version number of the client config rpm (required)
- --rpm_name - name of the client config rpm (required)
- --dir - directory where the rpm will be created (required)
- --unprotected_repos - comma-separated list of unprotected repos to include
labels: list the labels required for client certificate creation
- (this subcommand has no options)
cert: create a content certificate for a rhui client
- --repo_label - identifies the repositories to add. Comma delimited string of repo labels (required)
- --name - identifies the certificate name (required)
- --days - number of days cert will be valid (required)
- --dir - directory where the certificate will be stored (required)
Example
[root@rhua ~]# rhui-manager client cert --repo_label rhel-rhui-server-7-rh-gluster-samba-3.1-os --name gluster-samba-31 --days 365 --dir /tmp .........................................................................................+++ ..............+++ Entitlement certificate created at /tmp/gluster-samba-31.crt
1.5. Additional Utility Commands
See Appendix A, Red Hat Update Infrastructure Management Tool Menus and Commands for a list of the other functions you can perform using Red Hat Update Infrastructure Management Tool.
1.6. Component Communications
With so many instances being used, it is helpful to understand the communication protocols between each. All protocols are existing standards, so there is little need to explain how they work. Standard ports are used in all cases (443 for HTTPS, 22 for rsync over SSH).
Table 1.1. Red Hat Update Infrastructure Communication Protocols
| Source | Destination | Protocol | Purpose |
|---|---|---|---|
Red Hat Update Appliance | Red Hat Content Delivery Network | HTTPS | Downloads packages from Red Hat |
Load Balancer | Content Delivery Server | HTTPS | Forwards the client’s yum request |
Client | Load Balancer | HTTPS | Used by yum on the client to download packages from a content delivery server |
Comments