Create a CDN

Latest response

Hi everyone,

For of a project, I would like to set up a repository server (CDN?) exposed on the web to download content and make it available to an offline Satellite server.
Is this possible?
How to do it?
How to tell the Satellite server that its CDN is this server?

Best regards,

Diego

Responses

Hi Diego,

Why do you or your client want to setup a local CDN?

Would it be easier to setup a squid or other kind of proxy and connect the Satellite to Red Hat's CDN?

Just my thoughts.

Regards,

Jan Gerrit Kootstra

Hi Jan,

For security reasons, this choice was made but I don't know why.

The CDN, Satellite server and a capsule are in different VLANs.

The local CDN will be behind a proxy. Is this configuration possible?

Best regards,

Diego

Hi Diego,

A hint I got from a fellow Red Hat Accelerator @khalid: RH Satellite 6.9 installation guide for disconnected networks Here they setup a local CDN using the content iso's.

Regards,

Jan Gerrit Kootstra

I guess below doc might help

3.1. Configuring Satellite to Synchronize Content with a Local CDN Server https://access.redhat.com/documentation/en-us/red_hat_satellite/6.9/html/installing_satellite_server_from_a_disconnected_network/performing-additional-configuration#configuring-satellite-to-synchronize-content-with-a-local-cdn-server_satellite

Thanks for your answers.

I am not sure I understand how this local CDN works.

I have to download the content by iso images?

I can't download the content like satelitte server by Red Hat CDN?

This local CDN is not offline, it can go out on the internet

Best regards,

Diego

Diego Zucchiatti

I shall post instructions on how to do this. I've been doing this every month for years and have found a method that works well (if someone knows a better method, I'm all ears)

More info shortly,

Regards,
RJ

Can't wait to read these proven instructions for years.

Best regards,

Diego

Diego Zucchiatti

Overview

Inter Satellite Sync is a feature that's been available since satellite 6.2, this article is by Rich Jerrido and works with Satellite 6.9

Ragged functional instructions, this will be refined in a new discussion post later
  • This article assumes a current Satellite, version 6.9.x, as I type this, it is 6.9.5 They update this often! - the export command in the details below will fail on an older satellite circa 6.3 for example, but works on 6.9.x
  • From a public-facing satellite server, have a place where you can safely export a footprint of at minimum 2TB (our own example is 900G, however, it's better to be safe, because other companies who do this may have a larger footprint of repositories than we do). We execute a command to hardlink duplicated exported rpms from 900G to 190G in our specific case use.
  • Use the traditional methods of ingesting rpm repositories in the Red Hat Content Management guide prior to export.
  • Establish a synchronization plan. Ours grabs new updates every hour of every day, so that ours is always current.
  • Please see the video at the link above by Rich Jerrido, take notes, watch it more than once
  • See the comments in the link above for the export command that works for Satellite 6.9.x
  • In the public-facing Satellite, go to the web interface, "Administer >> Settings" "Content tab" under "Default Custom Repository download policy" select "immediate". Do the same for "Default Red Hat Repository download policy"
  • In the public-facing Satellite, in the same web interface as the step above, go to "Pulp export destination file path" and put in the path for where you will drop your content view export this is important to make sure to have sufficient space for at minimum one (1) entire content view, go overkill on the storage please or your export will fail (ask me how I know)`" Example, maybe you point it to "/var/www/html/pub/export" and if so, your satellite Content View exports will land there. **If your organization name is "prod" then it will export as "prod-Default_Organization_View-v1.0/prod/Library/content/"
  • We have an obscene-overkill amount of storage for that location, I recommend total overkill on storage.
  • If you have not already done so, watch the video by Rich Jerrido in the link I put above
  • Make sure you have established your manifest at the Red Hat Customer Portal, Subscriptions, Subscription Allocations. Add all the relevant entitlements for everything you intend to connect in your disconnected environment(s).
  • After successful establishing your manifest, export it, it is a zip file. We named ours a date and time. Never delete it, just modify it as needed and export again when your needs change over time.
  • Ingest the manifest you exported above into your public-facing satellite so your satellite can grab the necessary repositories.
  • Go to the Satellite web UI and navigate to "Content >> Red Hat Repositories" and follow the guidance in the current official Red Hat documentation to appropriately select the repositories you intend to present to your disconnected satellites. Note, we never have a use for "Fastrack repositories" and it caused us issues the one time we had it 5 or so years ago, we do not use it ever
  • Establish a synchronization plan on your public-facing Satellite that points to Red Hat. Perform a synchronization (attempt this) and watch in the Satellite Web Interface "Content > Sync Status" and ensure there are no failed synchronizations. If needed, we've discovered it often helps to refresh our manifest in the satellite web UI if there are rabid failed repos that fail to sync.
  • After you get your first successful sync, then, (and only then) do your content view export with the updated comments in the guide made by Rich Jerrido in the link above. I put in the specific commands for Satellite 6.9.x in the comments at that link. **THE EXPORT TAKES ABOUT 24 HOURS, use a tmux session or screen session if you remote into your satellite, because if your session dies, so does your export!!!
  • Sometimes if an export fails, there may be failed tasks in the satellite getting in the way, you can examine these in Web UI at "Monitor > Tasks" and I will some times clear out successful tasks (tens of thousands of at times) **Recommend a ticket for those who are un-initiated with using hammer commands to delete previous tasks)
  • We have a lot of RAM and CPUs on our virtual satellites due to pulp workers requiring these repository actions, otherwise it can take forever, or flat fail if it is underpowered. We have 8CPUs, and 24G of RAM on our VMware Satellite that faces the public, and at minimum, that much on our disconnected satellites.
  • When you get a successful export, look at the location you established in your Satellite web UI for the export.
  • After your successful export from the above, we use the hardlink command mentioned above to bring our footprint from 900G to 190G. Red Hat has similar RPMs riddled throughout the repositories and when you hardlink them with the "hardlink" command, it looks for duplicates, and hardlinks them. I have a ticket in for this for Red Hat to make their export use hardlinks, not sure if it will go through.
  • After the successful hardlink command above, we use "rsync -H -au --progress $source $target" and of course, we define the variables in advance. The placement of the trailing "/" in $source $target is important. Example, your source is /var/www/html/pub/export/prod-Default_Organization_View-v1.0 and your rsync target is a large capacity drive mounted as /mnt/4tbxfs/ then your rsync could be rsync -H -au --progress /var/www/html/pub/export/prod-Default_Organization_View-v1.0/prod/Library/content/ /mnt/4tbxfs/ and then you'll end up with a directory named "content" under "/mnt/4tbxfs/content" that you will carry over to your disconnected satellite servers, and ingest appropriately using the Red Hat Content Management guide. I can provide guidance later if needed.

This will probably generate more questions - so let me know with a reply here and I'll be happy to assist.

  • Again, make sure in advance, you have sufficient space to ingest the repositories on all your satellite servers. Please use the "-H" with rsyncs to avert over-usage of your storage.

Regards,
RJ

Diego Zucchiatti,

Please see the previous post above... (scroll up)

I have SELinux on with my satellites.

Regards,
RJ

Diego,

I updated the post above with the official Red Hat instructions for export/import

Regards,
RJ

RJ,

Thank you very much for the quality of your answer. I have read the whole procedure and the various links useful for it.

For the moment I don't really have any questions as I haven't tested it but I will surely have some later.

On the other hand I thought I could find a CDN solution without having a second satellite server.

Best regards,

Diego

Diego,

Arguably, you could use hearty rounds of reposync for various repositories. The only potential danger would be the timing of repositories and the reposync. If you grabbed a repo today, another repo tomorrow, there's a chance you could end up with a dependency missing in an updated RPM that was made available the next day... maybe. and I might be completely wrong here. If you decide to use rousing rounds of reposync, I'd put in a ticket with Red Hat and mention this discussion. We've been using the Satellite CDN method for many years. It works great for us. I have no data for case uses for those who might try an alternative such as I've mention in this specific post/reply.

Kind Regards,
RJ