Menu Close

Chapter 12. Adding sources for public cloud metering

Most of the data collection tools that gather, process, and analyze data for the subscriptions service are either established subscription management tools or additional components that work with or enhance the functions of tools. Examples include Red Hat Satellite and the Satellite inventory upload plugin, or OpenShift Cluster Manager and the monitoring stack tools.

In addition to these tools, there are Cloud Services platform tools for the Hybrid Cloud Console that perform data collection. One type of these tools is a source. The sources application is how the services and applications in the Hybrid Cloud Console connect with public cloud providers and with each other to collect and exchange data. You can think of a source as another data collection tool, but remember that it is set up with a different process than the other data collection tools. A source is created from within the Hybrid Cloud Console.

For the subscriptions service, you can add sources to enable high-precision data collection for your RHEL based Amazon Web Services instances in the public cloud. Although the subscriptions service currently has the ability to identify RHEL based instances for multiple cloud providers, it is not able to identify and track the activities of individual instances as they start and stop, sometimes multiple times per day. The public cloud metering tool adds that capability for AWS instances, resulting in more accurate monitoring of usage for those instances by the subscriptions service.

To use the public cloud metering tool for public cloud data collection, you must add sources to represent each of your AWS accounts. You add sources by using the sources application in the Hybrid Cloud Console settings.

Note

For organizations where the subscriptions service has not already been activated for the Red Hat organization account, adding an AWS source to enable the public cloud metering tool also activates the subscriptions service for the Red Hat account.

Select from these steps

Learn more

12.1. Adding an AWS source with the account authorization configuration mode

If you are using public cloud metering as the data collection tool for Red Hat Enterprise Linux usage in Amazon Web Services (AWS) accounts, add each account as a cloud source.

Note

The account authorization configuration mode is an automated mode for creating sources. When you select this mode, you provide your AWS account root user credentials in the form of the access key ID and secret access key. These credentials are used briefly to complete the automated steps and are then discarded. If you do not want to use the account authorization configuration mode, you can instead use the manual configuration mode for source creation.

When you add an AWS account as a source, the automated steps for the account authorization configuration mode create a specialized AWS Identity and Access Management (IAM) policy and role and add a connection between your AWS account and public cloud metering. The policy and role enable public cloud metering to perform the tasks that are required to identify and to meter public cloud usage of RHEL in that account.

Prerequisites

To create a source, you must meet the following prerequisites:

  • You must have the ability to create AWS resources in the us-east-1 region. If your AWS policies do not allow the creation of AWS resources in the us-east-1 region, you might be able to complete the steps to create the source, but the source might not complete the enablement process.
  • You must have the Sources administrator role in the role-based access control (RBAC) system for the Hybrid Cloud Console.

    Note

    Beginning in September 2021, the creation of a source requires the Sources administrator RBAC role. The Red Hat Customer Portal organization administrator (org admin) account role for your organization no longer has sufficient permissions to create sources.

Procedure

  1. In a browser window, go to cloud.redhat.com.
  2. If prompted, enter your Red Hat Customer Portal login credentials. The Hybrid Cloud Console opens.
  3. Click Settings (the gear icon) to show the settings options.
  4. In the navigation menu, click Sources.
  5. Click the Cloud sources tab if this page is not displayed by default. Click Add Source. The Add a cloud source wizard opens.

    Note

    You can also edit an existing source to add an association to the subscriptions service.

  6. Select the Amazon Web Services icon as the source type. Click Next.
  7. Enter a name for the source. This name is not required to be the same as the AWS account name. However, use a name that is easy to distinguish if you have multiple AWS accounts and must create multiple sources for them. Click Next.
  8. Select Account Authorization as the configuration mode. The window refreshes to display the fields for the AWS account root user credentials.
  9. Enter the access key ID and secret access key for the AWS account root user. Click Next.
  10. Select RHEL management as the application. This selection provides the high-precision data capabilities of public cloud metering for the subscriptions service. Select other options as appropriate. Click Next.
  11. Review the details for this source. Click Add to complete the source creation.

12.1.1. Verification steps

During the final step of source creation in the Add a cloud source wizard, the connection to the AWS account is verified and an AWS CloudTrail trail is created for the account. The CloudTrail trail is used to monitor the start and stop events for instances, the raw data that is used to calculate usage data for display in the subscriptions service. If the verification and trail creation is successful, the source creation is successful. This process normally takes only a few seconds.

To find the RHEL images and the associated instances that it is going to track, public cloud metering must then perform an inspection of the AWS account. The length of this inspection process can vary according to many factors, including AWS performance, the number of images in the account, the size and type of each image, the number of instances for an image, and others. As a general rule, the inspection process for an image and its instances can take approximately one hour.

After the inspection process is complete, public cloud metering can begin reporting usage data to subscriptions. In most cases, reporting begins in subscriptions within 24 hours. However, because of the timing of source creation, the amount of time required for the inspection process, and the reporting intervals, or heartbeats, for Cloud Services platform tools, in rare cases you might have to wait up to 48 hours for this data to begin appearing in the subscriptions service.

12.2. Adding an AWS source with the manual configuration mode

If you are using public cloud metering as the data collection tool for Red Hat Enterprise Linux usage in Amazon Web Services (AWS) accounts, add each account as a cloud source.

Note

The manual configuration mode enables you to create a source without providing your AWS account root user credentials. When you select this mode, you manually create a specialized AWS Identity and Access Management (IAM) policy and role and add a connection between your AWS account and public cloud metering. The policy and role enable public cloud metering to perform the tasks that are required to identify and to meter public cloud usage of RHEL in that account.

Prerequisites

To create a source, you must meet the following prerequisites:

  • You must have the ability to create AWS resources in the us-east-1 region. If your AWS policies do not allow the creation of AWS resources in the us-east-1 region, you might be able to complete the steps to create the source, but the source might not complete the enablement process.
  • You must have the Sources administrator role in the role-based access control (RBAC) system for the Hybrid Cloud Console.

    Note

    Beginning in September 2021, the creation of a source requires the Sources administrator RBAC role. The Red Hat Customer Portal organization administrator (org admin) account role for your organization no longer has sufficient permissions to create sources.

  • The following process requires you to complete steps in both the cloud.redhat.com Add a cloud source wizard and the IAM console. You must keep both applications open while you complete these steps. See the Additional Information links in the IAM console and the IAM documentation if you need help to complete the IAM tasks.

12.2.1. Adding the source type, name, and configuration mode

Select AWS as the source type, name the source, select the configuration mode, and create the application association.

Procedure

  1. In a browser window, go to cloud.redhat.com.
  2. If prompted, enter your Red Hat Customer Portal login credentials. The Hybrid Cloud Console opens.
  3. Click Settings (the gear icon) to show the settings options.
  4. In the navigation menu, click Sources.
  5. Click the Cloud sources tab if this page is not displayed by default. Click Add Source. The Add a cloud source wizard opens.

    Note

    You can also edit an existing source to add an association to the subscriptions service.

  6. Select the Amazon Web Services icon as the source type. Click Next.
  7. Enter a name for the source. This name is not required to be the same as the AWS account name. However, use a name that is easy to distinguish if you have multiple AWS accounts and must create multiple sources for them. Click Next.
  8. Select Manual configuration as the configuration mode. Click Next.
  9. Select RHEL management as the application. This selection provides the high-precision data capabilities of public cloud metering for the subscriptions service. Select other options as appropriate. Click Next.

12.2.2. Creating the IAM policy for public cloud metering

Create a policy for the AWS account. An IAM policy defines permissions for an AWS resource, for example, a role. This policy defines the actions that public cloud metering can perform on the AWS account.

Procedure

  1. Open the IAM console and then sign in to the console.
  2. Create a new IAM policy.
  3. In the Add a cloud source wizard, copy the policy document for public cloud metering.
  4. In the IAM console, paste the copied policy document into the JSON text box, replacing any default policy document information.
  5. Complete the process to create the new policy. Do not close the IAM console.
  6. In the wizard, click Next.

12.2.3. Creating the IAM role for public cloud metering

Create a role for the AWS account. An IAM role is an identity that can perform the actions that are defined by its associated policies. This role defines the actions that public cloud metering can perform on the AWS account.

Procedure

  1. In the IAM console, create a new role.
  2. For the trusted entity type, select Another AWS Account.
  3. In the Add a cloud source wizard, copy the public cloud metering account ID.
  4. In the IAM console, paste the copied public cloud metering account ID into the Account ID field for the role.
  5. In the permissions step of role creation, attach the new policy.
  6. Complete the process to create the new role. Do not close the IAM console.
  7. In the wizard, click Next.

12.2.4. Adding the IAM ARN to the source

Adding the ARN for the role to the source creates the connection between the subscriptions service and your account so that public cloud metering can begin collecting data.

Procedure

  1. In the IAM console, find and click the new role.
  2. In the Summary page for the role, copy the role ARN.
  3. In the Add a cloud source wizard, paste the copied ARN.
  4. Click Next.
  5. Review the details for this source. Click Add to complete the source creation.

12.2.5. Verification steps

During the final step of source creation in the Add a cloud source wizard, the connection to the AWS account is verified and an AWS CloudTrail trail is created for the account. The CloudTrail trail is used to monitor the start and stop events for instances, the raw data that is used to calculate usage data for display in the subscriptions service. If the verification and trail creation is successful, the source creation is successful. This process normally takes only a few seconds.

To find the RHEL images and the associated instances that it is going to track, public cloud metering must then perform an inspection of the AWS account. The length of this inspection process can vary according to many factors, including AWS performance, the number of images in the account, the size and type of each image, the number of instances for an image, and others. As a general rule, the inspection process for an image and its instances can take approximately one hour.

After the inspection process is complete, public cloud metering can begin reporting usage data to subscriptions. In most cases, reporting begins in subscriptions within 24 hours. However, because of the timing of source creation, the amount of time required for the inspection process, and the reporting intervals, or heartbeats, for Cloud Services platform tools, in rare cases you might have to wait up to 48 hours for this data to begin appearing in the subscriptions service.

12.3. How public cloud metering interacts with AWS

When you add an Amazon Web Services (AWS) account as a source and connect it to the RHEL management bundle, you are connecting the AWS account to the subscriptions service and the public cloud metering tool.

The public cloud metering data collection tool interacts with AWS to meter specific types of Red Hat Enterprise Linux usage in an AWS account. Public cloud metering communicates with your AWS account to gather high-precision data about the images and instances associated with the account.

To do those actions, public cloud metering must have access to your account and its data. This access is defined by a set of permissions. The public cloud metering tool must be able to assume an identity that has those permissions attached to it to communicate with the account.

You create objects that fulfill these requirements during source creation. You grant the access, permissions, and identity through the creation of an AWS Identity and Access Management (IAM) policy and role for the account. You then enable the connection between the account and the subscriptions service by associating the Amazon Resource Name (ARN) for the new role with the subscriptions service. Public cloud metering can then use the ARN for authentication into your account.

At the conclusion of this source creation, public cloud metering is enabled for the AWS account. It can start the image and instance inspection processes, determine which images and instances will be metered, and begin gathering data.

The permissions in the policy strictly limit the actions that public cloud metering can perform in your account. The allowed actions enable the inspection and metering tasks that public cloud metering performs, resulting in the gathering of usage analytics data for the account. This data is the basis for the data that is displayed in the subscriptions service.

The following information provides additional details about how public cloud metering interacts with your AWS accounts.

Note

The following information about the IAM role, policy, and ARN applies whether you select the account authorization configuration mode or the manual configuration mode during the creation of the AWS source. For the account authorization mode, these objects are created for you, but for the manual mode, you must create these objects.

12.3.1. How public cloud metering uses the IAM policy

During AWS source creation, you create a new policy in IAM. A policy defines which principal (for example, a role), has access to specific AWS resources. It also defines the actions that the principal can perform on those resources.

The newly created public cloud metering policy includes permissions for specific actions in your AWS account. The permissions defined by the policy, in combination with the newly created public cloud metering role, enable public cloud metering to do certain Amazon Elastic Compute Cloud (Amazon EC2) and AWS CloudTrail (CloudTrail) actions. These actions include discovering the current state of images and instances through inspection, copying images when needed to enable the inspection process, and creating and enabling an AWS CloudTrail trail.

The public cloud metering trail is configured to capture all write events in your AWS account. This trail directs its output to an Amazon Simple Storage Service (Amazon S3) bucket that is owned by Red Hat, so this new trail does not result in additional data storage costs for your account. When public cloud metering processes that trail output, it disregards any event that is not related to instance state and image tag changes.

The data that is collected from the Amazon EC2 activities and CloudTrail events enables public cloud metering to identify and to meter Red Hat Enterprise Linux usage.

12.3.2. How public cloud metering uses the IAM role

Also during AWS source creation, you create a role in IAM. A role is an AWS identity that is associated with one or more policies to govern the actions that the role can perform.

The newly created public cloud metering policy that grants permissions for specific actions in your AWS account attaches to the newly created public cloud metering role. Public cloud metering assumes the role to interact with your account to collect data about various Amazon EC2 activities.

12.3.3. How public cloud metering uses the IAM ARN

Lastly, during AWS source creation you associate the ARN with the subscriptions service.

This association enables the public cloud metering tool to authenticate to AWS. After authentication, public cloud metering can assume the new role and do the actions permitted by the new policy.

12.4. Actions allowed by the AWS Identity and Access Management policy

During the process to create a source for an Amazon Web Services (AWS) account, you create an AWS Identity and Access Management (IAM) policy. This policy includes permissions for public cloud metering to do specific actions in your AWS account.

The following information contains the actions that public cloud metering can perform in your AWS account.

12.4.1. Actions permitted in Amazon EC2

The Amazon Elastic Compute Cloud actions that public cloud metering can perform primarily include actions related to the inspection of images. An additional action relates to gathering details about existing instances for the metering process.

Table 12.1. Actions for Amazon EC2

ActionDescription

DescribeInstances

Enables public cloud metering to get information about the instances that are currently present in your AWS account.

DescribeImages

Enables public cloud metering to get information about the Amazon Machine Images (AMIs) that are used to start your instances.

DescribeSnapshots

Enables public cloud metering to get information about the snapshots for the AMIs.

ModifySnapshotAttribute

Enables public cloud metering to set an attribute that allows the copying of snapshots for inspection.

DescribeSnapshotAttribute

Enables public cloud metering to verify that the attribute that copies the snapshots is set.

CopyImage

Enables public cloud metering to make an intermediate copy of a privately shared third-party image into your account so that public cloud metering can subsequently copy the image into the public cloud metering AWS account for the purposes of inspection.

CreateTags

Enables public cloud metering to tag an intermediate copy of a privately shared third-party image to indicate where it came from.

12.4.2. Actions permitted in CloudTrail

The AWS CloudTrail actions that public cloud metering can perform are primarily related to the metering process.

Table 12.2. Actions for AWS CloudTrail

ActionDescription

CreateTrail

Enables public cloud metering to create an AWS CloudTrail trail in your account.

UpdateTrail

Enables public cloud metering to update a CloudTrail trail in your account.

PutEventSelectors

Enables public cloud metering to select the events that CloudTrail processes and logs.

DescribeTrails

Enables public cloud metering to get information about existing CloudTrail trails.

StartLogging

Enables public cloud metering to turn on logging for CloudTrail.

DeleteTrail

Enables public cloud metering to turn off logging and delete the CloudTrail trail when the source is deleted or when the subscriptions service is removed from its association with the source.

12.5. What happens during public cloud metering image inspection

After you create a source for an AWS account, public cloud metering inspects the contents of that account. The inspection process first finds each visible instance in the account, ignoring all instances that are in the process of being terminated. The inspection associates each instance with its parent Amazon Machine Image (AMI). The AMI ID for an image is saved for future instance identification.

After locating the image for a visible instance, the inspection process determines whether the image is a RHEL image. The inspection also determines whether it is appropriate to report the usage data for the instances of an image to the subscriptions service.

The amount of inspection required to determine if the image is a RHEL image and whether it is appropriate to report its instance usage data varies according to the type of image. For some types of images, a simple metadata inspection is enough to find known markers that identify it as a RHEL image and identify its origin. For other types of images, where these markers are not present, a deeper inspection of the file system is required for image identification.

The following information explains common types of images, how they are inspected, and whether usage data is reported for the instances:

Images that are ignored
Images with operating systems that are not RHEL are not relevant to the subscriptions service. Images that are encrypted or are marked as non-copyable cannot be fully inspected to discover the operating system metadata or the running instances. These images are ignored.
AWS Marketplace images

Amazon is an authorized reseller of Red Hat cloud platform products. The RHEL images available in AWS Marketplace might be offered directly by Amazon or by trusted third-party resellers. These RHEL images are inspected by public cloud metering to locate metadata that identifies them as AWS Marketplace images. However, usage data for the associated instances is not reported in the subscriptions service because the terms of use for these images, including any usage tracking or billing, are managed by Amazon.

Note

For some images that are offered in AWS Marketplace, the metadata inspection is not sufficient. For example, for copies of shared images, image metadata shows the owner as the user who made the copy. Such images are subject to the file system inspection process to discover more information about the images.

Red Hat Cloud Access

The Red Hat Cloud Access program enables you to use certain Red Hat product subscriptions on certified public cloud providers. Cloud Access images contain metadata that public cloud metering can use to bypass file system inspection of those images. The instance usage data that is associated with these Cloud Access images is reported in the subscriptions service.

Note

For some Cloud Access images, the metadata inspection is not sufficient. For example, for copies of shared images, image metadata shows the owner as the user who made the copy. Such images are subject to the file system inspection process to discover more information about the images.

Other images

For images that are not obtained directly from AWS Marketplace or Cloud Access but are obtained through other sources, the images are inspected and the instance usage data is reported through public cloud metering. These images could be copies of AWS Marketplace, AWS Community, or Cloud Access shared images or they could be images obtained through some other resource.

With these types of images, it is possible that they could contain markers that identify them as RHEL images, but the metadata inspection might not be sufficient for image identification. For example, for a copy of a shared image the owner metadata changes to the entity that made the copy, so owner data cannot be used to help identify the image. Therefore, a deeper inspection of the file system is needed to discover the markers for image identification.

The file system inspection process includes mounting the image into a running Red Hat instance and looking for markers that, among other data, show that the image is a RHEL image. As the phases of this file system inspection process are completed, artifacts such as image copies, snapshots, or volumes are deleted from the Red Hat instance.

For all images, regardless of type, the Amazon Machine Image (AMI) ID is retained to match instances to the correct image. When an instance is started, it is either matched to its parent image, or, if that image AMI ID is not found, the inspection process runs on that image to identify it and determine whether usage data is tracked for its instances.

12.5.1. Manually tagging AMIs as RHEL

The inspection process is optimized for more commonly used file systems that might be present in the AMI. For less commonly used file systems, RHEL cannot always be found during inspection. To work around this problem, you can manually tag the AMI as RHEL instead of using the inspection process to find RHEL.

When an AMI is tagged as RHEL and this tag is found during the initial steps of inspection, the remainder of the inspection process is skipped. The instances for that tagged image will be tracked by public cloud metering.

It is important to remember that not all AMIs that use less common file systems need to be tagged as RHEL. For example, a swap file system would not be used to run instances, so an AMI that has RHEL only in the swap file system would not need to be tagged. Current testing of the inspection process has shown that Oracle ZFS is an example of a file system where RHEL is more difficult to find. For these types of file systems, tagging AMIs as RHEL bypasses inspection while also ensuring that the instances will be tracked by public cloud metering.

Note

Previously, the Logical Volume Manager (LVM) file system was listed as a file system where AMIs needed to be tagged as RHEL to bypass inspection. The LVM file system is now a supported file system for RHEL image inspection as of October 2021. Those RHEL based AMIs no longer need to be tagged. No action is needed on AMIs that were previously manually tagged as RHEL.

To add and apply a custom tag for RHEL:

  1. From the AWS Management Console, navigate to the Tag Editor.
  2. Use Find a Resource to find AMI as a resource type.
  3. Add a tag and add the Tag key and Tag value values for the custom tag, using the following value for both fields:

    cloudigrade-rhel-present
  4. Navigate to the AMI resources, and then select the AMI for which you want to apply the custom RHEL tag.
  5. Repeat these steps for each AMI that is using a less common file system where RHEL is present in any partition in the AMI.