Chapter 3. Adding sources to cost management

To use cost management to monitor your cloud costs, you must first connect a data source to the cost management application.

Currently, cost management can track costs for Amazon Web Services (AWS), Microsoft Azure, and Red Hat OpenShift Container Platform sources.

3.1. Adding an Amazon Web Services (AWS) source to cost management

To add an AWS account to cost management, you must configure your AWS account to provide metrics, then add your AWS account as a source from the cost management user interface.

Note

You must have a Red Hat account user with Organization Administrator entitlements before you can add sources to cost management.

When you add your AWS account as a source, this creates a read-only connection to AWS in order to collect cost information hourly in cost management, but does not make any changes to the AWS account.

Important

You must use an AWS master account for this procedure, as a linked AWS account does not have sufficient access to create billing accounts. After you add the master account as a source, cost management will collect data from any linked accounts as well.

Before you can add your AWS account to cost management as a data source, you must configure the following services on your AWS account to allow cost management access to metrics:

  1. An S3 bucket to store cost and usage data reporting for cost management
  2. An Identity Access Management (IAM) policy and role for cost management to process the cost and usage data

As you will complete some of the following steps in the AWS console, and some steps in the cost management user interface, keep both applications open in a web browser.

Add your AWS source to cost management from the settings area at https://cloud.redhat.com/settings/sources/.

Note

As non-Red Hat products and documentation can change without notice, instructions for configuring the third-party sources provided in this guide are general and correct at the time of publishing. See the AWS documentation for the most up-to-date and accurate information.

3.1.1. Creating an S3 bucket for reporting

Cost management requires an Amazon S3 bucket with permissions configured to store billing reports.

Log into your AWS master account to begin configuring cost and usage reporting:

  1. In the AWS S3 console, create a new S3 bucket or use an existing bucket. If you are configuring a new S3 bucket, accept the default settings.
  2. In the AWS Billing console, create a Cost and Usage Report that will be delivered to your S3 bucket. Specify the following values (and accept the defaults for any other values):

    • Report name: <any-name> (note this name as you will use it later)
    • Additional report details: Include resource IDs
    • S3 bucket: <the S3 bucket you configured previously>
    • Time granularity: Hourly
    • Enable report data integration for: Amazon Redshift, Amazon QuickSight (do not enable report data integration for Amazon Athena)
    • Compression type: GZIP
    • Report path prefix: cost

      Note

      See the AWS Billing and Cost Management documentation for more details on configuration.

  3. In the cloud.redhat.com platform, open the Sources menu (https://cloud.redhat.com/settings/sources/) to begin adding an AWS source to cost management:

    1. Navigate to Sources and click Add a source to open the Sources wizard.
    2. Enter a name for your source and click Next.
    3. Select Cost Management as the application and Amazon Web Services (AWS) as the source type. Click Next.
    4. Paste the name of your S3 bucket and click Next.

3.1.2. Activating AWS tags for cost management

To use tags to organize your AWS resources in the cost management application, activate your tags in AWS to allow them to be imported automatically.

Procedure

  1. In the AWS Billing console:

    1. Open the Cost Allocation Tags section.
    2. Select the tags you want to use in the cost management application, and click Activate.
  2. In the cloud.redhat.com Sources wizard, click Next to move to the next screen.

3.1.3. Enabling minimal account access for cost and usage consumption

To provide data within the web interface and API, cost management needs to consume the Cost and Usage Reports produced by AWS. For cost management to obtain this data with a minimal amount of access, create an IAM policy and role for cost management to use. This configuration provides access to the stored information and nothing else.

Procedure

  1. From the AWS Identity and Access Management (IAM) console, create a new IAM policy for the S3 bucket you configured previously.

    1. Select the JSON tab and paste the following content in the JSON policy text box:

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "s3:Get*",
              "s3:List*"
            ],
              "Resource": [
              "arn:aws:s3:::bucket_name",
              "arn:aws:s3:::bucket_name/*"
            ]
          },
      
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
              "s3:HeadBucket",
              "cur:DescribeReportDefinitions"
            ],
            "Resource": "*"
          }
        ]
      }
  1. Provide a name for the policy and complete the creation of the policy. Keep the AWS IAM console open as you will need it for the next step.

    1. In the cloud.redhat.com Sources wizard, click Next to move to the next screen.
    2. In the AWS IAM console, create a new IAM role:
  2. For the type of trusted entity, select Another AWS account.
  3. Enter 589173575009 as the Account ID to provide the cost management application with read access to the AWS account cost data.
  4. Attach the IAM policy you just configured.
  5. Enter a role name (and description if desired) and finish creating the policy.

    1. In the cloud.redhat.com Sources wizard, click Next to move to the next screen.
    2. In the AWS IAM console under Roles, open the summary screen for the role you just created and copy the Role ARN (a string beginning with arn:aws:).
    3. In the cloud.redhat.com Sources wizard, paste your Role ARN and click Next.
    4. Review the details and click Finish to add the AWS account to cost management.

Cost management will begin collecting cost and usage data from your master AWS account and any linked AWS accounts.

The data can take a few days to populate before it shows on the cost management dashboard (https://cloud.redhat.com/cost-management/).

3.1.3.1. Enabling additional account access for cost and usage consumption

Cost management can display additional data that might be useful. For example:

  • Include the Action iam:ListAccountAliases to display an AWS account alias rather than an account number in cost management.
  • Include the Actions organization:List* and organizations:Describe* to obtain the display names of AWS member accounts if you are using consolidated billing rather than the account ID.

The following configuration provides access to additional stored information and nothing else.

Procedure

  1. From the AWS Identity and Access Management (IAM) console, create a new IAM policy for the S3 bucket you configured previously.
  2. Select the JSON tab and paste the following content in the JSON policy text box:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
            "s3:Get*",
            "s3:List*"
          ],
          "Resource": [
            "arn:aws:s3:::bucket",
            "arn:aws:s3:::bucket/*"
          ]
        },
        {
          "Sid": "VisualEditor1",
          "Effect": "Allow",
          "Action": [
            "iam:ListAccountAliases",
            "s3:HeadBucket",
            "cur:DescribeReportDefinitions",
            "organizations:List*",
            "organizations:Describe*"
          ],
          "Resource": "*"
        }
      ]
    }

    The remainder of the configuration steps are the same as in Section 3.1.3.1, “Enabling additional account access for cost and usage consumption”.

You have completed adding your AWS account as a source.

3.2. Adding a Microsoft Azure source to cost management

This section describes how to configure your Microsoft Azure account to allow cost management access.

Configuring your Azure account to be a cost management source requires:

  1. Creating a storage account and resource group
  2. Configuring a Storage Account Contributor and Reader roles for access
  3. Scheduling daily cost exports
Note

As non-Red Hat products and documentation can change without notice, instructions for configuring the third-party sources provided in this guide are general and correct at the time of publishing. See the Microsoft Azure documentation for the most up-to-date and accurate information.

Add your Azure source to cost management from https://cloud.redhat.com/settings/sources/.

3.2.1. Creating an Azure resource group and storage account

Cost export data is written to a storage account, which exists within a resource group. The resource group must be accessible by cost management in order to read the Azure cost data.

Create a new storage account in Azure to contain the cost data and metrics that cost management will collect. This requires a resource group; Red Hat recommends creating a dedicated resource group for this storage account.

Note

You must have a Red Hat account user with Organization Administrator entitlements before you can add sources to cost management.

  1. In the cloud.redhat.com platform, open the Sources menu (https://cloud.redhat.com/settings/sources/) to begin adding an Azure source to cost management:

    1. Navigate to Sources and click Add a source to open the Sources wizard.
    2. Enter a name for your source and click Next.
    3. Select Cost Management as the application and Microsoft Azure as the source type. Click Next.
  2. Create a resource group and storage account in your Azure account using the instructions in the See Azure documentation Create a storage account.

Make a note of the resource group and storage account. They will be needed in subsequent steps.

  1. In the cloud.redhat.com Sources wizard, enter the Resource group name and Storage account name and click Next.

3.2.2. Configuring Azure roles

Red Hat recommends configuring dedicated credentials to grant cost management read-only access to Azure cost data. Configure a Storage Account Contributor and Reader role in Azure to provide this access to cost management.

  1. In Azure Cloud Shell, run the following command to obtain your Subscription ID:

    $ az account show --query "{subscription_id: id }"
  2. In the cloud.redhat.com Sources wizard, enter your Subscription ID. Click Next to move to the next screen.
  3. In Azure Cloud Shell, run the following command to create a Cost Management Storage Account Contributor role, and obtain your tenant ID, client (application) ID, and client secret:

    $ az ad sp create-for-rbac -n "CostManagement" --role "Storage Account Contributor" --query '{"tenant": tenant, "client_id": appId, "secret": password}'
  4. In the cloud.redhat.com Sources wizard, enter your Azure Tenant ID, Client ID, and Client Secret.
  5. In Azure Cloud Shell, run the following command to create a Cost Management Reader role with your subscription ID. Copy the full command from the cloud.redhat.com Sources wizard, which will automatically substitute your Azure subscription ID obtained earlier for <SubscriptionID>:

    $ az role assignment create --role "Cost Management Reader" --assignee http://CostManagement --subscription <SubscriptionID>
  6. Click Next.

3.2.3. Configuring a daily Azure data export schedule

Create a recurring task to export your cost data on a daily basis automatically to your Azure storage account, where cost management will retrieve the data.

  1. In Azure, add a new export as described in the instructions in the Azure article Create and manage exported data.

    • For Export type, select Daily export of billing-period-to-date costs.
    • For Storage account, select the account you created earlier.
    • Enter any value for the container name and directory path for the export. These values provide the tree structure in the storage account where report files are stored.
    • Click Run now to start exporting data to the Azure storage container.
  2. In the cloud.redhat.com Sources wizard, click Next when you have created the export schedule and review the source details.
  3. Click Finish to complete adding the Azure source to cost management.

After the schedule is created, cost management will begin polling Azure for cost data, which will appear on the cost management dashboard (https://cloud.redhat.com/cost-management/).

You have completed adding your Azure account as a source.

3.3. Adding an OpenShift Container Platform source to cost management

To add an OpenShift Container Platform cluster as a source to cost management, you must first configure your cluster to provide usage data (metrics) using the Cost Management Operator.

Note

You must have a Red Hat account user with Organization Administrator entitlements before you can add sources to cost management.

The Cost Management Operator (cost-mgmt-operator) collects the metrics required for cost management by:

  • Using Operator Metering to create usage reports specific to cost management.
  • Collecting and packaging these reports to a tarball which is uploaded to cost management through cloud.redhat.com.
Note

An OpenShift Container Platform 4.3 or newer cluster is required to use the Cost Management Operator.

To add your OpenShift Container Platform cluster as a cost management source:

  1. Install the Cost Management Operator in OpenShift from OperatorHub
  2. Configure the Cost Management Operator to collect OpenShift usage data (metrics) using Operator Metering
  3. Provide the cluster identifier to cost management

As you will complete some of the following steps in OpenShift Container Platform, and some steps in the cloud.redhat.com platform (https://cloud.redhat.com/settings/sources/), have both applications open in a web browser, as well as a terminal to access the command line interface (CLI).

3.3.1. Installing the Cost Management Operator

The Cost Management Operator collects the metrics required for cost management.

Begin adding your OpenShift Container Platform cluster as a source to cost management, then install the Cost Management Operator from OperatorHub.

Note

See Operators in the OpenShift documentation for more information about Operators and OperatorHub.

Prerequisites

  • OpenShift Container Platform 4.3 or newer

Procedure

  1. In the cloud.redhat.com platform, open the Sources menu (https://cloud.redhat.com/settings/sources/) to begin adding an OpenShift source to cost management:

    1. Navigate to Sources and click Add source to open the Sources wizard.
    2. Enter a name for your source and click Next.
    3. Select Cost Management as the application and OpenShift Container Platform as the source type. Click Next.
  2. In OpenShift, create a namespace called openshift-metering if one does not exist, and label the namespace with openshift.io/cluster-monitoring=true.
  3. In OpenShift, install the Cost Management Operator in the openshift-metering namespace, using either the OpenShift web console (search for cost management in OperatorHub) or the CLI.

    Important

    You must install the Cost Management Operator in the openshift-metering namespace. Other namespaces are not supported for installation.

    See Adding Operators to a cluster in the OpenShift documentation for instructions for installing an Operator.

Additional resources

  • See Metering in the OpenShift documentation for more information about installing Metering.

3.3.2. Configuring the Cost Management Operator

The Cost Management Operator (cost-mgmt-operator) collects the metrics required for cost management.

After installing the Cost Management Operator, configure authentication and the operator-metering namespace, then configure the Cost Management Operator.

Prerequisites

  • OpenShift Container Platform 4.3 or newer
  • The Cost Management Operator installed in the openshift-metering namespace
  • A user with access to the openshift-config namespace

Procedure

  1. Configure authentication inside the openshift-metering project. This allows you to upload OpenShift data to cloud.redhat.com.

    Note

    For most installations you can use token authentication or basic authentication to upload the usage reports (metrics) to cost management. Except for Azure RedHat OpenShift installation, the default and recommended method is token authentication.

    Note

    If you are performing an Azure RedHat OpenShift installation managed by Azure, you must use basic authentication. Token authentication is not supported for Azure-managed installations.

    1. Copy the following into a file called auth_secret.yaml:

      kind: Secret
      apiVersion: v1
      metadata:
        name: auth-secret-name
        namespace: openshift-metering
        annotations:
          kubernetes.io/service-account.name: cost-mgmt-operator
      data:
        username: >-
          Y2xvdWQucmVkaGF0LmNvbSB1c2VybmFtZQ==
        password: >-
          Y2xvdWQucmVkaGF0LmNvbSBwYXNzd29yZA==
        token: >-
          Y2xvdWQucmVkaGF0LmNvbSB0b2tlbg==
    2. Choose a name for your authentication secret and replace the metadata.name value with it.
    3. To configure token authentication (the default method), obtain the correct auth token and then edit the secret to replace the token value:

      1. Install the jq JSON processor.
      2. Change to the openshift-config namespace:

        $ oc project openshift-config
      3. Replace the token value in auth_secret.yaml with the authentication token for cloud.openshift.com. Obtain the token by running the following command, and copy only the "tokenvalue" to auth_secret.yaml (excluding the quotation marks):

        $ oc get secret pull-secret -o "jsonpath={.data.\.dockerconfigjson}" | base64 --decode | jq '.auths."cloud.openshift.com".auth'
        Note

        To use basic authentication, edit the secret to replace the username and password values with your base64-encoded username and password for connecting to cloud.redhat.com.

    4. Deploy the secret to your OpenShift cluster in the openshift-metering namespace:

      $ oc create -f auth-secret.yaml

      For both methods of authentication, the name of the secret should match the authentication_secret_name set in the CostManagement custom resource configured in the next steps.

  2. Configure the Metering Operator.

    Cost management uses the Metering Operator to create, collect, package, and upload metrics to cost management. In order for metering to work properly, configure operator-metering using the OpenShift documentation to create a MeteringConfig resource.

  3. Configure the Cost Management Operator by creating the CostManagement and CostManagementData custom resources.

    Creating these resources also starts the roles that create the resources to obtain the usage reports (metrics). This takes about an hour to run and the reports are collected, packaged, and uploaded every six hours.

    Note

    The Cost Management Operator requires the clusterID, reporting_operator_token_name, and authentication_secret_name to be specified in a CostManagement custom resource.

    1. Copy the following CostManagement resource template and save it to a file called cost-mgmt-resource.yaml:

      apiVersion: cost-mgmt.openshift.io/v1alpha1
      kind: CostManagement
      metadata:
        name: cost-mgmt-setup
      spec:
        clusterID: '123a45b6-cd8e-9101-112f-g131415hi1jk'
        reporting_operator_token_name: 'reporting-operator-token-123ab'
        validate_cert: 'false'
        authentication: 'basic'
        authentication_secret_name: 'basic_auth_creds-123ab'
    2. Edit the following values in your cost-mgmt-resource.yaml file:

      • The clusterID value to your cluster ID. Obtain your cluster ID by running:

        $ oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'
      • The reporting_operator_token_name to the reporting-operator-token secret name inside the openshift-metering namespace. Obtain this by running:

        $ oc get secret -n openshift-metering | grep reporting-operator-token
        Note

        Depending on your configuration, this command can return two token names. You can use either to configure the Cost Management Operator.

      • Specify the authentication type you are using (token or basic). If you are using token authentication, you can remove the authentication field as token authentication is the default.
      • Change the authentication_secret_name to the name of your authentication secret you created earlier.
    3. Deploy the CostManagement resource:

      $ oc create -f cost-mgmt-resource.yaml
    4. Create a CostManagementData resource to start the collection. Copy the following template and save it as cost-mgmt-data-resource.yaml:

      apiVersion: cost-mgmt-data.openshift.io/v1alpha1
      kind: CostManagementData
      metadata:
        name: cost-mgmt-data-example
    5. Deploy the CostManagementData resource:

      $ oc create -f cost-mgmt-data-resource.yaml

      The Cost Management Operator will now create, collect, package, and upload your OpenShift usage reports to cost management.

  4. When configuration is complete, enter the cluster identifier into the cloud.redhat.com Sources wizard, click Next.

    Note

    The cluster identifier can be found in Help > About in OpenShift.

  5. In the cloud.redhat.com Sources wizard, review the details and click Finish to add the OpenShift Container Platform cluster to cost management.

Additional resources

  • See Operators in the OpenShift documentation for more information about Operators and OperatorHub.

Cost management will begin collecting usage data (metrics) from your OpenShift Container Platform cluster. The data can take a few days to populate before it shows on the cost management dashboard.

You have completed adding your OpenShift Container Platform cluster as a source.