Exporting cost data using the API

OpenShift Container Platform 4.3

Using the data export request API

Red Hat Customer Content Services

Abstract

This guide provides instructions for using the data export API to obtain cost data for reporting.

Chapter 1. Exporting cost data using the API

You can use the cost management data export request API to obtain cost data for your sources, including information about usage, tags, cost model assignments, and more.

You can then use the exported cost data with your external business integration tools to generate custom reports, and to view and compare your data in additional ways to those provided by the cost management application.

The API also allows you to view historical cost data beyond what is available to view in the cost management user interface, back to the date a source was originally added to cost management. (Currently, the cost management user interface shows the previous 2-3 months of data).

This guide provides instructions for using the API to export your cost data.

Note

Cost management is currently Technology Preview. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

If you have a suggestion for improving this guide or have found an error, please submit a Bugzilla report at http://bugzilla.redhat.com against Cloud Software Services (cloud.redhat.com) for the Cost Management component.

Chapter 2. How the data export request API works

The API allows you to export cost data for your sources from cost management’s database.

The cost management application runs a process every 24 hours to collect cost data from each AWS source, using AWS Cost and Usage Reports. Cost management stores these files in an AWS S3 bucket which only the application can access.

The data export request API retrieves this data using an API request. The data export includes results from cost management’s daily and daily summary database tables and are bound by specific requested dates. Data in the exported files is separated by cost management account, source, date, and table name.

Each individual exported file contains the data for one specific cost management account on one specific date from one database query.

The API writes these files in CSV format to the AWS S3 bucket specified in the request.

Chapter 3. Exporting cost data to an AWS bucket

To export cost data to your AWS bucket as a CSV file, provide a POST request with your desired start date, end date, and AWS bucket.

You can request data as far back as the date you added the source to cost management.

Prerequisites

Using the cost management data export request API requires:

  • An AWS bucket configured to receive (list and write) files from cost management. As administrator for your AWS account:

    1. Create a new AWS S3 bucket or choose an existing one to use for receiving data exports from cost management.
    2. Edit this bucket’s Access Control List.
    3. Add Access for other AWS accounts to allow cost management to list and write files with the following settings:

      • Canonical ID (for cost management): 0674b6ccec2e5d6212b35910e6c5fac24e94cee66371847b46c981edfa3df63c
      • List objects: Yes
      • Write objects: Yes
  • Your Red Hat login and password to authenticate to the data export request API.

Procedure

  1. Determine the time frame you want to obtain cost data for.
  2. Create a POST request specifying:

    • start_date - This value is inclusive: the first date in the time frame to obtain data for, formatted as yyyy-mm-dd.
    • end_date - This value is exclusive: the date following the last date in your desired time frame, formatted as yyyy-mm-dd.
    • bucket_name - The name of your destination AWS S3 bucket.

      Use the following example for reference:

      http localhost:8000/api/cost-management/v1/dataexportrequests/ \
          HTTP_X_RH_IDENTITY:"${IDENTITY}" \
          start_date='2019-09-01' \
          end_date='2019-10-01' \
          bucket_name='cost management-customer-sample-bucket'

After running the request, cost management will return a response to confirm it has started the request process, but the remainder of the operations are asynchronous. To see the current state of your data export request, you must check back at the API.

Note

Cost management also checks for any in-progress requests with the same date parameters as that request. If cost management is already working on an export with the same dates, the new request is rejected with a 400 error and the message "A pending or processing data export already exists with the given "start_date" and "end_date"."

When complete, the exported files are written with hierarchical paths that include the cost management account ID, source/provider UUID, and date. An example file path looks like:

/data_archive/acct1460290/aws/4cef9a22-4302-4776-a4b7-fac050da23a0/2019/11/24/reporting_awscostentrylineitem_daily_summary.csv.gz

Additional resources

Chapter 4. Viewing previous data export requests

You can use the API to list previous data export requests to prevent duplicating data. This also checks if the user has requested the same data multiple times before the request has completed.

Most data export requests complete in a few seconds, however, this can be useful for troubleshooting processing delays or other issues.

Procedure

  1. To retrieve a list of all data export requests that were created by the authenticated user, run a GET request with the URL /api/cost-management/v1/dataexportrequests/:

    HTTP GET localhost:8000/api/cost-management/v1/dataexportrequests/

    For example, using curl, run:

    $  curl -s -u "$MYUSER:$MYPASS" -F start_date='2019-11-01' -F end_date='2019-11-10' -F bucket_name='koku-customer-sample-bucket' localhost:8000/api/cost-management/v1/dataexportrequests/ | jq
    {
     "uuid": "8c619402-3cf9-4b19-a01b-9942d0cd6c1b",
     "created_timestamp": "2019-11-26T19:35:13.437243Z",
     "updated_timestamp": "2019-11-26T19:35:13.437275Z",
     "start_date": "2019-11-01",
     "end_date": "2019-11-10",
     "status": "pending",
     "bucket_name": "koku-customer-sample-bucket"
    }

    The response provides the request UUID, and shows the request is still in progress.

  2. To retrieve a requested data export request by UUID (assuming it is reachable by the authenticated user), run a GET request with the URL /api/cost-management/v1/dataexportrequests/{uuid}/, using the request UUID:

    HTTP GET localhost:8000/api/cost-management/v1/dataexportrequests/{uuid}/

    For example, using curl, run:

    $  curl -s -u "$MYUSER:$MYPASS" localhost:8000/api/cost-management/v1/dataexportrequests/8c619402-3cf9-4b19-a01b-994
    2d0cd6c1b/ | jq -C
    {
     "uuid": "8c619402-3cf9-4b19-a01b-9942d0cd6c1b",
     "created_timestamp": "2019-11-26T19:35:13.437243Z",
     "updated_timestamp": "2019-11-26T19:35:25.232298Z",
     "start_date": "2019-11-01",
     "end_date": "2019-11-10",
     "status": "complete",
     "bucket_name": "koku-customer-sample-bucket"
    }

The response shows the request status is complete.

Chapter 5. Data export request API parameters

This table provides details on the parameters available to use for data export requests in the API.

Table 5.1. Data Export Request API parameters

ParameterFormatExampleDetails

start_date

ISO-8601 formatted start date (inclusive)

2019-09-01

The first day in a date range you want to view cost data for.

end_date

ISO-8601 formatted end date (exclusive)

2019-10-01

The end date is outside the desired date range for the request.

bucket_name

Destination Amazon S3 bucket name

koku-customer-sample-bucket

The bucket must be configured to allow cost management to write files to it.

Example request

The following example requests data for the entire month of September 2019, to be written to the koku-customer-sample-bucket AWS S3 bucket:

http localhost:8000/api/cost-management/v1/dataexportrequests/ \
    HTTP_X_RH_IDENTITY:"${IDENTITY}" \
    start_date='2019-09-01' \
    end_date='2019-10-01' \
    bucket_name='koku-customer-sample-bucket'

Legal Notice

Copyright © 2020 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.