Chapter 32. Exporting analytics

This chapter describes how to create scripts that extend the current capabilities of the built-in 3scale analytics.

By using Account Management and Analytics API (Enterprise only), you can create scripts to retrieve information that you need in your preferred format. This chapter describes a particular use case, but the same approach can be used to get the data you need out of 3scale.

32.1. Reasons for customized scripts

3scale continuously improves the features available on your API Dashboard. However, you may be ahead of our development plans and have a very specific need not yet supported.

To satisfy the needs for API management, 3scale always gives you the tools to access all your data. However, customized scripts have some costs – it takes some resources to write the scripts. More importantly, customized scripts gives you total flexibility and control to implement use cases.

32.2. A real-world example

A customer was in the process of onboarding thousands of new developers per week. This customer could tackle some aspects of the onboarding because 3scale provides automates necessities such as key provisioning, sign-up workflows, and email communication. There was, however, something that was not possible to do with 3scale, which was quite important for them.

Since the customer was onboarding many people, the company needed a straight-forward way to classify the new developers based on their engagement with APIs so that their operations and marketing teams could interact with the new developers more effectively. At least at the required level of detail, such a feature was not yet available in the built-in analytics tools provided by 3scale. However, it was possible to extract data using 3scale Account and Analytics API because all the data was available in the system.

32.3. Example: Customer requirements

They would like to know how many new developers have signed up for the free evaluation plan in the last 10 days, split up different ways.

First, they wanted to know how many developers signed up but never used their API.

Second, they wanted to split the developers that had used their API into two groups:

  • Developers that used it for a period of time – say the first half of the 10 days – and then stopped using the API. These developers tried it out, but became inactive.
  • Developers that have been using the API consistently. For those, they would like to know the percent growth (or decline).

This information is available on 3scale’s built-in analytics. The problem is that there is no view to show it aggregated, which makes the whole experience quite cumbersome.

A typical answer to this problem would be this classification will be part of the upcoming analytics revamp. (Funny thing is that’s actually the case.) But that does not solve the problem if you need it now. We at 3scale want to give you the tools to do everything you need to do without depending on our release schedule.

32.4. Hands-on implementation

32.4.1. Winning recipe

This recipe usually gives better results in achieving custom work.

  • Tinker a bit with ActiveDocs. 3scale ActiveDocs, available in your Admin Portal, under the Account Settings (Gear icon in the top right corner) > Integrate > 3scale API Docs. 3scale has all of its APIs available as ActiveDocs so that you can try them out from your browser. This allows you to find the request that best serves your needs, get the request (curl-like) and get a grasp of the response. See an example…​

    DIY Analytics

This is the ActiveDocs for the API request to fetch all applications that will be used on the script to extend the analytics.

  • Once you have done the research with the ActiveDocs, put down the request on your scripting language of choice (ours is pedestrian Ruby).
  • Repeat until you have a script that does exactly what you need. For the example of the extended analytics, the script is available as a gist. You can try it out in your account.

ActiveDocs lets you quickly get a grasp of what the API is able to do. Then, it is a matter of finding which 3 or 4 requests are needed for the task you want to do and putting a script together.

32.4.2. Step-by-step guide

These were the steps taken to achieve the custom analytics that this customer wanted:

  1. Retrieve the full list of applications. This operation requires pagination. 

    def api_call_applications_list(domain, provider_key)
  done = false
  res = Array.new
  page = 1

      while !done
    url = "https://#{domain}/admin/api/applications.xml?provider_key=#{provider_key}&page=#{page}&per_page=100"
    page += 1
    response = RestClient.get url
    raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
    document = Nokogiri::XML(response.to_str)    done = document.xpath("applications/@current_page").text == document.xpath("applications/@total_pages").text
    document.xpath("//application").each do |item|
      app = Hash.new
      app["created_at"] = DateTime.parse(item.xpath("created_at").text)
      app["plan_name"] = item.xpath("plan/name").text
      app["service_id"] = item.xpath("plan/service_id").text
      app["account_id"] = item.xpath("user_account_id").text
      app["id"] = item.xpath("id").text
      res << app
    end
  end
  return res
end

  2. Filter the applications that do not meet the criteria, i.e. the plan must be "evaluation" and they have to be newer than 10 days. 

    def filter_applications(domain, provider_key, plan_name, num_of_days)
  res = api_call_applications_list(domain, provider_key)
  res.each do |item|
    res.delete(item) if item["plan_name"] != plan_name
    res.delete(item) if item["created_at"] > (DateTime.now - num_of_days)
  end
  return res
end

  3. Then for each application that meets the criteria, get its usage, meaning how many hits the application has had in the last 10 days. 

    def api_call_application_usage(domain, provider_key, application_id, metric, from, to, granularity)
  url = "https://#{domain}/stats/applications/#{application_id}/usage.xml?provider_key=#{provider_key}&metric_name=#{metric}&since=#{from}&until=#{to}&granularity=#{granularity}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  return document.xpath("//usage/data/values").text.split(",")
end

  4. Cross-reference the applications to the accounts, because the information of the developers is stored in the account object. 

    def api_call_account_read(domain, provider_key, account_id)
  url = "https://#{domain}/admin/api/accounts/#{account_id}.xml?provider_key=#{provider_key}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  account = Hash.new
  account["email"] = document.xpath("//users/user/email").text
  account["name"] = document.xpath("//users/user/first_name").text + " " + document.xpath("//users/user/last_name").text
  return account
end

    Now you only need to put everything together, and the script is done. You have a script that gets the information that was not yet available on 3scale’s built-in analytics. You can also get the full script as a gist.

32.5. Conclusions

3scale is very DIY-friendly. We believe that anyone should be able to extend the current capabilities of any service that they’ve contracted. As much as we would like to think that we can cover all the bases, we will always be a bit behind in certain aspects, whether because it’s a super special need or simply because we are busy building other features. Whatever the case, we want to avoid blocking you when something is not possible right away. We give you full access to your data and a complete API toolset to work with it.

If you have scripts that you want to share with us and the rest of 3scale’s users, please send them to us. We would be happy to collect them and publish them.