Chapter 33. Exporting analytics

This article shows how to create scripts that extend the current capabilities of the built-in 3scale analytics.

By using the Account Management and Analytics API (Enterprise only), you can create scripts that get the custom information that you need in the format that you prefer. This article describes a particular use case, but the same approach can be used to get whatever data you need out of 3scale.

33.1. Reasons for customized scripts

3scale continuously improves the features available on your API Dashboard. However, you may be ahead of our development plans and have a very specific need that’s not yet supported.

To satisfy the needs for API management, 3scale always gives you the tools to access all your data. However, customized scripts have some costs – it takes some resources to write the scripts – but the cost is not too high, as we will show in this article. More importantly, DIY gives you total flexibility and control to implement any use case you can think of.

33.2. A real-world example

Recently a customer came to us with a very specific need that could not be solved with 3scale in a streamlined way.

They were onboarding thousands of new developers per week. They were able to survive such success thanks to the 3scale API Management Platform. Onboarding so many developers would be a daunting task – luckily 3scale provides automates necessities such as key provisioning, sign-up workflows, and email communication. So far so good.

There was, however, something that was not possible to do with 3scale, which was quite important for them. Since they are onboarding so many people, they needed a straight-forward way to classify the new developers based on their engagement with their API so that their operations and marketing teams could interact with the new developers more effectively.

Such a feature, at least at the required level of detail, is not yet available in 3scale’s built-in analytics tools. However since all the data is already available, it can be extracted using 3scale’s Account and Analytics API.

33.3. Example: Customer requirements

They would like to know how many new developers have signed up for the free evaluation plan in the last 10 days, split up different ways.

First, they wanted to know how many developers signed up but didn’t ever use their API. What they wanted to do with this information is out of scope for this article, but it’s clearly valuable information that would help adoption of your API.

Second, they wanted to split the developers that had used their API into two groups:

  • Developers that used it for a period of time – say the first half of the 10 days – and then stopped using the API. These developers tried it out, but became inactive.
  • Developers that have been using the API consistently. For those, they would like to know the percent growth (or decline).

This information is available on 3scale’s built-in analytics. The problem is that there is no view to show it aggregated, which makes the whole experience quite cumbersome.

A typical answer to this problem would be this classification will be part of the upcoming analytics revamp. (Funny thing is that’s actually the case.) But that does not solve the problem if you need it now. We at 3scale want to give you the tools to do everything you need to do without depending on our release schedule.

33.4. Hands-on implementation

33.4.1. Winning recipe

This recipe usually gives better results in achieving custom work.

  • Tinker a bit with ActiveDocs. 3scale ActiveDocs, available in your Admin Portal, under the Account Settings (Gear icon in the top right corner) > Integrate > 3scale API Docs. 3scale has all of its APIs available as ActiveDocs so that you can try them out from your browser. This allows you to find the request that best serves your needs, get the request (curl-like) and get a grasp of the response. See an example…​

    DIY Analytics

This is the ActiveDocs for the API request to fetch all applications that will be used on the script to extend the analytics.

  • Once you’ve done the research with the ActiveDocs, put down the request on your scripting language of choice (ours is pedestrian Ruby). Thanks to the request and responses provided by ActiveDocs, it has never been easier to explore what the API does.
  • Repeat until you have a script that does exactly what you need. For the example of the extended analytics, the script is available as a gist. You can try it out in your account.

As pedestrian as it sounds, this is the easiest way to proceed. The ActiveDocs let you quickly get a grasp of what the API is able to do. Then, it’s just a matter of finding which 3 or 4 requests are needed for the task you want to do and putting a script together.

33.4.2. Step-by-step guide

These were the steps taken to achieve the custom analytics that this customer wanted:

  • Retrieve the full list of applications. This operation requires pagination, but as you can see, it’s dead simple.
def api_call_applications_list(domain, provider_key)
  done = false
  res = Array.new
  page = 1

      while !done
    url = "https://#{domain}/admin/api/applications.xml?provider_key=#{provider_key}&page=#{page}&per_page=100"
    page += 1
    response = RestClient.get url
    raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
    document = Nokogiri::XML(response.to_str)    done = document.xpath("applications/@current_page").text == document.xpath("applications/@total_pages").text
    document.xpath("//application").each do |item|
      app = Hash.new
      app["created_at"] = DateTime.parse(item.xpath("created_at").text)
      app["plan_name"] = item.xpath("plan/name").text
      app["service_id"] = item.xpath("plan/service_id").text
      app["account_id"] = item.xpath("user_account_id").text
      app["id"] = item.xpath("id").text
      res << app
    end
  end
  return res
end
  • Filter the applications that do not meet the criteria, i.e. the plan must be "evaluation" and they have to be newer than 10 days.
def filter_applications(domain, provider_key, plan_name, num_of_days)
  res = api_call_applications_list(domain, provider_key)
  res.each do |item|
    res.delete(item) if item["plan_name"] != plan_name
    res.delete(item) if item["created_at"] > (DateTime.now - num_of_days)
  end
  return res
end
  • Then for each application that meets the criteria, get its usage, meaning how many hits the application has had in the last 10 days.
def api_call_application_usage(domain, provider_key, application_id, metric, from, to, granularity)
  url = "https://#{domain}/stats/applications/#{application_id}/usage.xml?provider_key=#{provider_key}&metric_name=#{metric}&since=#{from}&until=#{to}&granularity=#{granularity}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  return document.xpath("//usage/data/values").text.split(",")
end
  • Finally, you must cross-reference the applications to the accounts because the information of the developers is stored in the account object.
def api_call_account_read(domain, provider_key, account_id)
  url = "https://#{domain}/admin/api/accounts/#{account_id}.xml?provider_key=#{provider_key}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  account = Hash.new
  account["email"] = document.xpath("//users/user/email").text
  account["name"] = document.xpath("//users/user/first_name").text + " " + document.xpath("//users/user/last_name").text
  return account
end

And that’s pretty much it. Now you only need to put everything together (you can get the full script as a gist) and you’re done. You have a script that gets the information that was not yet available on 3scale’s built-in analytics.

33.5. Conclusions

3scale is very DIY-friendly. We believe that anyone should be able to extend the current capabilities of any service that they’ve contracted. As much as we would like to think that we can cover all the bases, we will always be a bit behind in certain aspects, whether because it’s a super special need or simply because we are busy building other features. Whatever the case, we want to avoid blocking you when something is not possible right away. We give you full access to your data and a complete API toolset to work with it.

If you have scripts that you want to share with us and the rest of 3scale’s users, please send them to us. We would be happy to collect them and publish them.