Chapter 32. Extending analytics

This article shows how to create scripts that extend the current capabilities of the built-in Red Hat 3scale API Management analytics.

By using the Account Management and Analytics API on Enterprise accounts only, you can create scripts that get the custom information that you need in the format that you prefer. This article describes a particular use case, but the same approach can be used to get whatever data you need out of 3scale.

32.1. Reasons for customized scripts

3scale continuously improves the features available on your API Dashboard. However, you may be ahead of our development plans and have a very specific need that is not yet supported.

To satisfy the needs for API management, 3scale always gives you the tools to access all your data. However, customized scripts have some costs – it takes some resources to write the scripts – but the cost is not too high, as we will show in this article. More importantly, DIY gives you total flexibility and control to implement any use case you can think of.

32.2. A real-world example

Recently a customer came to us with a very specific need that could not be solved with 3scale in a streamlined way.

They were onboarding thousands of new developers per week. They were able to survive such success thanks to the 3scale API Management Platform. Onboarding so many developers would be a daunting task – luckily 3scale provides automates necessities such as key provisioning, sign-up workflows, and email communication. So far so good.

There was, however, something that was not possible to do with 3scale, which was quite important for them. Since they are onboarding so many people, they needed a straight-forward way to classify the new developers based on their engagement with their API so that their operations and marketing teams could interact with the new developers more effectively.

Such a feature, at least at the required level of detail, is not yet available in 3scale built-in analytics tools. However since all the data is already available, it can be extracted using the 3scale Account and Analytics API.

32.3. Example: Customer requirements

They would like to know how many new developers have signed up for the free evaluation plan in the last 10 days, split up different ways.

First, they wanted to know how many developers signed up but did not use their API. What they wanted to do with this information is out of scope for this article, but it is clearly valuable information that would help adoption of your API.

Second, they wanted to split the developers that had used their API into two groups:

  • Developers that used it for a period of time – say the first half of the 10 days – and then stopped using the API. These developers tried it out, but became inactive.
  • Developers that have been using the API consistently. For those, who would like to know the percentage growth or decline.

This information is available on 3scale built-in analytics. The problem is that there is no view to show it aggregated, which makes the whole experience quite cumbersome.

A typical answer to this problem would be this classification will be part of the upcoming analytics revamp. But that does not solve the problem if you need it now. We at 3scale want to give you the tools to do everything you need to do without depending on our release schedule.

32.4. Hands-on implementation

32.4.1. Winning recipe

This recipe usually gives better results in achieving custom work.

  • Tinker a bit with ActiveDocs. 3scale ActiveDocs, available in your Admin Portal, under the Account Settings > Integrate > 3scale API Docs. 3scale has all of its APIs available as ActiveDocs so that you can try them out from your browser. This allows you to find the request that best serves your needs, get the request (curl-like) and get a grasp of the response. See an example:

    DIY Analytics

This is the ActiveDocs for the API request to fetch all applications that will be used on the script to extend the analytics.

  • Once you have done the research with the ActiveDocs, put down the request on your scripting language of choice, for example Ruby. Thanks to the request and responses provided by ActiveDocs, it has never been easier to explore what the API does.
  • Repeat until you have a script that does exactly what you need. For the example of the extended analytics, the script is available as a gist. You can try it out in your account.

As pedestrian as it sounds, this is the easiest way to proceed. The ActiveDocs let you quickly get a grasp of what the API is able to do. Then, it is just a matter of finding which 3 or 4 requests are needed for the task you want to do and putting a script together.

32.4.2. Step-by-step guide

These were the steps taken to achieve the custom analytics that this customer wanted:

  • Retrieve the full list of applications. This operation requires pagination. 
def api_call_applications_list(domain, provider_key)
  done = false
  res = Array.new
  page = 1

      while !done
    url = "https://#{domain}/admin/api/applications.xml?provider_key=#{provider_key}&page=#{page}&per_page=100"
    page += 1
    response = RestClient.get url
    raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
    document = Nokogiri::XML(response.to_str)    done = document.xpath("applications/@current_page").text == document.xpath("applications/@total_pages").text
    document.xpath("//application").each do |item|
      app = Hash.new
      app["created_at"] = DateTime.parse(item.xpath("created_at").text)
      app["plan_name"] = item.xpath("plan/name").text
      app["service_id"] = item.xpath("plan/service_id").text
      app["account_id"] = item.xpath("user_account_id").text
      app["id"] = item.xpath("id").text
      res << app
    end
  end
  return res
end
  • Filter the applications that do not meet the criteria, i.e. the plan must be "evaluation" and they have to be newer than 10 days. 
def filter_applications(domain, provider_key, plan_name, num_of_days)
  res = api_call_applications_list(domain, provider_key)
  res.each do |item|
    res.delete(item) if item["plan_name"] != plan_name
    res.delete(item) if item["created_at"] > (DateTime.now - num_of_days)
  end
  return res
end
  • Then for each application that meets the criteria, get its usage, meaning how many hits the application has had in the last 10 days. 
def api_call_application_usage(domain, provider_key, application_id, metric, from, to, granularity)
  url = "https://#{domain}/stats/applications/#{application_id}/usage.xml?provider_key=#{provider_key}&metric_name=#{metric}&since=#{from}&until=#{to}&granularity=#{granularity}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  return document.xpath("//usage/data/values").text.split(",")
end
  • Finally, you must cross-reference the applications to the accounts because the information of the developers is stored in the account object. 
def api_call_account_read(domain, provider_key, account_id)
  url = "https://#{domain}/admin/api/accounts/#{account_id}.xml?provider_key=#{provider_key}"
  response = RestClient.get url
  raise Exception.new("Wrong response code (#{response.code}) in request #{url}") if response.code!=200
  document = Nokogiri::XML(response.to_str)
  account = Hash.new
  account["email"] = document.xpath("//users/user/email").text
  account["name"] = document.xpath("//users/user/first_name").text + " " + document.xpath("//users/user/last_name").text
  return account
end

Now you need to put everything together. Get the full script as a gist and you are done. You have a script that gets the information that was not yet available on 3scale built-in analytics.

32.5. Conclusions

3scale is very DIY-friendly. We believe that anyone should be able to extend the current capabilities of any service that they have contracted. As much as we would like to think that we can cover all the bases, we will always be a bit behind in certain aspects, whether because it is a super special need or simply because we are busy building other features. Whatever the case, we want to avoid blocking you when something is not possible right away. We give you full access to your data and a complete API toolset to work with it.

If you have scripts that you want to share with us and the rest of 3scale users, please send them. We would be happy to collect and publish them.