scout_api gem released: query your time series data

May 05 Bullet_white By Derek Bullet_white Posted in Features Bullet_white Comments Comments

When it comes to understanding the health of our infastructure, nothing is more valuable than time series data:

  • How much has our web traffic increased over the past year?
  • How has our average request time changed since adding caching?
  • What’s our peak database query volume this week?

...and I want to view it in different ways:

  • Summarized – A single value
  • Raw – An array of data points
  • Visually – A sparkline chart

I’m happy to announce version 1.0 of the scout_api Ruby gem. scout_api lets you query your time series data stored on Scout in a fun, intuitive way. While it took a while to release this, we think you’ll be happy with the end result.

If you’ve used Rails and ActiveRecord, the code will look very familiar. If you haven’t, I don’t think that will be problem. You’ll see how readable it is below.

Lets cut to the examples.

What’s the maximum memory usage on our web server?

Scout::Server.first(:name => "Web").metrics.maximum(:name => 'Memory Used')
=> {:label => 'Memory Used', :value => 800, :units => 'MB'}

By default, metrics#maximum returns data from the past hour. You can set your own time range:

Scout:Server.first(:name => "Web").metrics.maximum(:name =>'Memory Used',:start => Time.now.utc-3600*5, :end => Time.now.utc-3600)

In the example above, we’re retrieving metrics from five hours ago up to an hour ago. Depending on your subscription level, you can query for data from up to a year in the past.

What about aggregation?

You likely have multiple servers that perform a similar purpose: web servers, database slaves, etc. In some cases, you’re interested in the average value of the metrics across servers:

Scout::Group.first(:name => 'web servers').metrics.average(:name => 'CPU Usage/idle')

In other cases, you’re interested in the sum of metrics across servers. For example:

Scout::Group.first(:name => 'web servers').metrics.average(:name => 'request_rate', :aggregate => true)
In the example above, metrics#average is called with the option :aggregate=>true. This adds the throughput on each of the web servers together.

Sparklines and Arrays

You may have an internal dashboard and you’d like to see your total traffic over the past day:

Scout:Group.first(:name => 'web servers').metrics.average(:name =>'request_rate', :aggregate => true, :start => Time.now.utc-3600*24).to_sparkline

In the example above, #to_sparkline returns a URL to a Google Chart that generates a sparkline of traffic across your web servers.

If you want the raw data, just use #to_array instead of #to_sparkline. You’ll get a 2D array that looks like this:

[
 [Wed May 04 18:40:00 UTC 2011, 12.0],
 [Wed May 04 18:45:00 UTC 2011, 6.2],
 [Wed May 04 18:50:00 UTC 2011, 3.0],
 [Wed May 04 18:55:00 UTC 2011, 1.5]
 ...
]

The first element in each inner array is the time in UTC, the second is the value.

Getting Started

We’ve tried to make it easy to play with Scout right from your irb session – most methods work without any options, so feel free to experiment:

$ irb
>> require 'scout_api'
>> Scout::Account.new('youraccountname', 'your@awesome.com', 'sekret')
=> <#Scout::Account>
>> Scout::Server.first
=> <#Scout::Server name="DB Slave">
>> Scout::Server.metrics.first
=> <#Scout::Metric name="Memory Used">
>> Scout::Metric.first.average
=> {:name => "Memory Used", :value => 800, :units => "MB"}
>> Scout::Group.metrics.first.average.to_sparkline
=> "http://chart.apis.google.com/chart?..."

For more:

If you’re working on something interesting with scout_api, let us know! Send an email to support@scoutapp.com.

Some History

Original work on a Scout API was done by the fine folks at Rails Machine and published as the scout_scout gem. scout_api is a fork of this project on Github.

Comments

comments powered by Disqus