How we handle background jobs

By Derek Bullet_white Posted in HowTo Bullet_white Comments Comments

We run background jobs on our Scout servers. Lots of them.

As we’ve grown, they’ve used dramatically more resources. We needed a way to simmer them down. Most of these jobs load the entire Rails environment – that’s a hefty overhead.

We’ve modified how we run background jobs and we’re seeing great results. If you’re interested in what we did, checkout Charles’ post on our Highgroove blog.

And finally, here’s the obligatory semi-promotional graph, generated by Scout, of how the load dramatically decreased after implementation:

Scout ~ Edit Graph


Scout Client 2.0 is on the Horizon

By Derek Bullet_white Posted in Features Bullet_white Comments 1 comment

We’ve been working on two major features that we’re sure will be a hit, and that we can’t wait to talk about:

1) Easier Plugin Creation—there’s a reason we love Ruby, and not just because it’s such a powerful language for talking to external system processes and manipulating strings (as pointed out by JEG2 in our RubyInside debut). Ruby makes writing powerful code down-right simple through a very natural language. And with our new plugin code helpers and syntax, you can write a plugin like this:

my_output = `run some wild script here`
remember(:last_time_output => my_output)
report(:data => my_output)
if my_output ~= /Error/
  alert("An Error Occurred", "Details: #{my_output}")

And that’s not all. We’ve built a clearer syntax for how you invoke Scout, and how you interact with options and how you can store your own settings and report your own data.

2) Automatic Client Provisioning—if you have any servers in the cloud, or just more than one server, you already know how easy it is to install Scout (one liner to install and another to setup), but it just got easier than that. You can now automatically provision clients using the scout client itself. I know, crazy-talk, but watch. On bootup, of our Amazon EC2 instance, we’ll just run:

scout clone [client-key] `hostname` >> /etc/crontab

And we’ve just setup a new client on scout, with all our pre-configured plugins, cloned from another client, and the name of our hostname, setup to run on our interval.

If you’re interested in helping us beta test the new 2.0 version of our client, you can install it from our beta-gems server:

sudo gem install scout --source

Let us know what you think. Be sure to check the README to find out more details on what’s new.


How To Customize a Scout Plugin

By Derek Bullet_white Posted in HowTo Bullet_white Comments Comments

Since Scout Plugins are open source, it’s easy to modify them to fit your needs.

Here’s a quick way to create a custom version of a Scout plugin.

1. Grab the current plugin and options

The code:

curl \ 
> mongrel_cluster_monitor.rb

The options:

curl \ 
> mongrel_cluster_monitor.yml

2. Edit the plugin code and test

scout -p /path/to/customized/mongrel_cluster_monitor.rb \
-o '{"mongrel_cluster_configuration_dir"=>"/etc/mongrel_cluster"}' \
-v -l debug

That says, use the -p option (or --plugin) and the -o (or --option) to test locally, passing these options, be verbose (--verbose) and set the log level to debug (--log-level debug).

Note that you could also add your own “option” there, as the path to one of the configuration files…. however you want to do it.

3. Host the Plugin

Scout ~ Bristol

Then, all you have to do is make that plugin and its options file (of the same name) URL accessible (put it in your rails public dir, perhaps, or on another server) for the scout server to be able to grab it when you install. When you install, choose, Manual—and type in the URL to your new customized version ( for example).

More on creating plugins


Plugin Code is now cached

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

One of the things people like most about Scout: the plugin code your clients execute is downloaded via a plain-vanilla HTTP or HTTPS connection like this one. This means you can simply update the plugin code once, and any clients that run the plugin grabs the new code. You don’t have to manually install scripts on each monitored server.

But what happens if the web server hosting the plugin code isn’t available? What if you are running a plugin someone else developed and they change the plugin code?

We’ve got you covered. Scout now caches the plugin code when you add a plugin to a Scout Client. By default, Scout locks to the version of the code available when the plugin was installed. However, you can override this and have Scout automatically grab the latest version. You can also perform a one-time update.

We check for code updates every 5 minutes, so if you are debugging a new plugin you developed, click the “Update Code” button when you push out a change to instantly refresh the code.

Read on for screenshots.

Read More →


Recording Historical Data in Scout

By Derek Bullet_white Posted in HowTo Bullet_white Comments Comments

By default, when you create a report in Scout, we assume that the data corresponds to the time the report was submitted.

Sometimes this isn’t the case – for example, you can retrieve the number of subscribers to your RSS feeds in Feedburner, but not from the current day.

You can override the time Scout records the data in a Plugin with the special :scout_time report key.

Here’s an example:

# typical report data 
{:circulation => 1000}

# correspond the data w/ February 24, 2008
{:circulation => 1000, :scout_time => Time.parse("2/24/08")}

When displaying the data on a graph, the data will be shown on 2/24 and not the current date:

Scout ~ Derek Laptop

Learn more about creating Scout Plugins


Taking the guesswork out of scaling

By Derek Bullet_white Posted in Examples Bullet_white Comments Comments

Determining a web application’s hardware resources isn’t easy (or cheap). Frankly, it’s often just guesswork. Even when you build benchmarking scripts, they can miss key behaviors and ignore important metrics.

Scaling becomes a lot less stressful when you can quickly compare a history of your application data with server performance.

For example, we did this to get a better understanding of how our Scout server performed during our invitation process. The graph below was generated through Scout and shows the relationship between user accounts and the server load. As we expected, the overall load on the server increased as the number of accounts increased. Scout shows us how this data is correlated – it gives us an idea of how many accounts our current hardware can support.

Scout Accounts vs. Server Load
accounts vs load

It’s trivial process to regularly feed Scout your application data (user signups, orders, revenue, etc):

  1. Start with this Rails App Plugin Sample (this assumes a Ruby on Rails application, but you can do this with any framework/language)
  2. Grab your application data – just use ActiveRecord!
  3. Put the plugin on your server (can protect behind basic auth)
  4. Add the plugin

Older posts: 1 ... 58 59 60 61 62 ... 67