UPDATED 6/30 – The fix for the old scout client (run via cron) is now available in version 2.0.7 (sudo gem install scout).
In rare (and difficult to reproduce) cases we’ve seen the Scout Agent not observe a Timeout during a checkin error with the Scout server. Scout uses Ruby’s RestClient gem to connect to the Scout Server and it uses the standard Net::HTTP library to manage the connection. Some versions of the Net::HTTP library can run into a bug in IO.select() on some platforms. This causes the request to hang forever in some rare cases.
Our fix? We added a redundant Timeout for the request, in addition to Net::HTTP’s own Timeout. You have to be careful how you nest those calls though, since they will throw the same Exception by default. We followed Eric Hodel’s advice to get our implementation right.
If you’re using Net::HTTP and notice the same issue, try adding a redundant Timeout with a custom halting Exception (our committed fix for this is on github).
This fix is included in version 3.2.6 of the Scout Agent. We’re planning on backporting the fix to the old client late next week (available now). Follow our Twitter feed to stay updated with the latest releases.
Rails Machine, one of the first specialized
Rails hosting providers, has selected Scout for
performance monitoring.
Why Scout? “Scout’s versatility was a key benefit: “the open-source
plugins are the killer app of Monitoring,” says Rails Machine CEOBradly Taylor. “We have so much flexibilty—and we’ll be
contributing some great plugins back to the community.”
“With Scout, we have the full picture – Rails, MySQL, CPU, Memory, IO,
and Disk Usage – all in one dashboard. You can’t beat that,” said
Jesse Newland, Senior Engineer at Rails Machine.
Rails Machine, which recently launched a Managed Hosting offering,
finds Scout particularly useful for proactively managing customer
accounts. “Scout’s trend detection have more than once caught an
application’s jump in memory usage due to RMagick-based image
uploads/resizes,” says Newland. “We’ve been able to jump in, restart
Apache, and then proposing alternatives to customers. It’s much better
than a server running out of a memory minutes later.”
Scout is included with Rails Machine’s new Managed Hosting service
which does everything but write the code – from MySQL tuning to
backups to performance monitoring. Self-managed customers can request a coupon for a free Basic Scout subscription or $14/off a
larger plan. See the announcement on the Rails Machine blog for more details.
From the original five-minute Rails
deployment gem, the Rails Machine
gem, to
Moonshine, an
open source configuration management and deployment system, Rails
Machine understands automating system administration. Combining
Scout’s proactive approach to monitoring with Rails Machine’s
encyclopedia-like knowledge-base for scaling Rails apps is a perfect
combination for managed hosting.
Curious how data moves from your Rails app to Scout? Wondering how Scout handles high-traffic Rails apps? Where does Scout add MySQL suggestions? How are alerts generated? I’ll guide you through the process below.
Logging each web request
At a basic level, Scout’s Rails benchmarking isn’t much different than what gets written to your Rails log. Instead of writing to a log file, the Scout Rails Instrumentation plugin sends a Hash to the Scout Agent like the one below:
{:num_requests=>2, :avg_request_time=>241.0, :actions=>
{"users/index"=>{:db_runtime_max=>156.152,
:num_requests=>2, :render_runtime_avg=>8.59355926513672, :render_runtime_max=>8.73303413391113,
:runtime_avg=>241.0, :other_runtime_avg=>77.4774407348633,
:queries=>[[[156.152, 0]], [[153.706, 0]]], :runtime_max=>243.0,
:other_runtime_max=>78.1149658660889, :db_runtime_avg=>154.929}},
:queries=>["SELECT * FROM \"users\""],
:scout_time=>"Mon Jun 15 21:04:00 UTC 2009"}
These messages are sent to the Scout Agent every 30 seconds by the Scout#Reporter.report! method:
How easy? Really easy. There are just two things you need to know to get started:
1. Starting the Agent in test mode
The agent sports a nifty autotest-like mode to help you quickly iterate new plugin code. To start:
scout_agent test my_new_plugin.rb
With that, the agent is watching my_new_plugin.rb, just waiting for you to hit save so it can come to life. Whenever you modify the target file, the Agent will immediately log to console the results of your plugin-in-progress.
2. The basic format of a plugin
Keeping this very simple—a plugin looks like this:
class ExampleReport < Scout::Plugin
def build_report
report(:minute=>Time.now.min)
end
end
Yeah, that’s a simple case all right. It just reports the minute every time it runs, so it will generate a single data series with a value between 0 and 59.
If you haven’t created a plugin before, I encourage you to try this very simple example. It will take you approximately two minutes, and you’ll see how easy the Agent’s test mode makes it to iterate.
Go a little deeper
There’s more: generating alerts, reading from option files, etc. When you’re ready, check out the developer documentation at http://scoutapp.com/info/plugin_anatomy
Scout is nearing 1/2 billon recorded metrics, but bragging about metrics is like bragging about the number of lines of code you’ve written. Metrics, like code, are a means to an end. It’s the quality, not the volume, that matters.
We took a big step in going beyond numbers today with the release of the Rails instrumentation plugin for Scout. It’s the first plugin to leverage Scout’s new features, and at just $7/server, we think it’s at a price that makes sense whether you have a small VPS or a cloud of servers.
A quick tour of Scout’s new features:
Deep Rails Instrumentation Plugin
Using the new Rails instrumentation plugin, you’ll get alerted of trends. Scout wants to help you correct and issue before it becomes a problem:
When things go wrong, Scout tries to lend a helping hand, giving optimization suggestions:
..and highlights the variables that changed the most:
Sometimes you just want an overview of your Rails application as well:
The Rails Instrumentation plugin is open source, like the rest of our plugins. View the source of the Rails plugin and its companion Scout plugin on GitHub.
Triggers
Leverage the same alerting functionality used by the Deep Rails instrumentation plugin in your own plugins with triggers. For example, let Scout notify you if user signups dramatically increase compared to the previous 7 days. Just pick a metric and set the thresholds!
The new Scout Agent
The old cron-based Scout client has been replaced by the Scout Agent, a daemon. The Scout Agent is an easier install and enables the more powerful and flexible metric-collecting goodness used by the Rails instrumentation plugin.
It’s been a big year for Scout at conferences. We sponsored GoGaRuCo here in San Francisco last month, and we had a booth on the floor at RailsConf last week.
A few highlights from the conference:
We came to the conference armed with all kinds of “selling points” for Scout 2.0. When all is said and done, the number one point people love about Scout is the open-source plugin model. Thanks for everyone’s feedback and support!
It was really cool talking to people in all stages of involvement with Ruby and Rails. Including those who were building a Rails app for the first time. It’s good to get that perspective after working with Rails for the last 3+ years.
Charles Quinn and I had the opportunity to treat Hongli Lai & Ninh Bui to Sushi and say thanks for one of our favorite pieces of software.
One cool interaction—Josh Goebel of Pastie came by and asked us about the new Scout API. Then, he tracked us down the next day to show us the Scout plugin he had created for vmstat! Great to see folks using Scout’s extensibility like that.
I didn’t hear one mention of “what happens in Vegas”—Rails community +1; tired cliches -1.
I despise sales pitches, but you wouldn’t know it if you were the first person to visit our Scout exhibit booth. After introducing myself, I delved into a 5 minute whirl-wind tour of Scout 2.0, trying to show every possible feature that might help him monitor his Rails stack.
We exhibited Scout at our first conference, Golden Gate Ruby
Conference, run with Scandinavian precision and
held, not ironically, at the Swedish American
Hall. I started a little
awkward telling the Scout story, but things really smoothed out
quickly. 8 lessons from our first conference:
NEW for Scout 2.0! This plugin monitors a Ruby on Rails Application, providing deep analysis of application performance, MySQL optimization hints and breakdowns on individual actions.
