Presenting the Scout Agent at Ruby Kaigi - Tokyo, Japan July 19

By Derek Bullet_white Posted in Updates Bullet_white Comments Comments

Scout takes a trek to Ruby’s birthplace – Japan – as James Gray presents How Lazy Americans Monitor Servers at the sold out Ruby Kaigi.

James’ July 19th talk focuses on the architecture of the Scout agent, the Ruby gem that is installed on a server you wish to monitor using Scout.

James will dig into the technical details of the agent’s division of labor approach for preventing memory leaks and crashes.

 

Q&A with the Scout Agent - An overview

By Derek Bullet_white Posted in Updates Bullet_white Comments 4 comments

Our recent update to Scout featured a revised UI, more functionality, and a new Scout Agent. While it’s easy to see the changes in the UI, a lot of the work conducted by the agent happens beneath the surface.

The Scout Agent, which is installed on a server you wish to monitor, was kind enough to sit down and walk me through its DNA (note that the ability to answer human questions is currently not available in the most recent release).

First, tell me a bit about what you’re made of.

I’m just a plain-old Ruby gem that you can install on any Linux-based server (sudo gem install scout_agent).

So, you’re a daemon right? Aren’t long-running Ruby tasks known to leak memory?

Yes, I’m a daemon. And yes, Ruby, like many programming languages, can leak memory when run for a long period of time.

My strategy for preventing memory leaks is simple: I do real work, like running plugins, in a separate short-lived process. I fork(), do whatever, and exit() so the OS can clean up any mess.

What’s your strategy to prevent the agent from crashing? Obviously, it’s important that monitoring software keeps running.

My work is divided into 2 main processes and several short-lived processes:
  • Lifeline – A single process that watches over all other agent processes. If a process fails to check-in with the lifeline regularly, I force it to stop and replace it with a healthy process.
  • Master – This is the event loop of the agent and is the main process monitored by the lifeline. It just sleeps and runs plugins in a never-ending cycle.
  • Missions – These processes execute the plugin code. These are small processes that exist only when plugins are running.

The reason for this division of labor? The real work is executed by the mission processes, which are short-lived. By offloading the work to such processes, the potential for degrading performance and a plugin’s execution raising an exception and killing me off is greatly reduced.

It’s easier to write 200 lines of bug-free code than 3000. The 200 LOC (my lifeline) keeps the rest alive.

Read More →

 

Scout Agent Updated - and do your Net::HTTP calls ever hang?

By Derek Bullet_white Comments Comments

UPDATED 6/30 – The fix for the old scout client (run via cron) is now available in version 2.0.7 (sudo gem install scout).

In rare (and difficult to reproduce) cases we’ve seen the Scout Agent not observe a Timeout during a checkin error with the Scout server. Scout uses Ruby’s RestClient gem to connect to the Scout Server and it uses the standard Net::HTTP library to manage the connection. Some versions of the Net::HTTP library can run into a bug in IO.select() on some platforms. This causes the request to hang forever in some rare cases.

Our fix? We added a redundant Timeout for the request, in addition to Net::HTTP’s own Timeout. You have to be careful how you nest those calls though, since they will throw the same Exception by default. We followed Eric Hodel’s advice to get our implementation right.

If you’re using Net::HTTP and notice the same issue, try adding a redundant Timeout with a custom halting Exception (our committed fix for this is on github).

This fix is included in version 3.2.6 of the Scout Agent. We’re planning on backporting the fix to the old client late next week (available now). Follow our Twitter feed to stay updated with the latest releases.

 

On Sinatra

By Derek Bullet_white Comments Comments

Sinatra, a Ruby DSL for quickly creating web applications with minimal effort, forms a key part of the Scout infrastructure.

James Gray talks about how we use Sinatra at Scout via RubyLearning.org – 20+ Rubyists are using Sinatra – Do you?

For more on how Scout works:

 

Rails Machine selects Scout for Performance Monitoring

By Derek Bullet_white Comments Comments

Rails Machine, one of the first specialized Rails hosting providers, has selected Scout for performance monitoring.

Why Scout? “Scout’s versatility was a key benefit: “the open-source plugins are the killer app of Monitoring,” says Rails Machine CEO Bradly Taylor. “We have so much flexibilty—and we’ll be contributing some great plugins back to the community.”

“With Scout, we have the full picture – Rails, MySQL, CPU, Memory, IO, and Disk Usage – all in one dashboard. You can’t beat that,” said Jesse Newland, Senior Engineer at Rails Machine.

Rails Machine, which recently launched a Managed Hosting offering, finds Scout particularly useful for proactively managing customer accounts. “Scout’s trend detection have more than once caught an application’s jump in memory usage due to RMagick-based image uploads/resizes,” says Newland. “We’ve been able to jump in, restart Apache, and then proposing alternatives to customers. It’s much better than a server running out of a memory minutes later.”

Scout is included with Rails Machine’s new Managed Hosting service which does everything but write the code – from MySQL tuning to backups to performance monitoring. Self-managed customers can request a coupon for a free Basic Scout subscription or $14/off a larger plan. See the announcement on the Rails Machine blog for more details.

From the original five-minute Rails deployment gem, the Rails Machine gem, to Moonshine, an open source configuration management and deployment system, Rails Machine understands automating system administration. Combining Scout’s proactive approach to monitoring with Rails Machine’s encyclopedia-like knowledge-base for scaling Rails apps is a perfect combination for managed hosting.

 

A tour of Scout's Ruby on Rails Monitoring Plugin

By Derek Bullet_white Posted in Plugins Bullet_white Comments Comments

Curious how data moves from your Rails app to Scout? Wondering how Scout handles high-traffic Rails apps? Where does Scout add MySQL suggestions? How are alerts generated? I’ll guide you through the process below.

Logging each web request

At a basic level, Scout’s Rails benchmarking isn’t much different than what gets written to your Rails log. Instead of writing to a log file, the Scout Rails Instrumentation plugin sends a Hash to the Scout Agent like the one below:


{:num_requests=>2, :avg_request_time=>241.0, :actions=>
{"users/index"=>{:db_runtime_max=>156.152, 
:num_requests=>2, :render_runtime_avg=>8.59355926513672, :render_runtime_max=>8.73303413391113, 
:runtime_avg=>241.0, :other_runtime_avg=>77.4774407348633, 
:queries=>[[[156.152, 0]], [[153.706, 0]]], :runtime_max=>243.0, 
:other_runtime_max=>78.1149658660889, :db_runtime_avg=>154.929}}, 
:queries=>["SELECT * FROM \"users\""], 
:scout_time=>"Mon Jun 15 21:04:00 UTC 2009"}

These messages are sent to the Scout Agent every 30 seconds by the Scout#Reporter.report! method:


response = ScoutAgent::API.queue_for_mission(Scout.config[:plugin_id], report)

Read More →

 

Older posts: 1 ... 51 52 53 54 55 ... 65