This is the blog for Scout, simple server monitoring
powered by plugins. From request times to memory usage,
put your monitoring in one place.
Install in minutes.
Have you noticed Scout feels snappier lately? We made some major simplifications that sped things up a lot. Here’s the CPU load on one of our DB servers:
(and yes, we use Scout to monitor itself!)
Even better, the response time for users improved dramatically.
Scout’s longest actions before and after the speedup:
The simplification
What yielded such a dramatic speedup? Earlier this year, we implemented a very generic datastore and reporting system. It could handle all sorts of data, relationships within the data, etc.
Unfortunately, we never got to demonstrate all the benefits of this cool system. It wasn’t viable from either a maintenance or a performance standpoint.
So we rolled it back. And we got back a ton of performance, as you can see.
The lessons …
I will be writing up some business lessons we learned from this experience—stay tuned!
If you're using Amazon EC2, you may be familiar with CloudWatch,
Amazon's analytic system that provides metrics on CPU usage,
Network I/O, and Disk I/O of your instances. While CloudWatch
collects metrics, it doesn't provide a web interface for viewing
the metrics, graphs, trending, or alerting.
Enter the BETA version of our Scout CloudWatch Plugin. Like any
other Scout plugin, you can graph the resulting metrics, set
triggers, track trends, and get email alerts when the numbers go
out of bounds. It's a good example of how Scout can wrap around
data collection and provide the required monitoring bits.
Here is the plugin link: EC2 Cloudwatch plugin. The plugin is in BETA, so we appreciate your feedback.
What does it monitor?
The CloudWatch plugin captures the following ("measures", as EC2 calls them): NetworkIn, NetworkOut, DiskReadOps, DiskWriteOps DiskReadBytes, DiskWriteBytes, CPUUtilization.
Note, this plugin does not fetch EC2 Load Balancer Metrics, only EC2 instance metrics.
Single Instance, Autoscaling Group, etc.
The EC2 CloudWatch plugin can capture metrics from a single EC2 instance, or it can aggregate metrics across a couple of dimensions. It can aggregate metrics across a given instance type, across all instances launched from a specific image (AMI), or by a specified autoscaling group. That means you can, for example, graph the performance of your application server autoscaling group as a whole, or graph just your memcached instance.
Enabling Cloudwatch
To use this plugin, you have to enable CloudWatch for the instance(s) you want to collect metrics from. See Amazon's CloudWatch docs for details. Basically, it's just ec2-monitor-instances ##### from the command line, or passing a monitoring parameter to the ec2-run-instances. It's covered nicely in Amazon's docs.
Feedback, please
As mentioned, this is a "beta release" of the CloudWatch plugin. We'd love to get your feedback on it. For example, do you monitor by instance id or autoscaling group more frequently?
New to Scout?
If you're learning about Scout through this plugin, sign up for our free monitoring service to give this plugin a try. You can graph all kinds of metrics and measurements from all your servers. It works with cloud instances, VPS's, and dedicated hardware.
Our recent update to Scout featured a revised UI, more functionality, and a new Scout Agent. While it’s easy to see the changes in the UI, a lot of the work conducted by the agent happens beneath the surface.
The Scout Agent, which is installed on a server you wish to monitor, was kind enough to sit down and walk me through its DNA (note that the ability to answer human questions is currently not available in the most recent release).
First, tell me a bit about what you’re made of.
I’m just a plain-old Ruby gem that you can install on any Linux-based server (sudo gem install scout_agent).
So, you’re a daemon right? Aren’t long-running Ruby tasks known to leak memory?
Yes, I’m a daemon. And yes, Ruby, like many programming languages, can leak memory when run for a long period of time.
My strategy for preventing memory leaks is simple: I do real work, like running plugins, in a separate short-lived process. I fork(), do whatever, and exit() so the OS can clean up any mess.
What’s your strategy to prevent the agent from crashing? Obviously, it’s important that monitoring software keeps running.
My work is divided into 2 main processes and several short-lived processes:
Lifeline – A single process that watches over all other agent processes. If a process fails to check-in with the lifeline regularly, I force it to stop and replace it with a healthy process.
Master – This is the event loop of the agent and is the main process monitored by the lifeline. It just sleeps and runs plugins in a never-ending cycle.
Missions – These processes execute the plugin code. These are small processes that exist only when plugins are running.
The reason for this division of labor? The real work is executed by the mission processes, which are short-lived. By offloading the work to such processes, the potential for degrading performance and a plugin’s execution raising an exception and killing me off is greatly reduced.
It’s easier to write 200 lines of bug-free code than 3000. The 200 LOC (my lifeline) keeps the rest alive.
Scout is nearing 1/2 billon recorded metrics, but bragging about metrics is like bragging about the number of lines of code you’ve written. Metrics, like code, are a means to an end. It’s the quality, not the volume, that matters.
We took a big step in going beyond numbers today with the release of the Rails instrumentation plugin for Scout. It’s the first plugin to leverage Scout’s new features, and at just $7/server, we think it’s at a price that makes sense whether you have a small VPS or a cloud of servers.
A quick tour of Scout’s new features:
Deep Rails Instrumentation Plugin
Using the new Rails instrumentation plugin, you’ll get alerted of trends. Scout wants to help you correct and issue before it becomes a problem:
When things go wrong, Scout tries to lend a helping hand, giving optimization suggestions:
..and highlights the variables that changed the most:
Sometimes you just want an overview of your Rails application as well:
The Rails Instrumentation plugin is open source, like the rest of our plugins. View the source of the Rails plugin and its companion Scout plugin on GitHub.
Triggers
Leverage the same alerting functionality used by the Deep Rails instrumentation plugin in your own plugins with triggers. For example, let Scout notify you if user signups dramatically increase compared to the previous 7 days. Just pick a metric and set the thresholds!
The new Scout Agent
The old cron-based Scout client has been replaced by the Scout Agent, a daemon. The Scout Agent is an easier install and enables the more powerful and flexible metric-collecting goodness used by the Rails instrumentation plugin.
If French suites you better than English (or you just like imagining yourself coding in a Paris coffee shop), Raphaël Emourgeon has penned a review of Scout in French.
Raphaël sums up Scout nicely:
Après avoir installé un gem et ajouté une tâche cron sur le serveur à surveiller, tout se fait à distance via le site Internet de Scoutapp. Il suffit de quelques clics pour installer un plugin sur une machine et commencer la surveillance. Les plugins font toute l’originalité du système puisqu’un plugin est un script Ruby qui peut faire tous les traitements imaginables et qui utilise ensuite l’API (très simple) de Scoutapp pour renvoyer des résultats.
The number one requested plugin for Scout has been a Phusion Passenger monitor. I’m happy to report that is now available.
Using the new Passenger Monitor plugin, Scout can keep track of Passenger’s VM size, process count, and the amount of private memory it has squirreled away. Aside from just tracking these numbers for graphs, you can also elect to be emailed if any of these statistics crosses a line you indicate. As an added bonus, this plugin also tracks the same statistics for the Apache instance managing Passenger.
Passenger has quickly become a great new way to deploy Rails applications and we want to make sure Scout is keeping up with the curve.
We’ve released the 2nd version of our Scout client. What I love about this release is that it doesn’t make anything more complex or break compatibility with our first release. It just makes writing plugins easier.
For example – lets say you want to generate an alert. How’s this for simplicity?
class INeedASandwich < Scout::Plugin
def build_report
alert(:subject => "Get me a sandwich")
end
end
We’ve also made the process for managing memory (data stored on the client between runs of the plugin) easier to work with. Memory is a great tool for storing state.
For example:
class INeedASandwich < Scout::Plugin
def build_report
if memory(:am_i_hungry)
alert(:subject => "Get me a sandwich")
remember(:am_i_hungry => false)
else
remember(:am_i_hungry => true)
end
end
end
As always, updating to the latest release is as simple as:
Andre
