April 24
Derek
Posted in Examples | 
Comments
Determining a web application’s hardware resources isn’t easy (or cheap). Frankly, it’s often just guesswork. Even when you build benchmarking scripts, they can miss key behaviors and ignore important metrics.
Scaling becomes a lot less stressful when you can quickly compare a history of your application data with server performance.
For example, we did this to get a better understanding of how our Scout server performed during our invitation process. The graph below was generated through Scout and shows the relationship between user accounts and the server load. As we expected, the overall load on the server increased as the number of accounts increased. Scout shows us how this data is correlated – it gives us an idea of how many accounts our current hardware can support.
Scout Accounts vs. Server Load
It’s trivial process to regularly feed Scout your application data (user signups, orders, revenue, etc):
- Start with this Rails App Plugin Sample (this assumes a Ruby on Rails application, but you can do this with any framework/language)
- Grab your application data – just use ActiveRecord!
- Put the plugin on your server (can protect behind basic auth)
- Add the plugin
April 23
Charles
Posted in Plugins | Comments
Sometimes, the simplest ideas make for the best plugins.
Let’s say you have some process that always needs to be running (maybe that’s a ferret server or a backgrounDRb server) or maybe it’s just Apache.
Install the Keep Process Running Plugin and Scout will help make sure that process is always running.
The plugin simply scans the active running processes, and checks to see if that process is running. If it can’t find it in the active process list, it will attempt to run the restart action. You can specify anything, so if you need to perform some special startup task (/usr/bin/apachectl start) it can do that. If you do not specify a restart action, it simply re-runs the command it is looking for. This actually works well in most cases.
If you have any feedback or suggestions on this plugin feel free to comment on the plugin itself.
April 22
Derek
Posted in Plugins | Comments
Ever wanted to track how many people are logged into your server? Mark Hasse built an Active Logins plugin that does just that, returning the total number of people logged in via console, ssh, telnet, etc.
Mark has built several other plugins as well:
It’s easy to create a Scout a plugin and we’re here to help. Some useful plugin resources:
April 21
Derek
Posted in Features
We’ve received great feedback on our existing Scout plugins, but we wanted a better way to organize it. Friday we called on our inner agileness and built plugin ratings and comments.
If you have a Scout account, you can rate and comment on plugins in our directory. If you don’t have a Scout account, it’s completely free to signup – no obligations, late night phone calls, etc.
We’ll use your feedback to tune our plugins. If you have an idea for a new plugin, just add it to this thread in our forums.
April 17
Derek
Posted in Plugins | Comments
Swap memory is like pushing your car downhill when it’s all out of gas. Sure you’ll keep going, but just not as fast. Oh, and use too much, and you could be in trouble.
In our experience, when a significant amount of swap memory is used, our web application performance slows down dramatically. Swap memory is just a lot slower than physical memory.
Eric Lindvall has created a nice plugin that profiles a system’s memory usage, returning the following information:
- Swap Total
- Swap Used
- % Swap Used
- Memory Total
- Memory Used
- % Memory Used
View the Memory Profiler Plugin
April 16
Derek
Posted in Plugins | Comments
An oft-forgotten maintenance issue on Ruby on Rails applications is forgetting to expire old sessions. Lots of sessions in your database can dramatically slow performance.
You can use a cron job to expire old sessions, but it’s not always foolproof. We’ve seen timeouts when attempting to delete lots of session records and it’s not out of the range of possibilities to mistakenly disable multiple cron jobs.
Tim Morgan provided the inspiration for our latest plugin, Rails Session Monitoring. The plugin is dirt simple, reporting back the total number of sessions and the number of old sessions (you can define what constitutes old – the default is 14 days).
Here’s what I saw when I ran it on of our apps:
This is also the type of plugin that you don’t really need to run frequently – once-per-day is probably fine.
April 15
Derek
3 comments
The floor is waxed, the windows are washed, and the paint is fresh.
Scout, our server monitoring and reporting application, is at your service.
As a web developer, there are few things I value more than a solid block of focused development time. Because Scout has largely removed the lingering question “Is everything working?”, I have more of this time than ever before.
We believe Scout represents a shift in how web resources are monitored, primarily in 3 ways:
1. Monitoring has to be Open-Source
No single organization can keep pace with the speed of technical advancements. We believe the best monitoring practices are abstracted from real-world experience, not from a single organization attempting to cover the monitoring field. It’s our responsibility to make it as easy as possible to build, share, and present data from monitoring scripts built by the community. We all win when web services become more reliable.
2. Monitoring isn’t just for Sys Admins
We’re asking more of people than ever before – you’re now a product manager/front-end developer/AJAX wiz/database administrator/sys admin. The service that collects the most data doesn’t win – it’s the one that makes it understandable and actionable to a variety of people quickly that does.
3. If you’re just looking at the server, you’re missing 1/2 the story
When you’re just looking at data collected from your server, you’re looking at symptoms. You’re forced to take educated guesses, and wrong decisions can be extremely costly and time consuming when determining your resource needs.
It’s far easier to make resource decisions when you can see trends between server data and outside information like web application data (number of users, orders, etc) and analytics data (unique vistors, page views, etc). This data needs to be collected along with internal data to give a real-time view. Before Scout, it was time-consuming to pull all of this data together.
So, go ahead and get started with Scout!
February 14
Charles
2 comments
Last night, I demoed Scout to a room-full of Rubyists at the Atlanta Ruby User Group Meeting.
I would love to share all the wonderful feedback, but instead, I’ll share some of the excellent questions (and more elaborate answers) that were asked of Scout:
What are the security pitfalls, i.e. can someone simply write a ‘rm -rf’ plugin?
To answer that, let’s look at the architecture of Scout first:
- You install the tiny Scout client (which is a Ruby gem) on your server.
- The client connects over https (always) through a 256-bit secure, encrypted connection (the same encryption your bank uses).
- Scout never logs in to any of your servers.
- All communication is initiated by the client.
- The client downloads a pre-loaded plugin plan, consisting only of plugins of your choosing, so it cannot run plugins you didn’t explicitly authorize.
- The server also uses that same secure encryption for all communication. Individual accounts are protected.
- Client keys (uniquely generated) can be revoked at any time, disabling the client.
The security measures needed for Scout are the same as for any other software. In fact, in some ways, it’s easier to be more secure – the plugins are relatively few lines of code and easy to review. For a more closed environent, you can create a copy of the plugin code and host it on one of your own servers (a plugin is plain text).
Is Scout open source?
The Scout client is completely open source. The gem is a normal Ruby gem, open for development, and distributed under the MIT and/or Ruby License (whichever you prefer). The Scout Plugins people write are also completely open, in fact, they are surrounded and fostered by a community that encourages branching, fixes, and general open-ness.
The Server, where you aggregate your data, do reporting, and in general, collect information about your account is not open-source. We maintain the server, and keep all your data safe and sound.
When does it launch?
We’re doing the plumbing now – account subscriptions, a new home page, privacy policies, backup procedures, etc. We’ve recognized that lots of people are anxious to get going and we’re working to get it ready for public use as fast as possible.
January 29
Derek
Posted in Examples | Comments
We’re using Scout, our monitoring and reporting application, to graph the performance of our Rails applications and servers.
I’ve uploaded a video that looks at how one of our applications, PlaceShout, impacts the server load and Mongrel memory usage. I also compare PlaceShout’s footprint to another server.
Watch the video:
Graphing in Scout 
(1 min 47 sec)
Past Videos on Scout:
Installing the Scout Client (1 min 39 sec)
Installing the Rails Requests Plugin (1 min 55 sec)
Signup for our launch email list
We’ve started emailing invites to Scout. Signup on our homepage, and we’ll give you access to Scout before the public launch.
January 14
Derek
Posted in Examples | Comments
Tracking the results of your blatant self-promotion campaign can be a time-consuming effort. You might be using Google Analytics for web traffic and FeedBurner for blog subscribers. You’re probably checking link referrals. You’re querying the database for usage statistics (user signups, logins, etc.), etc.
Scout is an honest friend that gives it to you straight. Our friendly retriever will track, mash, and graph all of this data in real-time.
For example, below is a graph generated by Scout. It shows the FeedBurner circulation of this blog (in red) vs. unique visits from Google Analytics on our Highgroove homepage (in blue):
It doesn’t look like there’s a huge correlation there. What about unique visitors on PlaceShout (in red) vs. unique visitors on our Highgroove site (in blue)? Data via Google Analytics:
There’s a correlation there. Traffic to PlaceShout appears to drives traffic to Highgroove.
Currently, 3 Scout Plugins exist for grabbing external data:
Seeing this data is extremely useful for answering questions that take quite a bit of work to find out manually (and can’t be updated in real-time):
- How many of our unique visitors create a shoutout on PlaceShout?
- As the number of sites linking to us increases, how does this impact traffic on our site?
- How is traffic impacted when we publish our email newsletter?
The great thing about these reports is they don’t require any updates – Scout continually grabs new data and updates the graph.
Signup for our launch email list
We’re launching Scout this winter – click here to signup for our launch notification. We’ll email that list before the public launch.
Get Immediate Access to Scout
I’d like to create a plugin to report back the number of people linking to a url on del.icio.us, but haven’t had time yet. Want to create this plugin? Shoot me an email at (derek at highgroove dot com) and I’ll give you immediate access.
Past posts on Scout:
