"HowTo" Posts
March 30
Derek
Posted in Development, HowTo | 
Comments
To inspire hard work, some young men hang a poster on their wall that includes: (1) an exotic sports car (2) a scantly clad lady and (3) a beach house. My inspirational poster would be much less attractive: a friendly butler who offers time-honored wisdom (with an accent because people with accents are smarter) and absolutely loves running errands for me.
I don’t like running errands because I don’t like waiting in lines. My nightmare: having to pickup groceries during a busy weekend afternoon. There are 3 queues at the grocery store that can cause a delay:
- Finding a parking spot
- Getting a shopping cart
- Checking out
Modern web apps face the same queuing issues serving web requests under heavy traffic. For example, a web request served by Scout passes through several queues:
That’s Apache (for SSL processing) to HAProxy on the load balancer, then Apache to Passenger to the Rails app on a web server.
A request can get stuck in any of those five spots. The worst part about queues? Time in queue is easy to miss. Most of the time, people look at the application log when they suspect a slowdown. However, a slowdown in any of the four earlier queues won’t show up in your application log. Just looking at your application and database activity for slowdowns is like recording the time it takes to get your groceries from the time you grab the first item on the shelf till you start waiting to checkout: you’re leaving out the time it takes to find a parking spot, get a cart, and checkout.
Now, before you start worrying about queues, take a deep breath. First, each of these systems are super reliable. For the most part, they just work. Second, it’s much more likely your application logic is the cause of a performance issue than a queuing problem. Look there first.
Third (and most importantly), each of these systems handles queues in remarkably similar ways. Understanding some basic queuing concepts will go a long way. Let’s take a look at some basics and then specific examples for Apache, HAProxy, and Passenger.
Read more... 
September 15
Derek
Posted in HowTo | Comments
A big part of providing good support is making it painless. At Scout, Andre and I handle all of the support requests. Once we’ve gathered the account information, it usually doesn’t take much time to help. The problem is quickly putting the account information together. We don’t want to use a dedicated support application – we usually handle just a couple of support requests per-day.
Why not view all of the account information right from Gmail, where the support request originates? We’re using Rapportive with a custom Raplet to make it happen. When we receive an email from a Scout customer, we see their Scout account info.
Read more...
April 08
Derek
Posted in HowTo | Comments
You maintain a growing Rails application and you’re seeing something peculiar. Sometimes when you use the application, it feels like the performance deteriorates significantly. However, all of your performance data shows no issues – requests in the Rails log file look speedy, CPU utilization is fine, database performance is solid, etc.
At first, you wave it off as a fluke. But then a customer reports the same issue. Now you’re concerned.
Read more...
February 01
Andre
Posted in HowTo | Comments
~ or ~
Sysadmin Eye for the Dev Guy
Developers! You can churn out a Rails or Sinatra app in no time. What about putting it out there in production? Occasionally forget the syntax for crontab or logrotate? Yeah, me too.
That's why I wrote up a few essential notes for a serviceable production environment.
This article covers Centos/Red Hat and Ubuntu, which is what I always end up on. My approach is to get some minimal configurations working quickly so I can see some results. From there, I can go back and refine the configurations.
Read more...
January 11
Andre
Posted in HowTo | Comments
How much memory is really available on your Linux box? Don't use /proc/meminfo to find out, use free -m instead. You may have more memory available than you thought.
Here's an example. /proc/meminfo says about 330MB is free:
~ $cat /proc/meminfo
MemFree: 340996 kB
..
free -m gives the following:
~ $free -m
total used free shared buffers cached
Mem: 1024 691 332 0 86 288
-/+ buffers/cache: 316 708
Swap: 2047 68 1979
You'll see the "buffers" and "cached" columns, which tell you about the amount of memory that the kernel is using for filesystem buffers, etc.
This sort of cached data will be freed by the kernel when an application tries to allocate more than what is "free", which is why the "-/+ buffers/cache" line is really the important line to pay attention to when you're checking out the free memory on a system.
So in this example, 708MB is how much memory is technically available for allocation should an application need it. The "buffers" (86MB) and "cached" (288MB) will be released by the kernal if they are needed.
All credit for this post goes to Eric Lindvall, who also wrote the memory profiler plugin.
Read more...
December 08
Derek
Posted in HowTo | Comments
Our co-author today is
Jesse Newland,
CTO of
RailsMachine. Jesse keeps RailsMachine customers up and running and troubleshoots their toughest problems. We’re pleased to have him share some of his expertise on
Phusion Passenger tuning.
Say your Rails application is running in production and it’s getting good traffic. Performance isn’t as good you would like. You’ve already determined that your database is not the bottleneck. What’s your next move?
There is a good chance that Passenger’s PassengerMaxPoolSize needs to be adjusted. PassengerMaxPoolSize specifies how many instances of your application Passenger will spin up to service incoming requests. If you were running Mongrels back in the day, PassengerMaxPoolSize is equivalent to the number of mongrels you configured for your app. The value of PassengerMaxPoolSize has a major bearing on your application’s performance.
Read more...
June 10
Andre
Posted in HowTo | 1 comment
How easy? Really easy. There are just two things you need to know to get started:
1. Starting the Agent in test mode
The agent sports a nifty autotest-like mode to help you quickly iterate new plugin code. To start:
scout_agent test my_new_plugin.rb
With that, the agent is watching my_new_plugin.rb, just waiting for you to hit save so it can come to life. Whenever you modify the target file, the Agent will immediately log to console the results of your plugin-in-progress.
2. The basic format of a plugin
Keeping this very simple—a plugin looks like this:
class ExampleReport < Scout::Plugin
def build_report
report(:minute=>Time.now.min)
end
end
Yeah, that’s a simple case all right. It just reports the minute every time it runs, so it will generate a single data series with a value between 0 and 59.
If you haven’t created a plugin before, I encourage you to try this very simple example. It will take you approximately two minutes, and you’ll see how easy the Agent’s test mode makes it to iterate.
Go a little deeper
There’s more: generating alerts, reading from option files, etc. When you’re ready, check out the developer documentation at http://scoutapp.com/info/plugin_anatomy
September 09
Derek
Posted in HowTo | Comments
Life as a cross-dresser must be unnerving. I could buy a wig, put some makeup on my adam’s apple. Buy a skirt. Does a sock-filled bra work for my bosom?
With a bit of mood lighting, I might actually look like a woman, but it would be difficult to relax. Knowing my manhood could be exposed in a multitude of ways – some stubble on my chin, hair on my legs, forgetting to make my voice less deep, etc. would leave me on-edge.
In some ways, I feel like I identified with cross-dressers before we solidified our Rails monitoring stack over the past year. I just felt vulnerable – that one wrong move could send our Rails stack tumbling.
We’ve standardized our setup, and I thought I’d share the 3 tools we’re using that make me feel much less like a cross-dresser these days.
The 3 pillars of our Rails monitoring stack
We break Rails monitoring into the 3 parts below (along with the tools we use):
Process Monitoring
|
System Performance
|
Exception Notifications
|
“The Diaper”
A safeguard for emergencies
|
“The Nerves”
Preventing future problems
|
“The Megaphone”
A loud voice when your app is breaking
|
|
Monit
|
Scout
|
Exception Notification plugin
|
|
Ensure Mongrels are running & restart leaking processes
|
Catch disturbing trends before they become problems – disk space usage,
server load, slow requests, etc.
|
Organizing and collecting application exceptions
|
So, to be clear, even though we built Scout, we use a combination of tools to make sure our Rails apps are running.
Read more...
July 29
Derek
Posted in HowTo | 11 comments
In Blink, Malcom Gladwell’s book on split-second decisions, Gladwell tells the story of how the Emergency Department at Chicago’s Cook County Hospital changed their process for diagnosing chest pain.
Dr. Brendan Reilly instituted a simple test to determine whether a patient was suffering from a heart attack. It combined just 4 questions with the results of an ECG. This simple test was 70% better than the barrage of questions previously asked by hospital staff to identify patients that weren’t having a heart attack and was nearly 20% better at identifying patients that were having heart attacks.
More information on the patient’s symptoms often led to an incorrect diagnosis. It distracted doctors from the real issues.
I’ve seen this many times with developers trying to debug performance issues in Rails applications. They look at outliers instead of the obvious culprits. It’s part of the reason I’ve never felt a need for a deep, detailed Rails monitoring application (i.e. – benchmarks from the controller to the database level on every request).
The majority of the time, our performance problems have nothing to do with the Rails framework (and we’ve worked through a lot of issues since we started building Rails apps in 2005). Why benchmark the entire request cycle when the vast majority of issues are isolated at the database layer? After I’ve ruled out the database, I can see benchmarking a single request (there’s a great free tool below), but I simply don’t want the other, often irrelevant information clouding my mind.
The root symptom we want to avoid in our apps is slow requests. Our Scout plugin for analyzing slow Rails requests has been installed nearly 250 times, so we’re not alone there.
Read more...
