"HowTo" Posts
June 07
Derek
Posted in HowTo | 
Comments
Back in 2010, we suggested using /bin/bash -l -c to run scout via Cron when using RVM. However, this was a brute approach: /bin/bash -l -c tells bash to behave as a login, interactive process. However, as Daniel Szmulewicz elequently stated in the comments for the original blog post, "Cron jobs are by nature non-login, non-interactive processes".
Fast-forward to today: RVM usage is continuing in production, and to make things more complicated, Cron jobs often need to account for both RVM and Bundler. So, what's our preferred approach when running Ruby executables via Cron in an RVM, RVM+Bundler, or Bundler environment? A shell script.
Cron Shell Script: RVM + Bundler
Lets say we want to run a Ruby executable (scout [KEY]) via Cron with (1) Ruby 1.9.2 and (2) my Rails App's Gem bundle:
Make the shell script executable: chmod +x FILE.sh.
Add the Cron job:
* * * * * shell_script.sh
But that's a lot of typing...
It's tempting to use /bin/bash -l -c when you are busy/lazy. To get around this, the scout install [KEY] command will detect if you are using (1) RVM and/or (2) Bundler. If so, we generate the shell script for you and make it executable.
scout install BNrIneEBMwE8h6VlhO4Bw4WmOVSLmnygSFZEPCfi
=== Scout Installation Wizard ===
It looks like you've installed Scout under RVM and/or Bundler.
We've generated a shell script for you.
Run `crontab -e`, pasting the line below into your Crontab file:
* * * * * /Users/dlite/.scout/scout_cron.sh
How do we detect RVM and Bundler? We've encapsulated it into an Environment class:
May 31
Derek
Posted in Development, HowTo | Comments
Scout’s realtime charts have been a big hit. Once you start using them for major deploys or performance incidents, going back to ten terminal windows running “top” feels like the dark ages.
Realtime was a lot of fun to implement and it’s been rock-solid so far. A big reason it was so much fun? We were able to implement all of it in Ruby (outside the Javascript used to handle websockets in the browser) and didn’t need to deploy any infrastructure .
So, how did we go about it?
Read more... 
March 30
Derek
Posted in Development, HowTo | Comments
To inspire hard work, some young men hang a poster on their wall that includes: (1) an exotic sports car (2) a scantly clad lady and (3) a beach house. My inspirational poster would be much less attractive: a friendly butler who offers time-honored wisdom (with an accent because people with accents are smarter) and absolutely loves running errands for me.
I don’t like running errands because I don’t like waiting in lines. My nightmare: having to pickup groceries during a busy weekend afternoon. There are 3 queues at the grocery store that can cause a delay:
- Finding a parking spot
- Getting a shopping cart
- Checking out
Modern web apps face the same queuing issues serving web requests under heavy traffic. For example, a web request served by Scout passes through several queues:
That’s Apache (for SSL processing) to HAProxy on the load balancer, then Apache to Passenger to the Rails app on a web server.
A request can get stuck in any of those five spots. The worst part about queues? Time in queue is easy to miss. Most of the time, people look at the application log when they suspect a slowdown. However, a slowdown in any of the four earlier queues won’t show up in your application log. Just looking at your application and database activity for slowdowns is like recording the time it takes to get your groceries from the time you grab the first item on the shelf till you start waiting to checkout: you’re leaving out the time it takes to find a parking spot, get a cart, and checkout.
Now, before you start worrying about queues, take a deep breath. First, each of these systems are super reliable. For the most part, they just work. Second, it’s much more likely your application logic is the cause of a performance issue than a queuing problem. Look there first.
Third (and most importantly), each of these systems handles queues in remarkably similar ways. Understanding some basic queuing concepts will go a long way. Let’s take a look at some basics and then specific examples for Apache, HAProxy, and Passenger.
Read more...
September 15
Derek
Posted in HowTo | Comments
A big part of providing good support is making it painless. At Scout, Andre and I handle all of the support requests. Once we’ve gathered the account information, it usually doesn’t take much time to help. The problem is quickly putting the account information together. We don’t want to use a dedicated support application – we usually handle just a couple of support requests per-day.
Why not view all of the account information right from Gmail, where the support request originates? We’re using Rapportive with a custom Raplet to make it happen. When we receive an email from a Scout customer, we see their Scout account info.
Read more...
April 08
Derek
Posted in HowTo | Comments
You maintain a growing Rails application and you’re seeing something peculiar. Sometimes when you use the application, it feels like the performance deteriorates significantly. However, all of your performance data shows no issues – requests in the Rails log file look speedy, CPU utilization is fine, database performance is solid, etc.
At first, you wave it off as a fluke. But then a customer reports the same issue. Now you’re concerned.
Read more...
February 01
Andre
Posted in HowTo | Comments
~ or ~
Sysadmin Eye for the Dev Guy
Developers! You can churn out a Rails or Sinatra app in no time. What about putting it out there in production? Occasionally forget the syntax for crontab or logrotate? Yeah, me too.
That's why I wrote up a few essential notes for a serviceable production environment.
This article covers Centos/Red Hat and Ubuntu, which is what I always end up on. My approach is to get some minimal configurations working quickly so I can see some results. From there, I can go back and refine the configurations.
Read more...
January 11
Andre
Posted in HowTo | Comments
How much memory is really available on your Linux box? Don't use /proc/meminfo to find out, use free -m instead. You may have more memory available than you thought.
Here's an example. /proc/meminfo says about 330MB is free:
~ $cat /proc/meminfo
MemFree: 340996 kB
..
free -m gives the following:
~ $free -m
total used free shared buffers cached
Mem: 1024 691 332 0 86 288
-/+ buffers/cache: 316 708
Swap: 2047 68 1979
You'll see the "buffers" and "cached" columns, which tell you about the amount of memory that the kernel is using for filesystem buffers, etc.
This sort of cached data will be freed by the kernel when an application tries to allocate more than what is "free", which is why the "-/+ buffers/cache" line is really the important line to pay attention to when you're checking out the free memory on a system.
So in this example, 708MB is how much memory is technically available for allocation should an application need it. The "buffers" (86MB) and "cached" (288MB) will be released by the kernal if they are needed.
All credit for this post goes to Eric Lindvall, who also wrote the memory profiler plugin.
Read more...
December 08
Derek
Posted in HowTo | Comments
Our co-author today is
Jesse Newland,
CTO of
RailsMachine. Jesse keeps RailsMachine customers up and running and troubleshoots their toughest problems. We’re pleased to have him share some of his expertise on
Phusion Passenger tuning.
Say your Rails application is running in production and it’s getting good traffic. Performance isn’t as good you would like. You’ve already determined that your database is not the bottleneck. What’s your next move?
There is a good chance that Passenger’s PassengerMaxPoolSize needs to be adjusted. PassengerMaxPoolSize specifies how many instances of your application Passenger will spin up to service incoming requests. If you were running Mongrels back in the day, PassengerMaxPoolSize is equivalent to the number of mongrels you configured for your app. The value of PassengerMaxPoolSize has a major bearing on your application’s performance.
Read more...
June 10
Andre
Posted in HowTo | 1 comment
How easy? Really easy. There are just two things you need to know to get started:
1. Starting the Agent in test mode
The agent sports a nifty autotest-like mode to help you quickly iterate new plugin code. To start:
scout_agent test my_new_plugin.rb
With that, the agent is watching my_new_plugin.rb, just waiting for you to hit save so it can come to life. Whenever you modify the target file, the Agent will immediately log to console the results of your plugin-in-progress.
2. The basic format of a plugin
Keeping this very simple—a plugin looks like this:
class ExampleReport < Scout::Plugin
def build_report
report(:minute=>Time.now.min)
end
end
Yeah, that’s a simple case all right. It just reports the minute every time it runs, so it will generate a single data series with a value between 0 and 59.
If you haven’t created a plugin before, I encourage you to try this very simple example. It will take you approximately two minutes, and you’ll see how easy the Agent’s test mode makes it to iterate.
Go a little deeper
There’s more: generating alerts, reading from option files, etc. When you’re ready, check out the developer documentation at http://scoutapp.com/info/plugin_anatomy
September 09
Derek
Posted in HowTo | Comments
Life as a cross-dresser must be unnerving. I could buy a wig, put some makeup on my adam’s apple. Buy a skirt. Does a sock-filled bra work for my bosom?
With a bit of mood lighting, I might actually look like a woman, but it would be difficult to relax. Knowing my manhood could be exposed in a multitude of ways – some stubble on my chin, hair on my legs, forgetting to make my voice less deep, etc. would leave me on-edge.
In some ways, I feel like I identified with cross-dressers before we solidified our Rails monitoring stack over the past year. I just felt vulnerable – that one wrong move could send our Rails stack tumbling.
We’ve standardized our setup, and I thought I’d share the 3 tools we’re using that make me feel much less like a cross-dresser these days.
The 3 pillars of our Rails monitoring stack
We break Rails monitoring into the 3 parts below (along with the tools we use):
Process Monitoring
|
System Performance
|
Exception Notifications
|
“The Diaper”
A safeguard for emergencies
|
“The Nerves”
Preventing future problems
|
“The Megaphone”
A loud voice when your app is breaking
|
|
Monit
|
Scout
|
Exception Notification plugin
|
|
Ensure Mongrels are running & restart leaking processes
|
Catch disturbing trends before they become problems – disk space usage,
server load, slow requests, etc.
|
Organizing and collecting application exceptions
|
So, to be clear, even though we built Scout, we use a combination of tools to make sure our Rails apps are running.
Read more...
