"Development" Posts


Detect and Fix HAProxy+Apache+Passenger Queue Backlogs

By Derek Bullet_white Posted in Development, HowTo Bullet_white Comments Comments

To inspire hard work, some young men hang a poster on their wall that includes: (1) an exotic sports car (2) a scantly clad lady and (3) a beach house. My inspirational poster would be much less attractive: a friendly butler who offers time-honored wisdom (with an accent because people with accents are smarter) and absolutely loves running errands for me.

I don’t like running errands because I don’t like waiting in lines. My nightmare: having to pickup groceries during a busy weekend afternoon. There are 3 queues at the grocery store that can cause a delay:

  • Finding a parking spot
  • Getting a shopping cart
  • Checking out

Modern web apps face the same queuing issues serving web requests under heavy traffic. For example, a web request served by Scout passes through several queues:

web_queues

That’s Apache (for SSL processing) to HAProxy on the load balancer, then Apache to Passenger to the Rails app on a web server.

A request can get stuck in any of those five spots. The worst part about queues? Time in queue is easy to miss. Most of the time, people look at the application log when they suspect a slowdown. However, a slowdown in any of the four earlier queues won’t show up in your application log. Just looking at your application and database activity for slowdowns is like recording the time it takes to get your groceries from the time you grab the first item on the shelf till you start waiting to checkout: you’re leaving out the time it takes to find a parking spot, get a cart, and checkout.

Now, before you start worrying about queues, take a deep breath. First, each of these systems are super reliable. For the most part, they just work. Second, it’s much more likely your application logic is the cause of a performance issue than a queuing problem. Look there first.

Third (and most importantly), each of these systems handles queues in remarkably similar ways. Understanding some basic queuing concepts will go a long way. Let’s take a look at some basics and then specific examples for Apache, HAProxy, and Passenger.

Read More →

 

Tina Fey: go with the bran muffin on the white plate

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

First, Tina Fey needs to write more. Both of her New Yorker articles have been terrific. Second, her latest New Yorker piece had a great nugget on how producing is about discouraging creativity:

You may have an occasion where the script calls for a bran muffin on a white plate, and people from the props department show up with a bran cake in the shape of Santa Claus sitting on a silver platter that says “Welcome to Denmark” on it. “We just thought it would be funny”, they say. And you have to find a polite way to explain that the character is Jewish, so her eating Santa’s face might have negative connotations, and the silver tray, while beautiful, is creating a weird glare on camera, and maybe let’s just go with the brain muffin on the white plate.

It’s easy to get absorbed in code and build something that’s technically awesome yet far more than you need. In other words, that Redis-backed, OpenID-compatible, audible capatcha, two-factor identification login system idea of mine is creative, but wouldn’t a session cookie and that old-school relational database my grandpa talked about work for now?

Read More →

 

When you are not the smartest programmer in the room

By Derek Bullet_white Posted in Development, Business Bullet_white Comments Comments

Most of the freshman engineering students at my college took two computer science courses together. These were the hardest classes I would take during college (my colleague was a TA for these courses so he deserves partial blame).

The experience for Sam Epstein, a friend of mine, was decidedly different. A project that might take me a week to finish (poorly) would take Sam a couple of hours. This was a seminal moment in my life: while I knew there were smarter people than myself, I thought I could out-work my way to the top. In programming this wasn’t the case. I’d never be Sam Epstein. Not only would I not be Sam, I was pretty average compared to my peers.

Read More →

 

Understanding Disk I/O - when should you be worried?

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Our co-author today is Christian Paredes, Senior System Administrator at Blue Box Group, a Ruby on Rails-focused web host that specializes in providing the operations expertise required to keep powerful apps running at peak performance. Christian keeps Blue Box Group’s internal infrastructure in top-shape and provides tier 3 customer support. He also volunteers for LOPSA, a guild for system administrators. We’re pleased to have him share some of his expertise on disk I/O.

If you’re old enough to remember floppy drives, you’ve heard the symptoms of a disk I/O bottleneck. For example, while Oregon Trail loaded the next scene, you’d hear the drive grinding away, reading data from the disk. The CPU would sit idle during this time, twiddling its fingers waiting for data. If that floppy drive was faster, you’d be running the Columbia River rapids by now.

It’s more difficult to detect an I/O bottleneck if the disk isn’t on your desktop. I’ll look at four important disk I/O questions for web apps:

  • Do you have an I/O bottleneck?
  • What impacts I/O performance?
  • What’s the best path to fixing an I/O bottleneck?
  • How do you monitor disk I/O?

A banana slug vs. an F-18 Hornet

Disk I/O encompasses the input/output operations on a physical disk. If you’re reading data from a file on a disk, the processor needs to wait for the file to be read (the same goes for writing).

The killer when working with a disk? Access time. This is the time required for a computer to process a data request from the processor and then retrieve the required data from the storage device. Since hard disks are mechanical, you need to wait for the disk to rotate to the required disk sector.

Disk latency is around 13ms, but it depends on the quality and rotational speed of the hard drive. RAM latency is around 83 nanoseconds. How big is the difference? If RAM was an F-18 Hornet with a max speed of 1,190 mph (more than 1.5x the speed of sound), disk access speed is a banana slug with a top speed of 0.007 mph.

This is why caching data in memory is so important for performance – the difference in latency between RAM and a hard drive is enormous*.

Do you have an I/O bottleneck?

Read More →

 

How much slower is Disk vs. RAM latency?

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

It takes longer to access data stored on your hard disk vs. RAM. But how big is the difference? It’s really big.

How can you tell if you your web application is being impacted by slow disk access? What impacts I/O performance? What’s the best path to fixing the bottleneck? How do you monitor it? On Thursday, Christian Paredes of Blue Box Group joins us to talk about disk i/o.

UPDATED: Christian’s article has been published:

Understanding Disk I/O – when should you be worried?


Subscribe to our RSS feed or follow us on Twitter for more.

 

Sleep Better with a Proper Staging Environment

By Andre Bullet_white Posted in Business, Development Bullet_white Comments Comments

Nothing helps you sleep better at night like a staging environment that’s faithful to your production setup. That means your staging environment has the same Linux distro, same version of Ruby and gems, the same Apache and Passenger configuration, etc.

VPS not cloud

We’ve found that an inexpensive “always-on” VPS instance is better as a staging environment than a cloud instance we have to spin up and down. Why? Spinning up a cloud instance takes time. We’re more likely to actually use our staging environment if it’s as low-friction as possible to do so.

A staging environment isn’t free—you’ll spend money on the VPS, and you’ll spend time configuring and maintaining it. However, the peace of mind you’ll get is a great return on investment.

Setting up your staging environment

If setting up your staging environment is difficult, you have something to work on: a repeatable process for configuring production-like boxes. Remember, your staging environment should mimic your production environment as closely as possible. If you have a scripted process for setting up production boxes, then setting up your staging environment will be trivial.

If you’re like many organizations, however, there is no authoritative definition for production. Instead, it has evolved over time with manual tweaks and optimizations. In that case, the staging environment is a perfect opportunity to pull together a repeatable script. It doesn’t have to be automated (ours is not)—but it does need to be written down.

Staging deployments with Capistrano

We Rubyists are lucky—there are tools for just about everything. We use capistrano multistage for staging deployments. It’s straightforward to set up, and makes staging deployments completely frictionless.

You should end up with a “staging” file In your config/deploy directory, but not in your config/environments directory. You’ll use the your production environment for staging.

The unsolved staging problem: production-like load

The harder part is simulating production-like traffic on your staging server. In a perfect world, you would have holodeck for deployments. We don’t have a solution for this yet—ideas are welcome!

Previously in Developer Happiness

This is Part 4 in our Developer Happiness series. See previous articles:

Read More →

 

Older posts: 1 ... 3 4 5 6 7 8