November 24 Derek

"I expected it to …" feature requests

Posted in Development |  Comments

When I bike into work, I coast down a decent hill. With gentle curves and smooth pavement for 99% of the decent, it’s a refreshing way to start the day. My one complaint: there’s a 2×2 ft. section in the final turn with bad pavement.

This tiny imperfection breaks up the natural flow: I need to remind myself of it before I descend. It’s a lot like my favorite feature requests: those that begin with “I expected it to …”.

These requests often hint at tiny imperfections our users experience, but live with. When fixed, it makes an entire workflow seem natural.

November 22 Derek

One change that lowered our DB I/O by 80%

Posted in Development |  Comments

When a server reports data to Scout, a small bit of data is updated in a MySQL MyISAM table. Each of these updates doesn’t change much data:

I assumed that only the updated data was written to disk. I was wrong: Jesse Newland of Rails Machine pointed out that updating a row in a MyISAM table rewrites the entire row to disk (not just the updated columns).

These updates were expensive: we were updating small bits of data, but our table contained a large text column that accounted for 75% of the data stored in the table. Every update to a row in our plugins table would rewrite the large text column (even though the text column wasn’t changing).

Read more...

November 09 Derek

Where are the Rails infrastructure support firms?

Posted in Development, Business |  Comments

There are no formal documents to sign if you and your spouse decide to have children. You don’t have to sit through an accreditation class. There is no credit check. You don’t need a high school diploma. Procreation can even happen accidentally.

A baby is a lot like a Rails application: the problem is caring for it, not creating it.

Five years ago, the typical Rails stack was just a couple of pieces: Apache/Mongrel, Rails, and MySQL. While Rails is remarkably similar to its original form even today, the stack around it is dramatically more diverse. We’re deploying to automated infrastructures, using NoSQL databases, messaging systems, queuing systems, and more. With the increased complexity of web applications, I’m surprised we’re not seeing companies dedicated to 24/7 infrastructure support: it doesn’t matter where your app is hosted, they manage it.

Read more...

October 28 Derek

Here's to the unsung developer heros

Posted in Development |  Comments

We’re fascinated with complexity. Write an article about the enormously high throughput/disk space/concurrency your application pushes and people will read it. You’ll speak at conferences. People will follow you.

It’s fun talking about a complex system, but it can be tiring caring for one. I love Woz’s focus on efficiency when starting Apple:

It’s can you, Steve Wozniak, design the same computer – maybe it’s a Varian 620i – can you design it on paper with fewer chips than last month? Can you design it with 79 chips instead of 80 chips?

Here’s to the oft-ignored, seldom blogged-about, patch-commiting developer doing the dirty work, making the tools we already use more efficient a line at a time.

October 21 Derek

Why we don't schedule deployments during off-hours

Posted in Development |  Comments

Years back, before Scout, I used to schedule deployments outside the regular workday. I didn’t give it much thought – it was what my consulting clients were used to. However, we changed that practice with Scout.

Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment. With that in mind, we started scheduling deployments when it’s best for us.

Read more...

October 19 Derek

5 Pillars of Rails Cluster Monitoring

Posted in Development |  Comments

Photo by mollypop

My how you’ve grown! A couple of years ago your little Rails app was on a single server. Now you’re on a whole cluster – you’ve got web servers, database servers, HAProxy servers, and more. I’m so proud of you!

Monitoring your Rails cluster has gotten more difficult though, huh? When it comes to monitoring a cluster of servers, there are lots of options with overlapping features. Some products are open source, some aren’t. Some are hosted, some aren’t. At Scout, we’re very happy with our monitoring stack. We know a bit about monitoring, so what are we using under the hood to monitor our Rails/Sinatra cluster?

Read more...

October 06 Derek

Determining free memory on Linux

Posted in Development |  Comments

When checking the amount of free memory on a Linux server, it’s easy to think you’re running out of memory when you’re not.

For example, here’s the output of free -m on a server with 4GB of RAM:

With a quick glance, you might start sweating. Only 39 MB of free memory? Put down the antacid – you’ve got a lot more free memory than you think.

Read more...

September 07 Andre

RVM and cron in production

Posted in Development |  Comments

This info is for Scout users, but it’s also applicable to anyone who’s wondered how to use cron together with RVM. This assumes 1) you are using bash; 2) you running you cron tasks as a non-root user (say, “deploy”).

Read more...

September 01 Derek

A Holodeck for deployments?

Posted in Development |  Comments

Recently I watched the documentary miniseries When We Left Earth. A constant theme from early space flight was the huge number of unknowns. For example, before the first astronaut went into orbit, NASA had concerns humans would lose their vision and wouldn’t be able to swallow in zero gravity. Thankfully, space worked out.

In devops lingo, it was impossible to build an accurate staging environment.

At Scout, we’ve grown almost 600% since last August. With that growth, our deployments are more carefully executed. This is monitoring, so Scout needs to be reliable when we deploy changes.

Read more...