The New York Times ran a popular Op-Ed on
Happiness last year.
The Cliff Notes version?
The daily activities most associated with happiness are sex, socializing after work and having dinner with others. The daily activity most injurious to happiness is commuting.
Many software developers are fortunate to have flexible work hours that reduce (or eliminate) commute times. But what about that those times when you’re writing code and can’t have sex, socialize, or meet friends for dinner?
(1 of 5): Make it Trivial to Grab Production Data
You need to be able to troubleshoot data-specific issues in the comfort and serenity of your development environment.
So, you need an easy way to grab production data and bring it local. In the simplest case, this is just a rake task that 1) ssh’s into your production db and
mysqldumps your database; 2)
gzips it and
scps it locally; 3)
unzips and imports the dump into your development DB.
That approach doesn’t scale. As your product grows, the all-or-nothing dump won’t be enough. Your database will be too big to dump quickly, and it will take too long to transmit over the wire. This is not the way to developer happiness.
What you need is a way to export a slice of your application’s data. There is no way to generalize this; it’s totally application specific. For Scout, the most sensible partitioning approach is to download one account and all its associated data. The trick is to identify the associated data to be export as it cascades through your application. The basic recipe we use for this is:
- a local rake task invokes a remote
SelectiveExporter class, passing the account ids we wish to export.
SelectiveExporter does a sequence of
mysqldump calls, assembling the set of data related to the account to export.
scp the result and import it locally.
The payoff is huge: we’re able to grab data needed to troubleshoot a problem in 15 seconds instead of 30 minutes. That puts a smile on our developer faces.
Last year I ponied up for a 256GB SSD for my Macbook Pro. Six months later, I can confidently give the verdict that it’s the best investment I made in my work setup. Just a few datapoints:
- Scout’s unit test suite runs in half the time: 1m34s vs. 2m58s with my old setup
- Photoshop opens in under four seconds
- XCode opens in under three seconds
- everthing else opens pretty much instantaneously—I don’t keep mental track of what’s open or not, because there’s essentially no overhead.
These little efficiencies add up over time. More importantly, though, if I know the friction is minimal, It’s much easier to do the right thing—run the test suite one more time or open up Photoshop for that final adjustment.
A quarter of returning visits to Scout are less than five minutes in length. It’s a tad longer than an elevator pitch on the high end.
Like a real-life elevator pitch, you need to make a good impression quickly for these short visits. A user that spends twenty minutes on a site may excuse a slow page load. On short visits, a single slow-loading page sticks out.
Much of our work is focused on making visits to Scout more efficient. One example: for a long time, our homepage didn’t have a login link. We didn’t think this was important as many of our customers bookmark their account url. However, we saw a couple of cancellations that mentioned the problem with this: if you forgot your account url, it’s a pain to dig back through your email inbox and find it. Taking one minute to remember how to login for a two minute visit is painful.
We added the login link: 30% of logins now start on our homepage.
Reviewing your site’s visit length data is a great reality check. Is your next big feature going to make life easier or more confusing for elevator pitch visits?
We lost the browser state when we replaced our Charts UI w/AJAX manipulation. While it’s possible to maintain state using an anchor ala Gmail (ie – gmail.com/mail?#inbox/THREAD), it breaks the separation between business and view logic. The server cannot access the state information contained in the anchor – it’s only accessible from the browser.
Your cluster of servers ping Scout every minute, but you may only login to our website once a day. There’s a large gap between the amount of work on the data-processing side and the number of times it results in a visit to Scout. It’s a typical profile of a data processing application.
For a write-heavy application like Scout, hardware costs can increase linearly with user growth. It doesn’t scale like a read-heavy web application that can leverage a fast, in-memory caching layer for frequent reads. You need to make writing data more efficient.
For us, teaser checks have dramatically decreased the time spent processing data.
When I bike into work, I coast down a decent hill. With gentle curves and smooth pavement for 99% of the decent, it’s a refreshing way to start the day. My one complaint: there’s a 2×2 ft. section in the final turn with bad pavement.
This tiny imperfection breaks up the natural flow: I need to remind myself of it before I descend. It’s a lot like my favorite feature requests: those that begin with “I expected it to …”.
These requests often hint at tiny imperfections our users experience, but live with. When fixed, it makes an entire workflow seem natural.
When a server reports data to Scout, a small bit of data is updated in a MySQL MyISAM table. Each of these updates doesn’t change much data:
UPDATE `plugins` SET `last_reported_at` = '2010-11-17 19:53:00' WHERE `id` = 999999
I assumed that only the updated data was written to disk. I was wrong: Jesse Newland of Rails Machine pointed out that updating a row in a MyISAM table rewrites the entire row to disk (not just the updated columns).
These updates were expensive: we were updating small bits of data, but our table contained a large text column that accounted for 75% of the data stored in the table. Every update to a row in our plugins table would rewrite the large text column (even though the text column wasn’t changing).
There are no formal documents to sign if you and your spouse decide to have children. You don’t have to sit through an accreditation class. There is no credit check. You don’t need a high school diploma. Procreation can even happen accidentally.
A baby is a lot like a Rails application: the problem is caring for it, not creating it.
Five years ago, the typical Rails stack was just a couple of pieces: Apache/Mongrel, Rails, and MySQL. While Rails is remarkably similar to its original form even today, the stack around it is dramatically more diverse. We’re deploying to automated infrastructures, using NoSQL databases, messaging systems, queuing systems, and more. With the increased complexity of web applications, I’m surprised we’re not seeing companies dedicated to 24/7 infrastructure support: it doesn’t matter where your app is hosted, they manage it.
We’re fascinated with complexity. Write an article about the
enormously high throughput/disk space/concurrency your application
pushes and people will read it. You’ll speak at conferences. People
will follow you.
It’s fun talking about a complex system, but it can be tiring caring for
one. I love Woz’s focus on efficiency when starting Apple:
It’s can you, Steve Wozniak, design the same computer – maybe it’s a
Varian 620i – can you design it on paper with fewer chips than last
month? Can you design it with 79 chips instead of 80 chips?
Here’s to the oft-ignored, seldom blogged-about, patch-commiting developer doing the dirty work, making the tools we already use more efficient a line at a time.
Years back, before Scout, I used to schedule deployments outside the regular workday. I didn’t give it much thought – it was what my consulting clients were used to. However, we changed that practice with Scout.
Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment. With that in mind, we started scheduling deployments when it’s best for us.