"Development" Posts

Getting down with Stackprof: how we added N+1 query detection to Scout

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Let's get the confusing part out of the way: we're going to get a little "meta". This post is about using one performance tool (Stackprof) to improve another performance tool (Scout's Rails Agent) so Scout can diagnose a common performance pitfall (N+1 queries).

Did I just put "performance" in a sentence three times? You can see I'm a little obsessed.

There's plenty written about why N+1 queries are bad. Is there anything good about an N+1?

Yes! Finding an N+1 is like finding a $20 bill in your couch. It's a performance-enhancing gift and often an easy fix - assuming you can find the spot in your code that is emitting the query.

...and there-in lies the problem. It is difficult to trace problematic N+1 queries on complex apps running in production back to a line-of-code. In fact, I don't know of a tool today you can run in production that does this. Why is it difficult? Grabbing backtraces is not cheap - you can't call Kernel#caller all willy-nilly in your instrumentation. It means you are left with deadends like this when trying to track down an N+1:

missing n+1

In the above screenshot, each ArDescriptor#find query executes quickly (22 ms), but there are 23(!) queries with the same SQL, adding up to nearly 500 ms.

Now, I don't know about you, but I find this incredibly frustrating. Armed with Stackprof, we set out to see: could we make it performant to find the source of N+1s with Scout?

Read More →


Understanding page faults and memory swap-in/outs: when should you worry?

By Derek Bullet_white Posted in Development, HowTo Bullet_white Comments Comments


Imagine this: your library is trying to step up its game and compete in the Internet age. Rather than you browsing the shelfs, trying to remember how the Dewey Decimal works, you'll enter your book selections from your phone. A librarian will then bring your books to the front desk.

You place your book order on a busy weekend morning. Rather than getting all of your books, the librarian just brings one back. Sometimes the librarian even asks for your book back, tells you to walk out the door to make room for others, and lets someone else read their book for a bit. They then call you back in, shuffling you and the other book readers in-and-out.

What's going on? Is the librarian insane?

This is the life of the Linux's memory management unit (librarian) and processes (you and the other book readers). A page fault happens when the librarian needs to fetch a book.

How can you tell if page faults are slowing you down, and - above all - how can you avoid being shuffled in-and-out of the library?

Read More →


From MySQL full-text search to Elasticsearch

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Migrating backend search technologies on a high-throughput production site is no easy task, but Vector Media Group was recently faced with this decision. With a popular client site struggling under the load of complex MySQL full-text search queries, they recently switched to Elasticsearch.

I spoke with Matt Weinberg to learn how the migration went. Was the switch to Elaticsearch worth the effort?

How did you handle search before Elasticsearch?

We created a custom search using MySQL queries and implemented it into our CMS for the project, ExpressionEngine.

What were the problems with this approach?

To support full-text search, we needed to use the MySQL MyISAM storage engine. This has major downsides, the primary one being full table locks: when a table is updated, no other changes to that table can be performed.

Our tables have considerable update activity, so this would result in sometimes-significant performance issues.

Read More →


From Ruby to Go: a rewrite for the future

By Derek Bullet_white Posted in Development


During a team camp among the lofty peaks of Breckenridge, Colorado, we talked a lot about the future of Scout and monitoring in general. Big mountains and nature have a way of doing that.

One thing that was getting our nerd juices flowing: Go.

At Monitorima in May, it was clear that Go was becoming the language of choice for performant yet fun-to-develop daemons.

After our morning hike fueled us with crip mountain air, we said: why not build a light Scout daemon in Go? As in, right this afternoon?

Read More →


From MySQL+MMM to MariaDB+Galera Cluster: A High Availability Makeover

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Kevin Lawver, President @ Rails Machine, is our guest author for this post.

Few things feel worst than rolling out a High Availability (HA) system, then regularly seeing that system collapse. For our team at Rails Machine, that failing HA system was MySQL Multi-Master Replication Manager (MMM).

We've been searching for a MMM replacement for a while, and a few months ago, we made the switch to MariaDB + Galera Cluster for High Availability MySQL.

What's wrong with MySQL MMM? What's special about Galera Cluster? Read on!

Read More →


Docker: Git for deployment

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

I've been hearing how Docker is the new awesome, but it didn't click for me until I dug in with a practical question: if we deployed Scout via Docker, would deployment be a more pleasurable experience?

My three takeaways are below.

Read More →


Older posts: 1 2 3 ... 8