How we're turbo-charging our traces with ScoutProf

By Derek Bullet_white Comments Comments

The transaction trace is application monitoring. If you can't track a performance problem to a line-of-code, you aren't app monitoring.

However, there's a gaping hole in the quality of transaction traces today.

We've been instrumenting the quality of our traces since our launch, and noticed that 46% of the time spent in an average Rails request is in custom application code. This falls outside the instrumentation that we, and every other app monitoring service, like New Relic, provides.

Let's fix that.

It started with Stackprof

The release of Ruby 2.1 introduced a C-api for fetching ruby backtraces. What's great about this API? Here's Aman Gupta of GitHub:

The (rb_profile_frames()) api performs no allocations and adds minimal cpu overhead making it ideal for profiling, even in production environments.

Amon wrote Stackprof, a terrific call-stack profiler that leverages rb_profile_frame(). We used it extensively to improve the performance of our agent.

However, with all of that visibility comes a problem: Stackprof output is difficult to decipher.

Enter ScoutProf

ScoutProf builds on StackProf: we're aiming to make all of that data easily decipherable plus work in threaded environments (like Puma). The key part to making it decipherable and actionable? The aggregation. From our docs:

ScoutProf attempts to sample your application every millisecond, capturing a snapshot backtrace of what is running in each thread. For each successful backtrace captured, that is a sample. Later, when we process the raw backtraces, identical traces get combined and the sample count is how many of each unique backtrace were seen.

Screenshot please?

Let's take a look at a trace from New Relic. Notice 2/3rds of the time is spent in "Controller":

nr trace

If you inspect the detailed trace, you'll see the following:

nr trace

Sad face :). That's a big black box of time.

Now, here's a trace to that same controller-action with ScoutProf:


The mystery of "Controller" time revealed: we're seeing a breakdown of time spent in our custom code with zero configuration.

The big consumer above is ChartPartData#descriptors. What's going on there?


We're seeing the overhead of ActiveRecord: all that ORM doesn't come for free. Now, who knows about this code? Let's find out with Scout's GitHub integration:


Looks like I should talk to Andre :).

Improving the performance of complex Ruby code is time-consuming: it frequently involves the coordination of several people on your team. We're hoping to make it easier to identify these bottlenecks and the humans that can solve them.

Using ScoutProf

Our instructions for enabling ScoutProf are on our help site. Note that ScoutProf is in BETA.


Know of an open-source Rails app in need of some performance love?

By Derek Bullet_white Comments Comments

We've just released the BETA version of our new Ruby Profiler (Stackprof-inspired, but hopefully more readable). To really give it a go, we're looking for some open-source Rails apps that are need some performance love.

If you know of - or our a contributor to - an open-source Rails app that needs some help with slow Ruby, ActiveRecord query optimizations, etc, reach out!


New Release: Memory Bloat Detection

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

When your app is experiencing memory bloat - a sharp increase in memory usage due to the allocation of many objects - it's a particularly stressful firefight.

Slow performance is one thing. Exhausting all of the memory on your host? That can bring your application down.

Memory bloat is a scary problem: our 2.0 release of the Scout agent aims to track down memory hotspots as easy as performance bottlenecks.

Read on: I'll share what's hard about memory bloat detection, Scout's approach, and some further reading.

Read More →


Faster PostgreSQL Searches with Trigrams

By Greg Bullet_white Comments Comments

There's nothing quite like having a "tool-belt" full of tricks for getting the most performance out of your Rails app. This week, Rails Postgres Guru Greg Navis shares another powerful tip.

A few months ago, I was working on a project that had about 100,000 users. Each user could have multiple names, emails, phone numbers and addresses associated with his account. All these objects totaled to about 500,000 database rows. Customer support staff would search this dataset multiple times a day. One day, they started seeing timeouts.

In this article, I'll show you how I sped things up with the PostgreSQL trigram extension.

Read More →


A Six-Pronged Rails Performance Philosophy

By Sudara Bullet_white Posted in Plugins Bullet_white Comments Comments

"An ounce of prevention is worth a pound of cure.” ― Benjamin Franklin

Application performance problems can be annoying. With luck, you'll spend an hour or two resolving the problem and get back to your real job: building things.

But what happens when the issues start piling up? What happens when "poor performance" becomes the norm?

When looking at performance issues in aggregate and cross-application, a deeper root cause can appear. This can be tough for a team to identify and even tougher to act upon.

Read More →


Product Update: enhanced zoom, 95th percentiles, trace diffs, and more.

By Derek Bullet_white Comments Comments

Today we're excited to announce the release of our latest Ruby on Rails monitoring agent and a major UI update. This release continues our focus on quickly revealing your path to a faster, more reliable Rails app.

Enhanced zoom

Determining how to make your Rails app faster often involves investigating the opposite: inspecting your app's worst performing controller-actions and their requests. We're working to make this investigation effortless. Just follow your intuition when you see a problem area on a chart:


Click-and-drag on the chart to reveal a list of endpoints we've collected detailed transactions traces for during that period. For context, you'll see a sparkline of each endpoint's response time. Sort by slowest response time or largest memory increase.

Read More →


Older posts: 1 ... 9 10 11 12 13 ... 68