New Relic vs. Scout: which is right for you?

October 15 Bullet_white By Derek Bullet_white Comments Comments

There's no type of monitoring tool that can get to the heart of a performance problem faster than Application Performance Management (APM).

But...that value comes at a cost. APM services aren't cheap. It's a complex product to maintain and there's a lot of data to ingest. Picking the wrong product for your team is especially painful when that product isn't a trivial expense.

If you're making this decision, you are possibly choosing between New Relic and Scout. New Relic is the original in the SaaS space (launched in 2009) and Scout is the new alternative (launched in late 2015). There are clear cases where New Relic is a better fit, clear cases where Scout fits like a glove, and some murky situations as well (this is real life, not Star Wars). Where do you fall?

In this detailed post, I'll walk through the factors to consider when choosing your best APM product.

ALL the data vs. the 80/20 rule

You've probably heard of the Pareto Principle via another name: "the 80/20 rule". For example, it's common for many businesses that 80% of the revenue comes from 20% of the customers.

The Pareto Principle applies to a dizzying array of natural phenomena. This includes web app performance. In almost every web app, there's a few dominant web endpoints, background jobs, and significant database queries.

Scout applies the 80/20 rule extensively to web performance: significant focus is applied to reducing the time spent in the most time-consuming areas of a web app. Alternatively, New Relic has more breadth and a more equal treatment of metrics.

Let's talk about which approach to monitoring applies to different engineering team compositions and development processes.

How does your team's DNA impacts monitoring?

You'll get more value out of your choice of APM service when more of your team uses it. Some questions to consider:

  • Does your team have a large percentage of senior engineers? New Relic's product breadth can be intimidating, but that's likely less of a factor if many folks on your team are senior engineers. Conversely, if the experience level is mixed, Scout can direct mid and junior-level engineers to problems in quick fashion.
  • Are you deploying daily or multiple times per day? If your code has frequent churn, it's easy for performance issues to be introduced. Scout's GitHub integration makes it easy to track issues to a developer and can save investigation cycles in high-churn environments.
  • Do you prefer to work with a single vendor? New Relic offers several products under one roof with integrations between each product. If the budgeting simplicity of a single vendor outweighs the investigation time spent in a best-of-breed approach, New Relic may be a better fit.

Let's start digging through the products, starting with the app overview.

App Overview

This is the first page you see when checking the health of your app. New Relic's version is on the left and Scout is on the right:

app overview

The top portion of the page is similar between New Relic and Scout: a breakdown of time spent by category (ex: Ruby, Database, External HTTP services, etc) over time. You can view data across similar timeframes in both Scout and New Relic (New Relic offers three months of data in their Pro package and Scout can do the same in their custom plans).

Both let you compare data to the past. Scout gives a few more options than New Relic, which is restricted to yesterday and last week comparisons.

The biggest difference with Scout's approach: insights on the app performance are provided directly on this page. Scout has several algorithms that continually dig through problem areas of your app, identifying sources of memory bloat, database query bottlenecks, and N+1s. These are ordered by impact, which help you decide where to get most improvements in the least amount of time.

Derek
Scout Founder's Take
Insights have become a customer favorite. They short-circuit a lot of the time spent exploring an APM tool. We've found it make performance work more approachable for more members of your team: there's no digging.

Let's move from the 10,000 foot view to the soul of application monitoring.

Transaction Traces

The soul of app monitoring is the transaction trace: all the metrics and analysis an APM platform delivers originates from data collected in a trace. A transaction trace is a breakdown of time spent in a single web request or execution of a single background job.

A New Relic trace is on the left and Scout on the right:

transaction trace

There are some differences here as well:

  • Scout provides a breakdown of memory allocations in addition to timing metrics. New Relic does not.
  • New Relic offers cross-application tracing if your app touches another app you've monitored with New Relic. Scout does not.
  • Scout is more likely to provide backtraces to slow method calls and can also display the code directly in the browser.
  • Scout indicates the number of rows returned by SQL queries
  • Scout can breakdown the time spent in custom code via ScoutProf. New Relic requires custom instrumentation.

Scout typically collects more traces than New Relic. Both New Relic and Scout have similar, one-week retention periods for traces.

Derek
Scout Founder's Take
For the trace, we focus on action: which code paths are significant bottlenecks? What is the code in those bottlenecks? What are the details around slow SQL queries (ie: number of records returned)?

Web Endpoints

Both New Relic and Scout let you view summary data on web endpoints in list format. New Relic on the left, Scout on the right:

web endpoints

New Relic opts for several different visualizations on this page. Scout goes with a filterable, sortable view.

Both sort web endpoints by percent time consumed by default, helping you focus on the controller-actions that consume the greatest amount of time.

Derek
Scout Founder's Take
My favorite feature on this page is the filter above the list of endpoints. It drastically reduces the time needed to navigate to an endpoint or a group of endpoints I'm investigating. Since this is a high-frequency action, the time savings adds up.

Web Endpoint Detail

In the same way the app overview provides a breakdown of time spent, both New Relic and Scout offer endpoint-specific views for this:

web endpoint detail

There are some differences on this page:

  • New Relic offers a histogram display option of response times and Scout does not.
  • Scout offers memory allocation metrics (more allocations lead to increased memory usage) while New Relic does not.
  • New Relic's breakdown provides more categories, while Scout focuses the breakdown on database queries.
  • Scout typically provides more transaction traces (up to 10 per-minute) and more options for sorting and selecting traces.

Background Jobs

Both New Relic and Scout monitor the core Ruby background job frameworks: Sidekiq, Delayed Job, and Resque. The UI is similar to endpoints, so I haven't provided a screenshot here.

Database Monitoring

The database is the most common bottleneck for web applications. Since a database is a shared resource, one expensive query may cause many other types of queries to run slower. New Relic and Scout provide additional analytics around database query performance:

database

Both of the UIs resemble each tool's web endpoints display, but made specific for database queries. There are some subtle yet important differences between the two:

  • Scout's chart displays data across every query - not just the top five most expensive queries - by combining less expensive queries into an "other queries" metric. This gives a more complete picture of performance.
  • New Relic provides monitoring beyond ActiveRecord. Scout focuses on ActiveRecord.
  • Scout's list of queries includes the calling controller-action or background job. Expensive queriers are frequently triggered by a few hotspots in the code: this makes it easier to identify those spots at a glance.

Finally, Scout makes it easy to compare performance in a small slice of time to the normal performance. This isn't possible with New Relic - you can change the entire timezone, but you can't compare between a small slice of time and a larger one. Here's how this looks within Scout:

db_zoom

Development Profiler

When you are actively resolving a performance issue, it's helpful to get immediate feedback on the results as you code. Otherwise, you need to deploy and wait for fresh data to verify the change had the intended impact.

New Relic used to have a developer mode, but it has since been removed from the agent.

Scout DevTrace gives immediate access to traces in your local development environment by clicking on a speed badge in the lower left of your browser:

development profiler

Derek
Scout Founder's Take
Slow feedback cycles - like a long-running test suite - are painful. DevTrace eliminates a lot of this as you can instantly verify that an optimization - like fixing an N+1 query with `includes` - is working as expected before deploying.

Weekly Trends Email

A recurring report of your app's health is a great way for managers to stay in the loop on performance and to uncover slowly building problems. New Relic puts more emphasis on high-level numbers. Scout digs more into the source of problems and trends:

weekly email

Derek
Scout Founder's Take
Weekly emails with lots of metrics become repetitive. We attempt to dig through the data and identify underlying causes - like a specific slow query - to push you towards a fix.

Alerting

Scout and New Relic have comparable alerting functionality.

Agent

Both the New Relic and Scout agents are battle-tested. In Scout's open-sourced benchmarks, Scout's agent was shown to have lower overhead:

APM Agent Response Time
(Mean)
Response Time
(95th Percentile)
Response Time
(Max)
Overhead
None 55.6 ms 106.4 ms 2,174.1 ms
New Relic 80.4 ms 149.5 ms 2,263.5 ms 44.5%
Scout 56.8 ms 102.7 ms 2,168.7 ms 2.2%

The impact of agent overhead increases as CPU resources on your app servers becomes more scarce.

Error Analytics

Scout tracks error rates per endpoint but does not offer details on exceptions. New Relic can provide this data, including backtraces to exceptions.

Pricing

Scout's monthly fee is typically about 30% lower than the monthly fee in a New Relic annual contract.

Language Support

Scout supports Ruby and Elixir. New Relic supports Ruby, Java, Node.js, PHP, .NET, Python, and Go.

Why don't you just tell me which to use?

If you've read this far, you're intentional in your choice of tools. That's important: your team will rely on your choice in your app's worst moments.

The great thing about comparing app monitoring tools? They require little configuration - outside a gem install - to try. You can run Scout and New Relic at the same time without causing conflicts.

My suggestion? Try New Relic and Scout at the same time, then ask: "which tool helps our team solve performance issues faster?"

Get notified of new posts.

Once a month, we'll deliver a finely-curated selection of optimization tips to your inbox.

Comments

comments powered by Disqus