Every man has their breaking point when it comes to deadweight code. Andre and I hit ours recently and decided to spend all of last week focusing soley on cleaning up Scout (a Rails app). Our goals:
- Faster tests – our tests took 8 minutes to complete. While it’s the perfect amount of time to catchup on Daily Show clips, it really tested our patience making application-wide changes.
- Removing deadweight – unused CSS rules, database tables + columns, views, and assets. It’s good having certainty that modifying code will change something in the application.
Here’s how we went about it:
We’ve added prolific plugin contributor Eric Lindvall’s latest plugins to Scout: Kestrel Overall and Kestrel Queue Monitoring. Kestrel is a simple message queue built from production needs at Twitter. Being the gentleman he is, Eric shared his experiences with Kestrel at Papertrail, a hosted log aggregation service.
As all good message queues should, it provides:
- transactional reads to prevent losing messages if a worker crashes before finishing processing a message
- durability to prevent losing messages if a queue server crashes or is stopped after messages have been sent to it
- “normal” queues (only one client sees a given message) and “fanout queues” (all clients see a given message, which many messaging systems call “topics”)
We picked Kestrel because it has bounded memory usage (it has a configuration setting to specify how many queued messages should reside in RAM, generally defaulting to 128MB), it’s small (I was able to read the entire Scala codebase in a weekend), and running on the JVM (which we have experiencing with) was a plus.
Last week, Sparrow became the latest poster boy for talent acquisitions (Google gets the team, kills the product). Paying customers complain (I supported it!). Indie devs get depressed as one of their rank sells out.
I disagree with Matt Gemmell that these are a good thing – this is not a feel-good rags-to-riches story. It’s about brilliant developers giving into reasonableness because they didn’t have the runway to be foolish.
There was a clear turning point for Andre and I with Scout – it happened in October 2009. It led to one of our most popular blog articles, “We Just Undid Three Months of Dev work. Here’s What We Learned.”
I’ll be telling that story at the Fort Collins New Tech Meetup tomorrow evening. If you’re a developer starting a business (or struggling turning a side project into one) drop by. If you’re interested in heckling me, stop by too!
We heard your feedback: adding charts to dashboards is a pain. No longer! Look for the Add Chart button on dashboards to add charts in one step. You don’t have to create a chart separately before adding it to a dashboard.
Big picture – we’re always looking for ways to make Scout easier to use. If there’s something clumsy or difficult you’d like to see improved, hit up the Scout suggestion page.
Recently in charts:
Josh Nichols of Rails Machine on their monitoring philosophy:
Measure all the metrics and alert on metrics that are actionable.
If you’ve got a bunch of servers, you’re going to want to read Josh Nichols’ How we roll with Scout article on the Rails Machine blog.
Rails Machine is a Ruby on Rails-focused managed hosting provider, which means they’re on the front lines when performance goes bad. From plugins to alerting, Josh details how they stay proactive on performance.
The only thing worse than being woken up by a critical alert: not being woken up when you should be. That’s where Pagerduty comes in—Pagerduty uses a combination of email, SMS, and phone calls to you and your team to ensure you never miss an important notification.
And now, Scout has baked-in support for notifications via PagerDuty. Scout can automatically create incidents when a trigger fires, and automatically resolve it too.
”Getting Scout alerts though Pagerduty works great. Setting it up was completely seamless.”—T.R. Missner, Firespotter Labs
Setup is Easy
Assuming you already have a Pagerduty account, integration is ridiculously easy:
- make sure you’re logged into Scout as an administrator.
- click on “Notifications”, then “Connect to PagerDuty.”
- You’ll be redirected to Pagerduty’s website. Enter your Pagerduty credentials and select the service to associate with Scout:
And you’re done! There’s no API key to copy / paste. If you ever need to disconnect, just click “disconnect” in Scout.
New to Pagerduty?
Pagerduty provides on-call schedules, configurable escalation rules, and incident acknowledgment for all your monitoring services. PagerDuty ensures the right person gets notified, every time. Browse the feature list here, and sign up here with a 10% discount just for Scout users.
Here’s a snapshot of Pagerduty’s notification dashboard:
Pagerduty is the Notifications Expert
If you need sophisticated notification options (like rotating schedules and escalation rules), Pagerduty is for you. Partnering with Pagerduty allows Scout to focus on what we do best—powerful, easy-to-setup monitoring tools.
Scout’s realtime charts have been a big hit. Once you start using them for major deploys or performance incidents, going back to ten terminal windows running “top” feels like the dark ages.
So, how did we go about it?
When you’re building a chart in Scout, you select metrics from a tree. It branches like this: group => server => plugin => metrics.
This makes it easy to drill-down to a given metric on a specific server. However, it’s very common to compare the same metric across servers. For example: how does our memory usage compare across all of our app servers? Say hello to the new metric context menu:
He’s a useful little guy. After selecting a metric in the chart tree, just click on the icon and select the instances you’d like to see on the chart.
Since we added low value support for peak triggers a few months ago, similar support for plateau triggers has been a common request.
We heard you: plateau triggers now also support low values. You can, for example, be alerted when when Apache requests drop below a certain threshold for 30 minutes or more.