January 18 Derek

Real Time Infographic

Posted in Updates |  Comments

It’s been a little over a week since we rolled out real-time monitoring. Some people think it’s pretty cool.

To commemorate one week of real time, our art department put together a basic infographic on its usage. You’re spending more time watching lines move on the screen than YouTube Nyan Cat videos. We’re damn proud of that.

Learn more about real time server monitoring with Scout.

January 09 Andre

Announcing Scout Real Time

Posted in Features |  Comments

Deploying a new feature? Load testing? Debugging a performance issue? Now you can get Scout metrics in real time for immediate feedback.

Real time in Action

(2 min 28 sec—we know your time is valuable).

Using Real Time

Choose the metrics you want to view in real time using the standard charts interface. Then, click the Real Time button:

The real time charts look a bit different from the standard Scout charts. Each metric is displayed separately:

Note that the real time session doesn’t start immediately. That’s because Scout needs to launch a separate process on your server(s) for the duration of the real time session. It can take up to a minute for all your servers to check in and get the message that real time needs to be started.

It’s Like Your Servers are There in the Room

If Scout is a periodic status report from your servers, real time is like a live, in-person meeting. You and your servers face-to-face.

Of course, you don’t want to be in meetings 24/7. Scout real time is the same way – it’s designed for short periods to address a specific issue. By default, real time sessions run for 30 minutes, and you can extend the sessions for longer if needed.

Try Real Time

For a limited time, real time is available FREE on ALL Scout accounts. You do need the latest Scout agent to try it. As long as your Scout gem is version 5.5.0+, real time will automatically be available on your charts.

Get Started!

1. Update your Scout gem: [sudo] gem install scout
2. click the Real Time button on any Scout chart

December 30 Derek

The Year At Scout: 2011 Edition

Posted in Updates |  Comments

I’ll remember 2011 as the year of fine-tuning at Scout. From your experience with the end product to our experience delivering it, we invested a lot of our time sanding Scout.

Feature Highlights

• Dashboards – Combine plugin displays and charts across servers for a complete view of your infrastructure’s health.
• Easier Plugin Development – Since we started Scout in 2008, we’ve believed a big reason services go unmonitored is that writing and testing monitoring scripts is painful. We’ll continue working to make plugin development easier.
• Scout API – With the scout_api Ruby gem, you can slice and dice your metrics as you see fit.
• Fullscreen Charts – Because you look smarter with pretty lines moving on a wall-mounted plasma display.
• Spotlight-like Server Navigation – You’re monitoring more servers with Scout, so we made it easier to navigate between them.

A Banner Year for Plugins: 200+ commits from 20 authors

With nearly 60 monitoring plugins in our directory, Scout’s breadth of coverage grew substantially in 2011. Some of the highlights:

• CouchDB – Authored by John Wood
• HAProxy
• JMX Monitoring – Authored by David Dossot

But it’s not just new plugins that I was excited about: Scout plugins grew incredibly more robust. Our plugin repository on Github had more than 200 commits from 20 authors in 2011. Thanks for all of your pull requests and bug reports: there are few things I love more than solidifying Scout’s core.

Blog Post Highlights

We think it’s important for you to know how our brains function at Scout. Some of the highlights (or lowlights, depending on your interpretation):

• The Only Two Business Metrics That Matter
• Don’t build to get acquired. Build to say “No”
• Tina Fey: go with the bran muffin on the white plate

Implementing our succession plan

We plan on Scout being around for a long time. 2011 saw the start of our grand succession plan: both Andre and I brought home baby girls this year. While my two-month old daughter is lacking focus at this point, I’m confident she’ll be ready to take over my share of Scout by 2035.

2012

I’ve never been more excited about Scout. We eliminated a lot of cruft from Scout in 2011 that’s freeing us up to make monitoring even easier in 2012. Thanks for your continued feedback, bug reports, and plugin commits: you’re making Scout better every year.

December 09 Derek

Silencing Scout during maintenance

Posted in Features |  Comments

While performing infrastructure maintenance, Scout suddenly fires emails to your team members telling them that MySQL, Redis, HAProxy, etc. are down. They are, but it’s planned. You’re forced to respond to a barrage of emails, SMS messages, Skype messages, and guttural screams telling everyone that things are fine. We’ve added a Disable notifications for everyone option to Scout that should suppress these panic attacks:

When selected, all notification emails and SMS messages will not be sent (they’ll still appear in the Scout UI). Once maintenance is complete, simply unselect the checkbox.

November 28 Derek

Do we really need a phone number?

Posted in Business |  Comments

Like a bug zapper on a Midwestern summer evening, our company phone number attracted lots of annoying visitors: partnership offers from the Middle East, RFP requests for the wrong product, support inquires for a dating service, etc. In fact, I can only recall a handful of calls from real-life customers.

Did we have a phone number because it solved a problem or because we thought businesses – serious ones – are supposed to have one?

For now, we’ve stopped displaying the phone number on our website. While I don’t think this makes sense for every business, for a developer-focused business like Scout I’m betting it won’t be an issue.

November 21 Andre

New Plugin: Simple Process Check

Posted in Plugins |  Comments

Simple Process Check is a new plugin in the spirit of the recent Simple Port Check.

Provide a comma-delimited list of process names (names only, no paths). This plugin checks that at least one instance of each named process is running.

Alerting

1. You’ll get an email if any of the processes have NO instances running.
2. You’ll get another email when any of the non-running processes are present again.

Simple Process Check vs. Process Usage

The Process Usage keeps tabs on one specific process name. It tracks the number of instances running, the total memory usage, number of restarts, etc.

Simple Process Check keeps tabs on multiple processes (according to the list of names you provide), but provides less information on each process. You should use Simple Process Check if you have a number of processes to monitor, and primarily need assurance that they all are running.

Try it out

That link again: Simple Process Check. Let us know if you have any questions or feedback.

October 25 Andre

New Plugin: Simple Port Check

Comments

Simple Port Check is, well, simple: give it a list of ports, and it checks that each port will accept a TCP connection.

If the plugin detects that one or more ports stops accepting TCP connections, you’ll get an email notification. When the ports are available again, you’ll get another email:

The link again: Simple Port Check Plugin. Any questions or feedback, feel free to drop us an email.

October 21 Derek

Monitoring HAProxy

Posted in Plugins |  Comments

HAProxy is a rock-solid, high performance TCP/HTTP load balancer. It’s what we use at Scout to proxy traffic to our app servers.

We’ve added a plugin to monitor HAProxy: just specify the URL to your HAProxy stats page and the name of the proxy. You’ll get the request rate, error rate, and proxy status for the specified proxy.

Why monitoring the load balancer is important

The load balancer is one of the best places to narrow down the cause of an outage. If all servers in a proxy aren’t processing requests, it’s often an indication of an outage further down the stack (ex: the database). If it’s isolated to a specific app server, the first place to look might be the downed app server.

The Scout HAProxy plugin gathers its metrics by parsing the output of the HAProxy Status page. It’s one of my favorite open source status pages: the page loads quickly and the color-coded statuses make it easy to focus on problem areas during an incident. The Scout plugin and the HAProxy status page play well together: if the plugin records a dramatic change in the throughput, error rate, or the proxy status, we’ll pull up the HAProxy status page and dig further.

Installation Notes

Like all Scout plugins, installing the HAProxy plugin is just a couple of mouse clicks away. Just click the button in the Scout UI and select the HAProxy Monitoring plugin. Note that the plugin has one dependency: the fastercsv gem must be installed. See the plugin directory entry for more details on the plugin.

Credit

A big thanks to Jesse Newland for the initial version of this plugin.

October 18 Derek

Our poor choice of words: "Server Down" alerts are now "No Data" alerts

Posted in Updates |  Comments

What we’ve been referring to as “Server Down Notifications” suffer from two problems:

• False Positives – The last thing you need is an alert from Scout at 3am telling you that your server is down when it isn’t.
• An overaggressive email subject – The subject line of these emails looks like “Server is DOWN”. However, this isn’t accurate. We just know that Scout hasn’t received data from this server. That doesn’t necessarily mean the server is down.

We’re making two changes to these alerts:

• The email subject now states “Server isn’t reporting”. This is really all we know.
• These alerts used to fire when the agent didn’t report for five minutes. This was overly aggressive. While it doesn’t happen often, things can go wrong between your server and ours that can cause a reporting pothole. Alerts are now sent when the agent hasn’t reported for 30 minutes. It’s important to tell you when monitoring isn’t working, but not important enough to risk sending a false positive because of a short reporting outage.

So, what do we suggest to verify that a server is down? An external monitoring service like Pingdom is a good option. Services like Pingdom can alert you if a server can’t be pinged and/or isn’t accepting SSH/TCP connections. Failed external checks and a lack of internal metrics from Scout often indicate that a server really is dead.

We’re confident (1) keeping our finger off the alarm button a bit longer and (2) calling these alerts what they really are will give you a more comfortable experience. We’re developers too and we know the heartburn a 3am SMS causes.

September 28 Derek

Making Scout feel young again: our 4-part tonic

Posted in Development |  Comments

Scout is no longer a puppy: in dog years, he’s old enough to drink, get drafted, and rent a car. During that time, cruft gathered around the edges of our server infrastructure.

We’ve been using a hodgepodge of server hardware, some performing multiple roles, some not, all individually configured and tuned. Small changes to our stack seemed to involve a lengthy checklist. Our staging environment didn’t mirror production: what happened on staging didn’t always happen on production. Finally, database changes were painful.

We wanted to get lean in the right places: could we make the young adult Scout as easy to manipulate as the baby Scout? 7.weeks.ago we followed a four-part process to get there.

Read more...