Last week, one of our application servers died. We have four app servers, so in theory, the death of one app server shouldn't bring the entire platoon down. However, real-life had other plans: 95% of requests were handled fine, but around 5% were being dropped. Here's the story of how we diagnosed and fixed the issue with our realtime charts.
I’m sitting in the Denver Airport – in a couple of minutes, I’ll board the plane to RailsConf in Portland, Oregon. I’m already getting amped for Voodoo Donuts, Stumpdown Coffee, well-trimmed beards, and of-course, lots of Rails-related chats.
I’m bringing a fresh load of Scout T-Shirts. These aren’t your normal heavy-weight, poor-fitting shirts. They are tastefully designed, American Apparel – Tri-Blend (otherwise known as the most comfortable shirt you’ll own). If you’re attending RailsConf, shoot us an email so we can meetup and improve your wordrobe at the same time. Or, just look for us (Andre and Derek). We don’t always rest our arms on each other, but when we do, we look like this:
It's been three weeks since the launch of the largest feature enhancement in Scout's existence: roles. Haven't heard of roles? Nutshell: roles let you monitor many serves with fewer clicks and more joy. Roles were driven by your feedback and it's showing in the fast adoption numbers below.
Time to give an awkward nerd high-five of thanks:
- Customers on our Roles BETA program - your feedback and willingness to try new things helped us iron out the edges for the public rollout.
- Contributors to our Chef recipe - we've already had six authors commit to our Chef recipe for deploying Scout. It's great to see a hardened Chef recipe based on real-world usage.
- Feedback since the launch - we built roles because of your feedback, and we've enjoyed reading your suggestions post-launch.
Haven't tried roles yet? To get started, see the "Roles" dropdown on your account, and read the FAQ on roles.
Whenever we’re asked how to make on-call notification schedules for Scout alerts, we recommend PagerDuty. PagerDuty has invested a ton of time in building a dedicated notification scheduling service, and it’s a great complement to Scout.
With our recent release of notification groups, Scout’s integration with PagerDuty got even more powerful:
- Multiple PagerDuty services: add as many PagerDuty services to Scout as necessary.
- Trigger-specific escalation policies: assign any PagerDuty escalation policy to any threshold in Scout. If you need to create multiple thresholds on a given metric with different escalation policies, it’s simple to do – just add another trigger.
- Automatic incident resolution from Scout: since all integrations are routed through PagerDuty’s API, Scout now auto-resolves any PagerDuty incidents when Scout’s trigger stops firing.
Multiple services in PagerDuty:
... and those same services integrated into Scout:
Adding PagerDuty services within Scout
You need to start in Scout to create a PagerDuty integration:
- Click on Notifications (in the top navigation bar),
- Click on “Add PagerDuty Integration.”
- You’ll be given the option to create a new PagerDuty service, or connect to an existing service within your account.
To assign a PagerDuty service to a trigger, ensure the PagerDuty integration is part of a notification group (the notification group can contain other items too, if needed), then assign that notification group to a trigger.
It’s been a month since I started attaching torture devices disguised as boots to my feet, long wooden sticks to each torture device, and tumbling down mountains. Skiing has changed my outlook on winter. It’s a season to enjoy, not a time where I gaze wistfully out the window, hoping the short, cold days pass by as quickly as possible.
However, there’s a problem when skiing becomes a favorite hobby: not everyday is a great day on the mountain. If it hasn’t snowed in a while, the surface is hard. The temperature might be in the single digits and the wind may be gusting 50 MPH+. It might dump snow in the backcountry, but the avalanche conditions may make it unsafe.
There’s something special about being able to sneak away when the conditions are the best, even if it’s during the work week. It feels a bit like being a kid again (correction: a kid with a receding hairline). It’s a fun reminder that it’s not always bad to feel redundant.
We recently decided it was time for a major update to the public side of Scout. We’d start with a more polished homepage. Since we’re both developers, the obvious next step seemed like hiring a designer. However, working with an outside designer isn’t a hire-and-forget experience:
- Good designers are difficult to find. Design doesn’t scale like a product business.
- Good designers are busy. It could take 30-60 days to start work, then another 30 days for it to come together. This means we could be looking at a 90 day timeline. We wanted to launch it faster.
Instead of starting work with a designer on a blank slate, we decided to start firming up what we wanted the homepage to look like. We’d end up with one of the following outcomes:
- We’re terrible at design, but we’ve at least thought it through. Hire a designer.
- We can get 80% of the way there, but we’ll need a designer for touchups.
- If we iterate enough, we can launch something we’ll be happy with.