Announcing the scoutd public BETA

By Dave Bullet_white Posted in Features Bullet_white Comments Comments

We've made our new monitoring agent, scoutd, publicly available. This new agent brings new capabilities to Scout's monitoring service - the most visible being seamless realtime charts on your dashboards. So what's different from the traditional scout agent, and what's in store for scoutd's future?

History

Some History, What's New?

Until now, our agent has been written in Ruby and distributed as a RubyGem. You'd install the gem and create a crontab entry so that it would run from cron every minute. While this is a reliable and easy way to report your system metrics to us, we want to lay the groundwork for some exciting things we have planned that only an always-running daemon could provide. So how does scoutd compare to the RubyGem agent?

  • Installed as a system package

    scoutd is installed by your system's package manager (yum/apt/etc). We also have a one-line install/upgrade command that makes new installs and upgrades from the old agent super simple: Install steps

  • No more cron entries - scoutd runs as a daemon

    scoutd is started by your server at boot and runs continuously. This allows instant communication between your server and Scout and lets us do things like instant realtime charts.

  • Simple config file, standard locations

    scoutd is easily configured via /etc/scout/scoutd.yml:

    account_key: abcd
    hostname: db1.scout.dev
    environment: development
    roles: database
    

    Local plugins, account key, etc are now located in /var/lib/scoutd

  • Written in Go

    While scoutd is written in Go, it doesn't introduce any new dependencies as we build distro-specific binaries. Scoutd still requires Ruby to be installed on the server, but our goal is to eventually drop this so you'll have a dependency-free monitoring agent that's guaranteed to work. Pretty cool, huh?

Future

"The Future, Conan?"

Yes. The future, Andy.

The future of scoutd is full of good things. Some features that we are eyeing for scoutd in the future:

  • More platforms

    Windows, Solaris, FreeBSD

  • More "instantness"

    scoutd can already fire up instant realtime charts on demand. We'll be making other things instant, like plugin installs and plugin configuration updates.

  • Completely standalone

    scoutd will be able to report system metrics without needing Ruby or any other dependencies.

  • Multiple plugin languages

    Ability to write plugins in multiple languages. Ruby, Python, Javascript, etc.

  • External metrics

    Support for sending metrics to scoutd from the network, simple shell scripts, etc.

Our new scoutd agent is key in our effort to provide the best and most useful monitoring available.

MOAR

Want to Know More?

What's under the hood? What did we encounter while building scoutd? What's our build process? What's the airspeed velocity of an unladen swallow? Glad you asked. We'll dig deep into scoutd in the coming weeks and talk about the details of building scoutd for MOAR.

 

The year at Scout - 2014 edition

By Derek Bullet_white Posted in Updates Bullet_white Comments Comments

2014 was a year of major updates to Scout. Some stats on what’s been a fun year:

infographic

Product Highlights

  • Scout Realtime - In January, we released our open-source standalone realtime monitoring agent. Scout Realtime was the second most popular repository on Github during its release week (trailing Popcorn - we'll take the runner-up spot behind free movie streaming).
  • New Server View UI - In February, we released our new d3-powered server view interface and introduced automatic process monitoring. We think there's no better single-page view of your server's health.
  • New Dashboards UI - July brought our new dashboards UI. From quick ad-hoc charts to a persistent display on an external monitor, we think there's no better way to view your key metrics than your new dashboards.
  • New API - in September we debuted our new RESTful API.
  • New Realtime Charts BETA - in December we announced our new realtime charts experience. Viewing every-second-updating charts has never been easier.

Blog Post Highlights

2015

Thanks for all of your support, feedback, and hard-earned money in 2014. Our mission of lightweight, non-enterprisey server monitoring continues next year.

 

From MySQL full-text search to Elasticsearch

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Migrating backend search technologies on a high-throughput production site is no easy task, but Vector Media Group was recently faced with this decision. With a popular client site struggling under the load of complex MySQL full-text search queries, they recently switched to Elasticsearch.

I spoke with Matt Weinberg to learn how the migration went. Was the switch to Elaticsearch worth the effort?

How did you handle search before Elasticsearch?

We created a custom search using MySQL queries and implemented it into our CMS for the project, ExpressionEngine.

What were the problems with this approach?

To support full-text search, we needed to use the MySQL MyISAM storage engine. This has major downsides, the primary one being full table locks: when a table is updated, no other changes to that table can be performed.

Our tables have considerable update activity, so this would result in sometimes-significant performance issues.

Read More →

 

BETA - "Live in the the now" with our new realtime charts

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

When your database server is under heavy load, an application server is running out of memory, or you are rolling out a major deploy, you want instant performance data. In these times, it's about the present, not the past.

We're happy to introduce in-place-on-your-dashboard, every-second-updating realtime charts.

It's a seamless transition from historical to now.

System Requirements

  • Our new scoutd monitoring agent. The new agent is a daemon (vs. running Scout via Cron).
  • Ruby 1.9.2+
  • Ubuntu 12.04+, Centos/Redhat 6+, or Fedora. We'll be adding support for more distros.

Email us for access

We're gradually rolling out the new realtime UI + scoutd to gather feedback. To try our new realtime charts, email support@scoutapp.com with your account name.

 

Restricting process CPU usage using nice, cpulimit, and cgroups

By Derek Bullet_white Posted in HowTo Bullet_white Comments Comments

juggle

The Linux kernel is an incredible circus performer, carefully juggling many processes and their resource needs to keep your server humming along. The kernel is also all about equity: when there is competition for resources, the kernel tries to distribute those resources fairly.

However, what if you've got an important process that needs priority? What about a low-priority process? Or what about limiting resources for a group of a processes?

The kernel can't determine what CPU processes are important without your help.

Most processes are started at the same priority level and the Linux kernel schedules time for each task evenly on the processor. Have a CPU intensive process that can be run at a lower priority? Then you need to tell the scheduler about it!

There are at least three ways in which you can control how much CPU time a process gets:

  • Use the nice command to manually lower the task's priority.
  • Use the cpulimit command to repeatedly pause the process so that it doesn’t exceed a certain limit.
  • Use Linux’s built-in control groups, a mechanism which tells the scheduler to limit the amount of resources available to the process.

Let's look at how these work and the pros and cons of each.

Read More →

 

Introducing Chart Markers

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

We're happy to announce our first enhancement to v2 of our API: chart markers.

Use chart markers to note significant events like production deploys, infrastructure upgrades, performance enhancements, etc.

Creating a marker is simple:

curl -X POST --data "notes=deployed production"  https://scoutapp.com/api/v2/KEY/markers

Markers are applied to all charts on a dashboard.

Adding a marker after a Capistrano Deploy

Here's an example of a simple hook to create a marker after a deploy if you are using Capistrano:

after "deploy:restart", "deploy:mark_release_via_api"

task :mark_release_via_api, hosts:"app1.acme.com" do
  run_locally %Q(curl --data "notes=deployed production" https://scoutapp.com/api/v2/API_KEY/markers)
end
 

Older posts: 1 ... 3 4 5 6 7 ... 52