From MySQL full-text search to Elasticsearch

By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Migrating backend search technologies on a high-throughput production site is no easy task, but Vector Media Group was recently faced with this decision. With a popular client site struggling under the load of complex MySQL full-text search queries, they recently switched to Elasticsearch.

I spoke with Matt Weinberg to learn how the migration went. Was the switch to Elaticsearch worth the effort?

How did you handle search before Elasticsearch?

We created a custom search using MySQL queries and implemented it into our CMS for the project, ExpressionEngine.

What were the problems with this approach?

To support full-text search, we needed to use the MySQL MyISAM storage engine. This has major downsides, the primary one being full table locks: when a table is updated, no other changes to that table can be performed.

Our tables have considerable update activity, so this would result in sometimes-significant performance issues.

Read More →

 

BETA - "Live in the the now" with our new realtime charts

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

When your database server is under heavy load, an application server is running out of memory, or you are rolling out a major deploy, you want instant performance data. In these times, it's about the present, not the past.

We're happy to introduce in-place-on-your-dashboard, every-second-updating realtime charts.

It's a seamless transition from historical to now.

System Requirements

  • Our new scoutd monitoring agent. The new agent is a daemon (vs. running Scout via Cron).
  • Ruby 1.9.2+
  • Ubuntu 12.04+, Centos/Redhat 6+, or Fedora. We'll be adding support for more distros.

Email us for access

We're gradually rolling out the new realtime UI + scoutd to gather feedback. To try our new realtime charts, email support@scoutapp.com with your account name.

 

Restricting process CPU usage using nice, cpulimit, and cgroups

By Derek Bullet_white Posted in HowTo Bullet_white Comments Comments

juggle

The Linux kernel is an incredible circus performer, carefully juggling many processes and their resource needs to keep your server humming along. The kernel is also all about equity: when there is competition for resources, the kernel tries to distribute those resources fairly.

However, what if you've got an important process that needs priority? What about a low-priority process? Or what about limiting resources for a group of a processes?

The kernel can't determine what CPU processes are important without your help.

Most processes are started at the same priority level and the Linux kernel schedules time for each task evenly on the processor. Have a CPU intensive process that can be run at a lower priority? Then you need to tell the scheduler about it!

There are at least three ways in which you can control how much CPU time a process gets:

  • Use the nice command to manually lower the task's priority.
  • Use the cpulimit command to repeatedly pause the process so that it doesn’t exceed a certain limit.
  • Use Linux’s built-in control groups, a mechanism which tells the scheduler to limit the amount of resources available to the process.

Let's look at how these work and the pros and cons of each.

Read More →

 

Introducing Chart Markers

By Derek Bullet_white Posted in Features Bullet_white Comments Comments

We're happy to announce our first enhancement to v2 of our API: chart markers.

Use chart markers to note significant events like production deploys, infrastructure upgrades, performance enhancements, etc.

Creating a marker is simple:

curl -X POST --data "notes=deployed production"  https://scoutapp.com/api/v2/KEY/markers

Markers are applied to all charts on a dashboard.

Adding a marker after a Capistrano Deploy

Here's an example of a simple hook to create a marker after a deploy if you are using Capistrano:

after "deploy:restart", "deploy:mark_release_via_api"

task :mark_release_via_api, hosts:"app1.acme.com" do
  run_locally %Q(curl --data "notes=deployed production" https://scoutapp.com/api/v2/API_KEY/markers)
end
 

Run your Nagios plugins with Scout (BETA)

By Derek Bullet_white Posted in Features

Many of you come to Scout from Nagios. We'd like to make the transition to Scout easier.

How about having our agent run your Nagios Plugins? To try it, SSH onto your server and run:

gem install scout --pre

Then, in your crontab entry for Scout use the --nagios flag:

* * * * * scout KEY --nagios

This will run the commands defined in your /etc/nagios/nrpe.cfg file. If your config file is somewhere else (or you only want to run a subset of commands) you can provide the path to that file:

* * * * * scout KEY --nagios /.scout/nagios.cfg

Data from Nagios plugins will show up w/each of your servers and can be placed on charts, just like any other metric in Scout.

Send your feedback to support@scoutapp.com. We'll give you a vintage Scout T-Shirt for your thoughts.

 

Understanding disk inodes

By Derek Bullet_white Posted in HowTo

You try creating a file on a server and see this error message:

No space left on device

...but you've got plenty of space:

df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1            10321208   3159012   6637908  33% /

Who is the invisible monster chewing up all of your space?

Why, the inode monster of course!

What are inodes?

An index node (or inode) contains metadata information (file size, file type, etc.) for a file system object (like a file or a directory). There is one inode per file system object.

An inode doesn't store the file contents or the name: it simply points to a specific file or directory.

Read More →

 

Older posts: 1 ... 19 20 21 22 23 ... 68