This is the blog for Scout, Enterprise-grade server monitoring without the bloat.
70+ plugins, realtime charts, alerts, and Chef/Puppet-friendly configuration.

Omnibus Tutorial: Package a standalone Ruby gem

Posted in HowTo | Comments Comments

stack o pancakes

A couple of years ago I visited Argentina. I have trouble enough pronouncing my limited English vocabulary and I don't speak Spanish, but after a bit of time, it was pretty easy to order food, buy groceries, and use a taxi. However, occasional hangups that happen during my regular life in the states would throw me out of sorts in Spanish: a taxi driver trying to explain he doesn't have enough change would send me off the rails.

Ruby is my English when it comes to writing software, so when I hit hangups installing something Ruby-related, I can usually work my way out of them. Our monitoring agent at Scout is a Ruby gem, and while most of our customers already have Ruby installed, for those that don't a seemingly small hangup to me can be frustrating for them.

Now, thanks to Omnibus, there's an easy way to distribute your Ruby gems as standalone, full-stack program. This means folks without Ruby can have as smooth of an experience with your hip new gem as a hardened Rubyist.

Here's how I've built a full-stack installer for our scout Ruby Gem.

Meet Omnibus

Omnibus is an old term for bus or wagonette, being the Latin for 'for all'.

via Wikipedia

Omnibus is a Ruby gem for creating full-stack installers - it's everything your customers have to install and configure to get your software to run. It's Chef for packages, and like Chef, it's built by the folks at Opscode. Omnibus lets you build full-stack installers that won't conflict with existing versions of software.

With Omnibus, you define a project, software, and its dependencies. Then, via Vagrant, you build the project on the VMs of your choice (Omnibus includes four flavors of Linux by default). Once the builds complete, you have platform-specific, full-stack versions of your software.

Opscode uses Omnibus as the default install method for Chef. Installing Chef is a one-liner vs. fetching dozens of Ruby dependencies on your own:

$ curl -L https://www.opscode.com/chef/install.sh | sudo bash

The shell script detects your platform, fetches the appropriate package, and installs Chef and all of its dependencies (like Ruby). Magic?

Lets build an embedded Ruby gem executable

I'm going to step through the process of building a package for a Ruby gem, Scout, with Omnibus. With this package, folks won't need to have Ruby installed to use Scout - the package will contain its own embedded Ruby.

Omnibus requires Ruby 1.9+.

Setup

Install the omnibus gem:

$ gem install omnibus

Create an Omnibus project:

$ omnibus project scout
$ cd omnibus-scout
$ bundle install

This creates an Omnibus project in the omnibus-scout directory. There are two primary configuration pieces, the project DSL (config/projects/scout.rb) and software DSLs (config/software/*.rb). The created directory is where we'll perform all of our omnibus commands (similar to how you'd perform all of your Rails-related commands inside a Rails project directory).

While a project may have multiple project DSLs, you'll just have one (I'm not sure of the use-case for multiple project DSLs - perhaps slightly different flavors of the same package?). However, its likely an Omnibus project will have several software DSLs. For example, if you were building a full-stack Ruby on Rails application, your single Rails project might contain several pieces pieces of software:

rails stack

After running bundle install, you'll have many sofware DSLs available to you via the omnibus-software gem. From Ruby to Nginx to PostgreSQL, omnibus-software contains a number of software DSLs that are likely to be a part of your full-stack package.

Lets start with the software DSL.

A software DSL for our Ruby gem

I'll start by creating a config/software/scout.rb DSL. Scout is a Ruby gem, so its full-stack is:

  • Ruby
  • Rubygems
  • Scout Ruby Gem

omnibus-software already contains software DSLs for Ruby and Rubygems (yeh!). So, adding those is simple:

With our dependencies added, it's time to add the build instructions for installing the Scout gem:

This installs the scout gem and removes files the end user doesn't need that were built during packaging.

Project DSL

This is a simple case, so the only addition to our config/projects/scout.rb project DSL is dependency 'scout':

Build Prep

I needed to create a couple of directories that Omnibus will use:

$ sudo mkdir -p /opt/scout /var/cache/omnibus
$ sudo chown [USER] /opt/scout
$ sudo chown [USER] /var/cache/omnibus

Building our package

Now for the magic! Lets build our package:

$ omnibus build project scout

This will build the package on your development environment. Since I develop on OSX, this will build a package for OSX. Note that building a package takes a while - it took me about 25 minutes on my MacBook Air.

The package will be placed in pkg/scout-*. Before installing, I'll clear out the /opt/scout directory that Omnibus populated during the build:

$ rm -Rf /opt/scout

Then I'll install:

$ sudo pkg/scout-*.mac_os_x.10.6.8.sh

To verify it works:

$ /opt/scout/bin/scout -v
=== Scout Installation Wizard ===
You need the 40-character alphanumeric key displayed on the account page.

If you're running RVM and see an error like this when running /opt/scout/bin/scout:

/opt/scout/embedded/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find scout (>= 0) amongst...

...switch to your system Ruby.

Where's your Ruby?

$ /opt/scout/embedded/bin/ruby -v
ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-darwin10.8.0]
$ /opt/scout/embedded/bin/gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
elif (0.1.0)
io-console (0.3)
json (1.5.4)
minitest (2.5.1)
rake (0.9.2.2)
rdoc (3.9.4)
scout (5.6.9)

Now you have a full-stack Scout, completely isolated from any existing software. It has its own Ruby, Rubygems, and Scout gem that is separate from the versions already installed on my machine.

Building Packages for other platforms

There's a good chance you'll want to build packages for other platforms. You can do that! When you ran omnibus build project scout above, you built the package for your OS. Omnibus uses Vagrant to fire up VM images of other platforms and runs omnibus build project scout on each, dumping the packages to the project's pkg/ directory.

Out-of-the-box, Omnibus supports a number of platforms:

$ vagrant status
Current machine states:

ubuntu-10.04             not created (virtualbox)
ubuntu-11.04             not created (virtualbox)
ubuntu-12.04             not created (virtualbox)
centos-5                 not created (virtualbox)
centos-6                 not created (virtualbox)

You're free to modify the Vagrant file to add or remove platforms.

Lets say you want to build a scout package for Ubuntu 10.04:

$ vagrant up ubuntu-10.04

This creates a Debian package, placing it in:

pkg/scout_*.ubuntu.10.04_amd64.deb

I've put this package on Dropbox, so you could install it on your Ubuntu 10 server:

$ wget https://dl.dropboxusercontent.com/u/468982/scout_0.0.0%2B20130617203100-1.ubuntu.10.04_amd64.deb
$ sudo dpkg -i scout_0.0.0+20130617203100-1.ubuntu.10.04_amd64.deb
(Reading database ... 30170 files and directories currently installed.)
Unpacking scout (from scout_0.0.0+20130617203100-1.ubuntu.10.04_amd64.deb) ...
Setting up scout (0.0.0+20130617203100-1.ubuntu.10.04) ...
Thank you for installing scout!

Again, just like building the package on your development machine, building a package on a VM takes a while.

As a Ruby developer, should I be using Omnibus to package my Ruby gems?

It likely doesn't make sense to package your Ruby gems in most cases. Our motivations:

  • We have customers that haven't worked much with Ruby and the Ruby install was intimidating.
  • The wider adoption of tools like Bundler and RVM have complicated the process of running our scout executable via Cron. A self-contained Scout can be a simpler option.
  • Our customers that have Ruby might not use it on all of their servers (for example, their database servers may not need Ruby).

Awkward Nerd Fist Bump

I experimented with Omnibus a bit before it reached 1.0 and didn't have much luck. An awkward nerd high-five to Tim Ray of Rackspace for building an initial version of omnibus-scout to get us started.

TL;DR

Omnibus lets you build full-stack installers for your software. It makes it easy for folks to install our software by avoiding dependency hell. Because it's fully-contained, it won't conflict with the existing pieces of your system. It's a great option for software that has a number of dependencies and/or isn't managed through an existing package management system.

Comments

blog comments powered by Disqus