A Holodeck for deployments?

September 01 Bullet_white By Derek Bullet_white Posted in Development Bullet_white Comments Comments

Recently I watched the documentary miniseries When We Left Earth. A constant theme from early space flight was the huge number of unknowns. For example, before the first astronaut went into orbit, NASA had concerns humans would lose their vision and wouldn’t be able to swallow in zero gravity. Thankfully, space worked out.

In devops lingo, it was impossible to build an accurate staging environment.

At Scout, we’ve grown almost 600% since last August. With that growth, our deployments are more carefully executed. This is monitoring, so Scout needs to be reliable when we deploy changes.

One of the complications with growth is a less accurate staging environment:

  • It’s more difficult to test against production data. Pulling down the production database for quick tests isn’t possible.
  • Our staging environment resembles a lifeless room. The load and data isn’t reflective of a production system.
  • Deployments can be more than application code updates – it’s not just about proper test coverage. New releases may require tweaks to configuration files, Ruby versions, etc. The impact of those changes can be difficult to observe on a server that isn’t experiencing a production load.
  • It’s expensive to replicate a full production environment, and even then you don’t have production traffic and load.
  • We use load testing tools, but reproducing actual visitor behavior isn’t a trivial task.

Building a Holodeck Environment

Moving several centuries later in space exploration, Commander Riker and Counselor Troi are seen above brainstorming naughty ways to use the Star Trek Holodeck. And why shouldn’t they? It’s a safe place – a simulated reality.

I’d like our staging environment to feel a lot more like a Holodeck than a lifeless room. Deploying to production shouldn’t feel like putting an astronaut on the moon since we know most of the production variables.

For example, a Holodeck web server might receive the same traffic as a production server but it wouldn’t do any harm (ex: deleting data). It’s a happy place: experimentation wouldn’t be punished.

It’s something I’d pay for

Everyone has anxiety over a major new release or infrastructure change. We’re human and we’ll make mistakes. Knowing that I could run our production releases through the Holodeck first would ease deployment anxiety. It would encourage experimentation. In short, it’d give us the freedom we loved in the beginning.

If you’ve solved part of this problem, even if it’s just for your application, tell me about it!

Get notified of new posts.

Once a month, we'll deliver a finely-curated selection of optimization tips to your inbox.


comments powered by Disqus