Implementing Docker event monitoring from scratch
Docker's API provides a ton of functionality around containers and images - but there is a hidden secret, one that is easy to miss in the documentation: Docker's API has the capability to report host wide events! Container events
I'll take a look at how we can tap into this functionality, and how we can convert raw data into meaningful dashboards and alerts.
A Simple Spike
First, I'll take a look at Docker's monitoring events API. From the documentation, I have two options: polling or streaming. I'll use streaming with a GET /events
command. A successful 200 response returns a JSON string indicating status
id
from
& time
of the containers reporting events. Here's a sample:
{"status": "create", "id": "dfdf82bd3881","from": "ubuntu:latest", "time":1374067924}
Luckily for us, Swipely's team has already released a great docker-
After doing gem install docker-api
Looks like everything is configured correctly and I have a great starting point.
Now, I want to check for events.
What did I do here? First, I set the :read_timeout
parameter to 100 minutes. That will give us time to test events vs. timing out (default is one minute). Next, I set up a simple block of code to execute whenever an event arrives.
In a new terminal tab - let's fire up a container:
docker run -it ubuntu date
..and in our original tab - we've got events!
Just to test again - let's start a longer running container:
docker run -it ubuntu /bin/bash
..and then in yet another terminal window, shut it down.
docker stop 3e2f58981df8
How can I take this further? How about a StatsD counter? A counter's job is to collect metrics over an interval and once that interval is complete - report the count of the metrics collected. After installing the statsd-ruby
gem, I've whipped up a little script.
What's the script doing? It's basically the same thing as the IRB commands above, except line #11 is stripping out some extra information. The Docker API exec_create
exec_start
exec_create
Bringing it home
Ok, I have to come clean. There's one issue with this script. The
I'll do a quick install on my VM:
curl -Sso scout_install.sh https://scoutapm.com/scout_install.sh
sudo /bin/bash ./scout_install.sh <YOUR ACCOUNT KEY HERE>
Update my script to be executable:
chmod +x <SCRIPT_NAME>.rb
Run it, and fire off a few container events:
docker run -it ubuntu date
docker rm <CONTAINER ID>
...then go check my Scout account. Docker events are showing up!
What else is cool? I can now add those metrics to my dashboards:
What else? I can create alerts on them. The next time any of my containers decide to puke and die, I can get an SMS message about their doomed state.
So what's next? I could run this script on every one of my containers - but that's not really the Docker way. Instead, Docker recommends that we should create our own container running this script.
Wait a minute.. since I'm already running Scout and I see that they've already got a Docker image for it - I'll just update the existing Scout container with the new script.
Now, all I have to do is follow the directions for docker-scout
TL; DR
The Docker Events API gives us a lot of visibility into the workings of a Dockerized-host. With 17 lines of Ruby, StatsD, and Scout, we've got monitoring + alerting on those events.
Follow us on Twitter for more Docker and StatsD monitoring bits.