We’ve added prolific plugin contributor Eric Lindvall’s latest plugins to Scout: Kestrel Overall and Kestrel Queue Monitoring. Kestrel is a simple message queue built from production needs at Twitter. Being the gentleman he is, Eric shared his experiences with Kestrel at Papertrail, a hosted log aggregation service.
As all good message queues should, it provides:
- transactional reads to prevent losing messages if a worker crashes before finishing processing a message
- durability to prevent losing messages if a queue server crashes or is stopped after messages have been sent to it
- “normal” queues (only one client sees a given message) and “fanout queues” (all clients see a given message, which many messaging systems call “topics”)
We picked Kestrel because it has bounded memory usage (it has a configuration setting to specify how many queued messages should reside in RAM, generally defaulting to 128MB), it’s small (I was able to read the entire Scala codebase in a weekend), and running on the JVM (which we have experiencing with) was a plus.