No Protobuffs love? Proxy it!

" Protocol Buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. "

Riemann (previous post) runs on protobuffs.

There are a ton of clients that you can use to send data around, its awesome. Step 1 in Riemann, use protobuffs to send data. But what if you didn't have a client? What if you couldn't run a compiled application because you need other people to support it?

It's hard to handover an in-house application if it's written in a language thats unfamiliar to the supporting staff. Sometimes the best language to use is what is common knowledge to your coworkers. We have a really good Powershell force so it made sense to make something that everyone can work with and keep relevant in an ever-changing environment like "systems monitoring".

So again, when stuck with a client-less language that you desperately need to use, what the hell are you supposed to do? I mean at the end of the day, all you are doing is posting a message right? What if we had a proxy!

A great implementation of this is the Riemann http proxy. It's written in Clojure, is extremely cheap in cpu cost and we have managed to push the throughput quite substantially. It was a reasonably easy method to implement and I extended it to take batches of messages to speed up throughput. Currently a single endpoint is receiving on the up side of 10k messages per second and our stress testing that Chris did pushed to to stupid amounts. I'll edit the post later with facts!

Yes I know this isn't close to the throughput of StatsD but we needed it to be somewhat reliable. Guarantees that come with TCP are a must, you can't roll with UDP if this potentially will take over as a datasource for anomaly detection, deviation monitoring or issue alerting.

In my next post I'll go into a bit more detail about how we collect data using a similar implementation to Matthew Hodgkins powershell functions lovingly named upDog...