It's hard to believe the original Nodeflakes Christmas demo was launched just over a year ago—time certainly flies when you're having fun (or at least when you're busy). Christmas hacks are always good fun and usually well received so I wanted to make sure I managed to get something out this year too. Unfortunately a combination of too much work and not enough imagination (probably more of the latter) meant that December 1st 2012 came and went and I'd not started thinking about anything, let alone building anything.
This year’s demo—with added… nothing
In a time honoured tradition I decided the only thing for it was to relaunch exactly the same demo. The code powering the demo is to all intents and purposes exactly the same as a year ago, warts and all (and wow, there are plenty of them). In fact, the only real change is the addition of some metric tracking facilitated by the wonderful King-of-the-Devops combination of the StatsD daemon feeding data into Graphite. One slight regret I had about last year's demo was that I missed out on a minefield of interesting data: not the content of the tweets themselves (just in case you're watching, Twitter) but more overarching trends such as tweet volume in the run up to and shortly after Christmas, how active the spam and profanity filters were and the rate of client connects & disconnects. At a more basic level I also had no real logging in place to let me know if the Twitter stream had dropped out at any point since I last happened to be actively watching the app's server side output. After I decommissioned the demo in January of this year I made a note to revisit it at some point to think about how best to store the data in a query-friendly way which would let me render some interesting graphs—my parting thoughts were loosely formed around just dumping the tweets in a database with a corresponding timestamp so I could at least write something to analyse them later. Naturally, I forgot all about any such plans until the other day.
Graphite + StatsD = metrics, monitoring & pretty graphs
Fortunately I've learned quite a bit this year about both metrics and monitoring, thanks in no small part to the mountain-steep learning curve that was faavorite.com, amongst whose many challenges was my first foray into the world of application metrics monitoring above and beyond high level service/port monitoring. The subject is enormous and detailed discussion of it is not for this article but I highly recommend taking a look at Etsy's fantastic Measure Anything, Measure Everything blog post in which they introduce StatsD and detail the various metrics they track & the ways in which they track them (as a side note I also strongly recommend you read up on Graphite too, since a common misconception for newcomers is that they are one in the same thing. They're not—they're very different and standalone pieces of software). Once you've seen the sort of stuff you can track and how easy it is to do so you'll be flocking to the “Church of Graphs” in no time, especially once you start to realise that it's equally simple to track things like releases too.
I'll be updating this article with any interesting tidbits and screenshots gleaned from Graphite over the course of the month—or if not at the very least I'll do a follow up article in the new year briefly discussing the metrics recorded in this year's demo. You're guaranteed to see some pretty graphs thanks to Graphite, so it's probably worth checking back for that if nothing else!