Graphite-ng: A next-gen graphite server in Go.

I've been a graphite contributor for a while (and still am). It's a great tool for timeseries metrics. Two weeks ago I started working on Graphite-ng: it's somewhere between an early clone/rewrite, a redesign, and an experiment playground, written in Golang. The focus of my work so far is the API web server, which is a functioning prototype, it answers requests like

/render/?target=sum(scale(stats.web2,5.12),derivative(stats.web2))

I.e. it lets you retrieve your timeseries, processed by function pipelines which are setup on the fly based on a spec in your http/rest arguments. Currently it only fetches metrics from text files but I'm working on decent metrics storage as well.

There's a few reasons why I decided to start a new project from scratch:

• With graphite, it's a whole ordeal to get all components properly installed. Deploying in production environments is annoying and even more so when you just want a graphite setup on your personal netbook.
• the graphite development process slows contributors down a lot. A group of 3rd/4th generation maintainers jointly manages the project, but it's hard to get big changes through, because they (understandably) don't feel authoritative enough to judge those changes and the predecessors have disappeared or are too busy with other things. And also ...
• there's a high focus on backwards compatibility, which can be a good thing, but it's hard to get rid of old design mistakes, especially when fixing unclear (but arguably broken) early design decisions (or oversights) lead to different outputs
• Graphite suffers feature creep: it has an events system, a PNG renderer, an elaborate composer web UI, etc. There's a lot of internal code dependencies holding you back from focusing on a specific problem
• Carbon (the metrics daemon) has a pretty hard performance and scalability ceiling. Peter's article explains this well; I think we'll need some form of rewrite. Peter suggests some solutions but they are basically workarounds for Python's shortcomings. I'm also thinking of using pypy. But last time I checked pypy just wasn't there yet.
• I want to become a good Go programmer

The API server I developed sets up a processing pipeline as directed by your query: every processing function runs in a goroutine for concurrency and the metrics flow through using Go channels. It literally compiles a program and executes it. You can add your own functions to collect, process, and return metrics by writing simple plugins.
As for timeseries storage, for now it uses simple text files, but I'm experimenting and thinking what would be the best metric store(s) that works on small scale (personal netbook install) to large scale ("I have millions of metrics that need to be distributed across nodes, the system should be HA and self-healing in failure scenarios, easily maintainable, and highly performant") and is still easy to deploy, configure and run. Candidates are whisper-go, kairosdb, my own elasticsearch experiment etc.
I won't implement rendering images, because I think client-side rendering using something like timeserieswidget is superior. I can also leave out events because anthracite already does that. There's a ton of dashboards out there (graph-explorer, descartes, etc) so that can be left out as well.