arch bash cakephp conf dauth devops drupal foss git golang information age life linux lua mail monitoring music mysql n900 netlog openstack perf photos php productivity python thesis travel uzbl vimeo web2.0

Graphite-ng: A next-gen graphite server in Go.

I've been a graphite contributor for a while (and still am). It's a great tool for timeseries metrics. Two weeks ago I started working on Graphite-ng: it's somewhere between an early clone/rewrite, a redesign, and an experiment playground, written in Golang. The focus of my work so far is the API web server, which is a functioning prototype, it answers requests like

/render/?target=sum(scale(stats.web2,5.12),derivative(stats.web2))

I.e. it lets you retrieve your timeseries, processed by function pipelines which are setup on the fly based on a spec in your http/rest arguments. Currently it only fetches metrics from text files but I'm working on decent metrics storage as well.

There's a few reasons why I decided to start a new project from scratch:

  • With graphite, it's a whole ordeal to get all components properly installed. Deploying in production environments is annoying and even more so when you just want a graphite setup on your personal netbook.
  • the graphite development process slows contributors down a lot. A group of 3rd/4th generation maintainers jointly manages the project, but it's hard to get big changes through, because they (understandably) don't feel authoritative enough to judge those changes and the predecessors have disappeared or are too busy with other things. And also ...
  • there's a high focus on backwards compatibility, which can be a good thing, but it's hard to get rid of old design mistakes, especially when fixing unclear (but arguably broken) early design decisions (or oversights) lead to different outputs
  • Graphite suffers feature creep: it has an events system, a PNG renderer, an elaborate composer web UI, etc. There's a lot of internal code dependencies holding you back from focusing on a specific problem
  • Carbon (the metrics daemon) has a pretty hard performance and scalability ceiling. Peter's article explains this well; I think we'll need some form of rewrite. Peter suggests some solutions but they are basically workarounds for Python's shortcomings. I'm also thinking of using pypy. But last time I checked pypy just wasn't there yet.
  • I want to become a good Go programmer
Note: the Graphite project is still great, the people managing do good work, but it's fairly natural for a code base that large and complicated to end up in this situation.I'm not at all claiming graphite-ng is, or ever will be better but I need a fresh start to try some disruptive ideas, using Go means having a runtime very suited for concurrency and parallelism, you can compile the whole thing down into a single executable file, and its performance looks promising. Leaving out the non-essentials (see below) allows for an elegant and surprisingly small, hackable code base.

The API server I developed sets up a processing pipeline as directed by your query: every processing function runs in a goroutine for concurrency and the metrics flow through using Go channels. It literally compiles a program and executes it. You can add your own functions to collect, process, and return metrics by writing simple plugins.
As for timeseries storage, for now it uses simple text files, but I'm experimenting and thinking what would be the best metric store(s) that works on small scale (personal netbook install) to large scale ("I have millions of metrics that need to be distributed across nodes, the system should be HA and self-healing in failure scenarios, easily maintainable, and highly performant") and is still easy to deploy, configure and run. Candidates are whisper-go, kairosdb, my own elasticsearch experiment etc.
I won't implement rendering images, because I think client-side rendering using something like timeserieswidget is superior. I can also leave out events because anthracite already does that. There's a ton of dashboards out there (graph-explorer, descartes, etc) so that can be left out as well.

For more information, see the Graphite-ng homepage.

PS: props to Felix Geisendörfer who suggested a graphite clone in Go first, it seemed like a huge undertaking but the right thing to do, I had some time so I went for it!

Comments

Good stuff and I completely agree with you about the apparent sluggishness of updates to graphite. You have extracted a lot of features though and a part of me wonders if you shouldn't be using a different moniker. Def. a replacement of the txt file backend will be warranted if you want to see this scale up/out. Opentsdb is pretty good stuff but I'd argue that it's nearly as annoying to setup as graphite is. The key to alleviating a lot of that pain is good packaging. Either way thanks for sharing this and also for spearheading something fresh and new.
I'm hoping for an open backend to almost any storage for the time series data. I want to use Cassandra but I might also want to use elastic search for a different install.

Thanks for doing this!
You may want to check out Seriesly by Dustin Sallings for an inspiration (it is written in Go too):
https://github.com/dustin/seriesly
@Jimmie I'm actually thinking of bundling timeserieswidget to have "easy graph-ability" out of the box

@daledude yep, we're working on a pluggable backend system. most interest so far is in elasticsearch and kairosdb, which should be fairly easy to integrate

@Alexander: thanks, i actually came across that earlier, but it seems to just depend on couchstore and doesn't really bring anything new to the table for me
Completely agree that Graphite is just too complicated to deploy. This looks promising.


Name:


E-mail:


URL:


Comment:


What is the first name of the guy blogging here?


This comment form is pretty crude. Make sure mandatory fields are entered correctly.
Basic html tags (a,i,b, etc) are allowed, others are sanitized