Thu, 29 May 2014

Monitorama PDX & my metrics 2.0 presentation

Earlier this month we had another iteration of the Monitorama conference, this time in Portland, Oregon.

(photo by obfuscurity)

I think the conference was better than the first one in Boston, much more to learn. Also this one was quite focused on telemetry (timeseries metrics processing), lots of talks on timeseries analytics, not so much about things like sensu or nagios. Adrian Cockroft's keynote brought some interesting ideas to the table, like building a feedback loop into the telemetry to drive infrastructure changes (something we do at Vimeo, I briefly give an example in the intro of my talk) or shortening the time from fault to alert (which I'm excited to start working on soon)
My other favorite was Noah Kantrowitz's talk about applying audio DSP techniques to timeseries, I always loved audio processing and production. Combining these two interests hadn't occurred to me so now I'm very excited about the applications.
The opposite, but just as interesting of an idea - conveying information about system state as an audio stream - came up in puppetlab's monitorama recap and that seems to make a lot of sense as well. There's a lot of information in a stream of sound, it is much denser than text, icons and perhaps even graph plots. Listening to an audio stream that's crafted to represent various information might be a better way to get insights into your system.

I'm happy to see the idea reinforced that telemetry is a key part of modern monitoring. For me personally, telemetry (the tech and the process) is the most fascinating part of modern technical operations, and I'm glad to be part of the movement pushing this forward. There's also a bunch of startups in the space (many stealthy ones), validating the market. I'm curious to see how this will play out.

I had the privilege to present metrics 2.0 and Graph-Explorer. As usual the slides are on slideshare and the footage on the better video sharing platform ;-) .
I'm happy with all the positive feedback, although I'm not aware yet of other tools and applications adopting metrics 2.0, and I'm looking forward to see some more of that, because ultimately that's what will show if my ideas are any good.

Sun, 18 May 2014

On Graphite, Whisper and InfluxDB

Graphite, and the storage Achilles heel

Graphite is a neat timeseries metrics storage system that comes with a powerful querying api, mainly due to the whole bunch of available processing functions.
For medium to large setups, the storage aspect quickly becomes a pain point. Whisper, the default graphite storage format, is a simple storage format, using one file per metric (timeseries).
  • It can't keep all file descriptors in memory so there's a lot of overhead in constantly opening, seeking, and closing files, especially since usually one write comes in for all metrics at the same time.
  • Using the rollups feature (different data resolutions based on age) causes a lot of extra IO.
  • The format is also simply not optimized for writes. Carbon, the storage agent that sits in front of whisper has a feature to batch up writes to files to make them more sequential but this doesn't seem to help much.
  • Worse, due to various implementation details the carbon agent is surprisingly inefficient and cpu-bound. People often run into cpu limitations before they hit the io bottleneck. Once the writeback queue hits a certain size, carbon will blow up.
Common recommendations are to run multiple carbon agents and running graphite on SSD drives.
If you want to scale out across multiple systems, you can get carbon to shard metrics across multiple nodes, but the complexity can get out of hand and manually maintaining a cluster where nodes get added, fail, get phased out, need recovery, etc involves a lot of manual labor even though carbonate makes this easier. This is a path I simply don't want to go down.

These might be reasonable solutions based on the circumstances (often based on short-term local gains), but I believe as a community we should solve the problem at its root, so that everyone can reap the long term benefits.

In particular, running Ceres instead of whisper, is only a slight improvement, that suffers from most of the same problems. I don't see any good reason to keep working on Ceres, other than perhaps that it's a fun exercise. This probably explains the slow pace of development.
However, many mistakenly believe Ceres is "the future".
Switching to LevelDB seems much more sensible but IMHO still doesn't cut it as a general purpose, scalable solution.

The ideal backend

I believe we can build a backend for graphite that
  • can easily scale from a few metrics on my laptop in power-save mode to millions of metrics on a highly loaded cluster
  • supports nodes joining and leaving at runtime and automatically balancing the load across them
  • assures high availability and heals itself in case of disk or node failures
  • is simple to deploy. think: just run an executable that knows which directories it can use for storage, elasticsearch-style automatic clustering, etc.
  • has the right read/write optimizations. I've never seen a graphite system that is not write-focused, so something like LSM trees seems to make a lot of sense.
  • can leverage cpu resources (e.g. for compression)
  • provides a more natural model for phasing out data. Optional, runtime-changeable rollups. And an age limit (possibly, but not necessarily round robin)
While we're at it. pub-sub for realtime analytics would be nice too. Especially when it allows to use the same functions as the query api.
And getting rid of the metric name restrictions such as inability to use dots or slashes.


There's a lot of databases that you could hook up to graphite. riak, hdfs based (opentsdb), Cassandra based (kairosdb, blueflood, cyanite), etc. Some of these are solid and production ready, and would make sense depending on what you already have and have experience with. I'm personally very interested in playing with Riak, but decided to choose InfluxDB as my first victim.

InfluxDB is a young project that will need time to build maturity, but is on track to meet all my goals very well. In particular, installing it is a breeze (no dependencies), it's specifically built for timeseries (not based on a general purpose database), which allows them to do a bunch of simplifications and optimizations, is write-optimized, and should meet my goals for scalability, performance, and availability well. And they're in NYC so meeting up for lunch has proven to be pretty fruitful for both parties. I'm pretty confident that these guys can pull off something big.

Technically, InfluxDB is a "timeseries, metrics, and analytics" databases with use cases well beyond graphite and even technical operations. Like the alternative databases, graphite-like behaviors such as rollups management and automatically picking the series in the most appropriate resolutions, is something to be implemented on top of it. Although you never know, it might end up being natively supported.

Graphite + InfluxDB

InfluxDB developers plan to implement a whole bunch of processing functions (akin to graphite, except they can do locality optimizations) and add a dashboard that talks to InfluxDB natively (or use Grafana), which means at some point you could completely swap graphite for InfluxDB. However, I think for quite a while, the ability to use the Graphite api, combine backends, and use various graphite dashboards is still very useful. So here's how my setup currently works:
  • carbon-relay-ng is a carbon relay in Go. It's a pretty nifty program to partition and manage carbon metrics streams. I use it in front of our traditional graphite system, and have it stream - in realtime - a copy of a subset of our metrics into InfluxDB. This way I basically have our unaltered Graphite system, and in parallel to it, InfluxDB, containing a subset of the same data.
    With a bit more work it will be a high performance alternative to the python carbon relay, allowing you to manage your streams on the fly. It doesn't support consistent hashing, because CH should be part of a strategy of a highly available storage system (see requirements above), using CH in the relay still results in a poor storage system, so there's no need for it.
  • I contributed the code to InfluxDB to make it listen on the carbon protocol. So basically, for the purpose of ingestion, InfluxDB can look and act just like a graphite server. Anything that can write to graphite, can now write to InfluxDB. (assuming the plain-text protocol, it doesn't support the pickle protocol, which I think is a thing to avoid anyway because almost nothing supports it and you can't debug what's going on)
  • graphite-api is a fork/clone of graphite-web, stripped of needless dependencies, stripped of the composer. It's conceived for many of the same reasons behind graphite-ng (graphite technical debt, slow development pace, etc) though it doesn't go to such extreme lengths and for now focuses on being a robust alternative for the graphite server, api-compatible, trivial to install and with a faster pace of development.
  • That's where graphite-influxdb comes in. It hooks InfluxDB into graphite-api, so that you can query the graphite api, but using data in InfluxDB. It should also work with the regular graphite, though I've never tried. (I have no incentive to bother with that, because I don't use the composer. And I think it makes more sense to move the composer into a separate project anyway).
With all these parts in place, I can run our dashboards next to each other - one running on graphite with whisper, one on graphite-api with InfluxDB - and simply look whether the returned data matches up, and which dashboards loads graphs faster. Later i might do more extensive benchmarking and acceptance testing.

If all goes well, I can make carbon-relay-ng fully mirror all data, make graphite-api/InfluxDB the primary, and turn our old graphite box into a live "backup". We'll need to come up with something for rollups and deletions of old data (although it looks like by itself influx is already more storage efficient than whisper too), and I'm really looking forward to the InfluxDB team building out the function api, having the same function api available for historical querying as well as realtime pub-sub. (my goal used to be implementing this in graphite-ng and/or carbon-relay-ng, but if they do this well, I might just abandon graphite-ng)

To be continued..

Wed, 23 Apr 2014

Metrics 2.0 now has its own website!

Metrics 2.0 started as a half-formal proposal and an implementation via graph-explorer, but is broad enough in scope that it deserves its own website, its own spec, its own community. That's why I launched and a discussion group.

from the website:

We have pretty good storage of timeseries data, collection agents, and dashboards. But the idea of giving timeseries a "name" or a "key" is profoundly limiting us. Especially when they're not standardized and missing information.
Metrics 2.0 aims for self-describing, standardized metrics using orthogonal tags for every dimension. "metrics" being the pieces of information that point to, and describe timeseries of data.

By adopting metrics 2.0 you can:
  • increase compatibility between tools
  • get immediate understanding of metrics
  • build graphs, plots, dashboards and alerting expressions with minimal hassle

Sun, 23 Feb 2014

Introduction talk to metrics 2.0 and Graph-Explorer

This week I had the opportunity to present metrics 2.0 and Graph-Explorer at the Full-stack engineering meetup.
I could easily talk for hours about this stuff but the talk had to be about 20 minutes so I paraphrased it to be only about the basic concepts and ideas and some practical use cases and features. I think it serves as a pretty good introduction and a showcase of the most commonly used features (graph composition, aggregations and unit conversion), and some new stuff such as alerting and dashboards.

The talk also briefly covers native metrics 2.0 through your metrics pipeline using statsdaemon and carbon-tagger. I'm psyched that by formatting metrics at the source a little better and having an aggregation daemon that expresses the performed operations by updating the metric tags, all the foundations are in place for some truly next-gen UI's and applications (one of them already being implemented: graph-explorer can pretty much generate all graphs I need by just phrasing an information need as a proper query)

The video and slides are available and also embedded below.

I would love to do another talk that allows me to dive into more of the underlying ideas, the benefits of metrics2.0 for things like metric storage systems, graph renderers, anomaly detection, dashboards, etc.

Hope you like it!

Mon, 30 Dec 2013

Pixie: simple photo management using directory layouts and tags.

So you have a few devices with pictures, and maybe some additional pictures your friends sent you. You have a lot of pictures of the same thing and probably too high of a resolution. Some may require some editing. How do you easily create photo albums out of this mess? And how do you do it in a way that keeps a simple and elegant, yet flexible file/directory layout for portability and simplicity?

I couldn't find a tool that I liked, so I rolled my own: Pixie.
It gives you vim-like keybindings to navigate pictures, create edits (stored in a "mirror directory") and add/remove tags to pictures. (Because n:m tag relationships are basically a must for organizing things and don't work on any common filesystem) To generate "album" directories, just define which tags they should(n't) match and run the script that synchronizes an export directory by symlinking (or resizing) the correct files into them. Note the source directory stays unaltered so you can easily keep syncing with devices and/or people.
What used to be a pain in the butt for me is now a pretty pleasant experience.
Does this workflow make sense to you? Is this useful to you? Why (not) ?

Sun, 22 Dec 2013

Vimeo holiday special & other great videos

We (the Vimeo Staff) just released the 2013 Vimeo Holiday Special, embedded below.
(and I have a line in it! "Who's behind this?")

Here's last year's Holiday video which is just as cool.

Not a lot of people know this but we actually produce a bunch of our own videos.

Some of my favorites include Auto-tuning, Idea, Vimeo goes bowling, Vimeo production reel 2012, Open your eyes to the new vimeo and Brad!.

While I'm at it... Over the last 2 years I've discovered many great videos from our diverse user base through staff screenings and by just browsing around. I've narrowed it down to this list of all-time favorites:
To this day, The Art of Making, Alma Flamenca, Record Breaker, Malaria, Portal: Terminal velocity (must-see for all half-life/portal fans), The arctic light, iPhone 4s // Thailand, Ending Overfishing.
I hope you have some time to check out this amazing content! (protip: watch later button & couch mode / smart tv vimeo app)

Sat, 14 Sep 2013

Metrics 2.0: a proposal

  • Graphite's metrics are strings comprised of dot-separated nodes which, due to their ordering, can be represented as a tree. Many other places use a similar format (stats in /proc etc).
  • OpenTSDB's metrics are shorter, because they move some of the dimensions (server, etc) into key-value tags.
I think we can do better...
I think our metrics format is restrictive and we do our self a disservice using it:
  • the metrics are not fully self-describing (they need additional information to become useful for interpretation, e.g. a unit, and information about what they mean)
  • they impose needless limitations (no dots allowed, strict ordering)
  • aggregators have no way to systematically and consistently express the operation they performed
  • we can't include additional information because that would create a new metric ID
  • we run into scale/display/correctness problems because storage rollups, rendering consolidation, as well as aggregation api functions need to be aware of how the metric is to be aggregated, and this information is often lacking or incorrect.
  • there's no consistency into what information goes where or how it's called (which field is the one that has the hostname?)
We can solve all of this. In an elegant way, even!

Over the past year, I've been working a lot on a graphite dashboard (Graph-Explorer). It's fairly unique in the sense that it leverages what I call "structured metrics" to facilitate a powerful way to build graphs dynamically. (see also "a few common graphite problems and how they are already solved" and the Graph-Explorer homepage). I implemented two solutions. ( structured_metrics, carbon-tagger) for creating, transporting and using these metrics on which dashboards like Graph-Explorer can rely. The former converts graphite metrics into a set of key-value pairs (this is "after the fact" so a little annoying), carbon-tagger uses a prototype extended carbon (graphite) protocol (called "proto2") to maintain an elasticsearch tags database on the fly, but the format proved too restrictive.

Now for Graphite-NG I wanted to rethink the metric, taking the simplicity of Graphite, the ideas of structured_metrics and carbon_tagger (but less annoying) and OpenTSDB (but more powerful), and propose a new way to identify and use metrics and organize tags around them.

The proposed ingestion (carbon etc) protocol looks like this:

<intrinsic_tags>  <extrinsic_tags> <value> <timestamp>
  • intrinsic_tags and extrinsic_tags are space-separated strings, those strings can be a regular value, or a key=value pair and they describe one dimension (aspect) of the metric.
    • an intrinsic tag contributes to the identity of the metric. If this section changes, we get a new metric
    • an extrinsic tag provides additional information about the metric. changes in this set doesn't change the metric identity
    Internally, the metric ID is nothing more than the string of intrinsic tags you provided (similar to Graphite style). When defining a new metric, write down the intrinsic tags/nodes (like you would do with Graphite except you can use any order), and then keep 'em in that order (to keep the same ID). The ordering does not affect your ability to work with the metrics in any way. The backend indexes the tags and you'll usually rely on those to work with the metrics, and rarely with the ID itself.
  • A metric in its basic form (without extrinsic tags) would look like:
    graphite: stats.gauges.db15.mysql.bytes_received
    opentsdb: mysql.bytes_received host=db15
    proposal: service=mysql server=db15 direction=in unit=B
  • key=val tags are most useful: the key is a name of the aspect, which allows you to express "I want to see my metrics averaged/summed/.. by this aspect, in different graphs based on another aspect, etc" (e.g. GEQL statements), and because you can use them for filtering, so I highly recommend to take some time to come up with a good key for every tag. However sometimes it can be hard to come up with a concise, but descriptive key for a tag, hence they are not mandatory. For regular words without a key, the backend will assign dummy keys ('n1', 'n2', etc) to facilitate those features without hassle. Note the two spaces between intrinsic and extrinsic. With extrinsic tags the example could look like:
    service=mysql server=db15 direction=in unit=B  src=diamond processed_by_statsd env=prod
  • Tags can contain any character except whitespace and null. Specifically: dots (great for metrics that contain an ip, histogram bin, a percentile, etc) and slashes (unit can be 'B/s' too)
  • the unit tag is mandatory. It allows dashboards to show the proper label on the Y-axis, and to do conversions (for example in Graph Explorer, if your metric is an amount of B used on a disk drive, and you request the increase in GB per day. it will automatically convert (and derive) the data). We should aim for standardization of units. I maintain a table of standardized units & prefixes which uses SI and IEC as starting point, and extends it with units commonly used in IT.

Further thoughts/comparisons


  • The concept of extrinsic tags is something I haven't seen before, but I think it makes a lot of sense because we often want to pass extra information but we couldn't because it would create a new metric. It also makes the metrics more self-describing.
  • Sometimes it makes sense for tags only to apply to a metric in a certain time frame. For intrinsic tags this is already the case by design, for extrinsic tags the database could maintain a from/to timestamp based on when it saw the tag for the given metric
  • metric finding: besides the obvious left-to-right auto-complete and pattern matching (which allows searching for substrings in any order), we can now also build an interface that uses facet searches to suggest/auto-complete tags, and filter by toggling on/off tags.
  • Daemons that sit in the pipeline and yield aggregated/computed metrics can do this in a much more useful way. For example a statsd daemon that computes a rate per second for metrics with 'unit=B' can yield a metric with 'unit=B/s'.
  • We can standardize tag keys and values, other than just the unit tag. Beyond the obvious compatibility benefits between tools, imagine querying for:
    • 'unit=(Err|Warn)' and getting all errors and warnings across the entire infrastructure (no matter which tool generated the metric), and grouping/drilling down by tags
    • '$hostname direction=in' and seeing everything coming in to the server, network traffic on the NIC, as well as files being uploaded.
    Also, metrics that are summary statistics (i.e. with statsd) will get intrinsic tags like 'stat=median' or 'stat=upper_90'. This has three fantastic consequences:
    • aggregation (rollups from higher to lower resolution) knows what to do without configurating aggregation functions, because it can be deduced from the metric itself
    • renderers that have to render >1 datapoints per pixel, will produce more accurate, relevant graphs because they can deduce what the metric is meant to represent
    • API functions such as "cumulative", "summarize" and "smartSummarize" don't need to be configured with an explicit aggregation function

From the Graphite perspective, specifically

  • dots and slashes are allowed
  • The tree model is slow to navigate, sometimes hard to find your metrics, and makes it really hard to write queries (target statements) that need metrics in different locations of the tree (because the only way to retrieve multiple metrics in a target is wildcards)
  • The tree model causes people to obsess over node ordering to find the optimal tree organization, but no ordering allows all query use cases anyway, so you'll be limited no matter how much time you spend organising metrics.
  • We do away with the tree entirely. A multi-dimensional tag database is way more powerfuland allows for great "metric finding" (see above)
  • Graphite has no tags support
  • you don't need to configure aggregation functions anymore, less room for errors ("help, my scale is wrong when i zoom out"), better rendering
  • when using statsd, you don't need prefixes like "stats.". In fact that whole prefix/postfix/namespacing thing becomes moot

From the OpenTSDB perspective, specifically

  • allow dots anywhere
  • 'http.hits' becomes 'http unit=Req' (or 'unit=Req http', as long as you pick one and stick with it)
  • probably more, I'm not very familiar with it

From the structured_metrics/carbon-tagger perspective, specifically

  • not every tag requires a key (on metric input), but you still get the same benefits
  • You're not forced to order the tags in any way
  • sometimes relying on the 'plugin' tag is convenient but it's not really intrinsic to the metric, now we can use it as extrinsic tag

Backwards compatibility

  • Sometimes you merely want the ability to copy a "metric id" from the app source code, and paste it in a "/render/?target=" url to see that particular metric. You can still do this: copy the intrinsic tags string and you have the id.
  • if you wanted to, you could easily stay compatible with the official graphite protocol: for incoming metrics, add a 'unit=Unknown' tag and optionally turn the dots into spaces (so that every node becomes a tag), so you can mix old-style and new style metrics in the same system.

Sat, 07 Sep 2013

Graphite-ng: A next-gen graphite server in Go.

I've been a graphite contributor for a while (and still am). It's a great tool for timeseries metrics. Two weeks ago I started working on Graphite-ng: it's somewhere between an early clone/rewrite, a redesign, and an experiment playground, written in Golang. The focus of my work so far is the API web server, which is a functioning prototype, it answers requests like


I.e. it lets you retrieve your timeseries, processed by function pipelines which are setup on the fly based on a spec in your http/rest arguments. Currently it only fetches metrics from text files but I'm working on decent metrics storage as well.

There's a few reasons why I decided to start a new project from scratch:

  • With graphite, it's a whole ordeal to get all components properly installed. Deploying in production environments is annoying and even more so when you just want a graphite setup on your personal netbook.
  • the graphite development process slows contributors down a lot. A group of 3rd/4th generation maintainers jointly manages the project, but it's hard to get big changes through, because they (understandably) don't feel authoritative enough to judge those changes and the predecessors have disappeared or are too busy with other things. And also ...
  • there's a high focus on backwards compatibility, which can be a good thing, but it's hard to get rid of old design mistakes, especially when fixing unclear (but arguably broken) early design decisions (or oversights) lead to different outputs
  • Graphite suffers feature creep: it has an events system, a PNG renderer, an elaborate composer web UI, etc. There's a lot of internal code dependencies holding you back from focusing on a specific problem
  • Carbon (the metrics daemon) has a pretty hard performance and scalability ceiling. Peter's article explains this well; I think we'll need some form of rewrite. Peter suggests some solutions but they are basically workarounds for Python's shortcomings. I'm also thinking of using pypy. But last time I checked pypy just wasn't there yet.
  • I want to become a good Go programmer
Note: the Graphite project is still great, the people managing do good work, but it's fairly natural for a code base that large and complicated to end up in this situation.I'm not at all claiming graphite-ng is, or ever will be better but I need a fresh start to try some disruptive ideas, using Go means having a runtime very suited for concurrency and parallelism, you can compile the whole thing down into a single executable file, and its performance looks promising. Leaving out the non-essentials (see below) allows for an elegant and surprisingly small, hackable code base.

The API server I developed sets up a processing pipeline as directed by your query: every processing function runs in a goroutine for concurrency and the metrics flow through using Go channels. It literally compiles a program and executes it. You can add your own functions to collect, process, and return metrics by writing simple plugins.
As for timeseries storage, for now it uses simple text files, but I'm experimenting and thinking what would be the best metric store(s) that works on small scale (personal netbook install) to large scale ("I have millions of metrics that need to be distributed across nodes, the system should be HA and self-healing in failure scenarios, easily maintainable, and highly performant") and is still easy to deploy, configure and run. Candidates are whisper-go, kairosdb, my own elasticsearch experiment etc.
I won't implement rendering images, because I think client-side rendering using something like timeserieswidget is superior. I can also leave out events because anthracite already does that. There's a ton of dashboards out there (graph-explorer, descartes, etc) so that can be left out as well.

For more information, see the Graphite-ng homepage.

PS: props to Felix Geisendörfer who suggested a graphite clone in Go first, it seemed like a huge undertaking but the right thing to do, I had some time so I went for it!

Thu, 04 Apr 2013

A few common graphite problems and how they are already solved.

metrics often seem to lack details, such as units and metric types

looking at a metric name, it's often hard to know
  • the unit a metric is measured in (bits, queries per second, jiffies, etc)
  • the "type" (a rate, an ever increasing counter, gauge, etc)
  • the scale/prefix (absolute, relative, percentage, mega, milli, etc)
structured_metrics solves this by adding these tags to graphite metrics:
  • what
    what is being measured?: bytes, queries, timeouts, jobs, etc
  • target_type
    must be one of the existing clearly defined target_types (count, rate, counter, gauge)
    These match statsd metric types (i.e. rate is per second, count is per flushInterval)
In Graph-Explorer these tags are mandatory, so that it can show the unit along with the prefix (i.e. 'Gb/s') on the axis.
This will also allow you to request graphs in a different unit and the dashboard will know how to convert (say, Mbps to GB/day)

tree navigation/querying is cumbersome, metrics search is broken. How do I organize the tree anyway?

the tree is a simplistic model. There is simply too much dimensionality that can't be expressed in a flat tree. There's no way you can organize it so that will it satisfy all later needs. A tag space like structured_metrics makes it obsolete. with Graph-Explorer you can do (full-text) search on metric name, by any of their tags, and/or by added metadata. So practically you can filter by things like server, service, unit (e.g. anything expressed in bits/bytes per second, or anything denoting errors). All this irrespective of the source of a metric or the "location in the tree".

no interactivity with graphs

timeserieswidget allows you to easily add interactive graphite graph objects to html pages. You get modern features like togglable/reorderable metrics, realtime switching between lines/stacked, information popups on hoover, highlighting, smoothing, and (WIP) realtime zooming. It has a canvas (flot) and svg (rickshaw/d3) backend. So it basically provides a simpler api to use these libraries specifically with graphite.
There's a bunch of different graphite dashboards with different takes on graph composition/configuration and workflow, but the actual rendering of graphs usually comes down to plotting some graphite targets with a legend. timeserieswidget aims to be a drop-in plugin that brings all modern features so that different dashboards can benefit from a common, shared codebase, because static PNGs are a thing from the past


events lack text annotations, they are simplistic and badly supported

Graphite is a great system for time series metrics. Not for events. metrics and events are very different things across the board. drawAsInFinite() is a bit of a hack.
  • anthracite is designed specifically to manage events.
    It brings extra features such as different submission scripts, outage annotations, various ways to see events and reports with uptime/MTTR/etc metrics.
  • timeserieswidget displays your events on graphs along with their metadata (which can be just some text or even html code).
    this is where client side rendering shines


cumbersome to compose graphs

There's basically two approaches:
  • interactive composing: with the graphite composer, you navigate through the tree and apply functions. This is painfull, dashboards like descartes and graphiti can make this easier
  • use a dashboard that uses predefined templates (gdash and others) They often impose a strict navigation path to reach pages which may or may not give you the information you need (usually less or way more)
With both approaches, you usually end up with an ever growing pile of graphs that you created and then keep for reference.
This becomes unwieldy but is useful for various use cases and needs.
However, neither approach is convenient for changing information needs.
Especially when troubleshooting, one day you might want to compare the rate of increase of open file handles on a set of specific servers to the traffic on given network switches, the next day it's something completely different.
With Graph-Explorer:
  • GE gives you a query interface on top of structured_metric's tag space. this enables a bunch of things (see above)
  • you can yield arbitrary targets for each metric, to look at the same thing from a different angle (i.e. as a rate with `derivative()` or as a daily summary), and you can of course filter by angle
  • You can group metrics into graphs by arbitrary tags (e.g. you can see bytes used of all filesystems on a graph per server, or compare servers on a graph per filesystem). This feature always results in the "wow that's really cool" every time I show it
  • GE includes 'what' and 'target_type' in the group_by tags by default so basically, if things are in a different unit (B/s vs B vs b etc) it'll put them in separate graphs (controllable in query)
  • GE automatically generates the graph title and vertical title (always showing the 'what' and the unit), and shows all metrics' extra tags. This also gives you a lot of inspiration to modify or extend your query

limited options to request a specific time range

GE's query language supports freeform `from` and `to` clauses.

Referenced projects

  • anthracite:
    event/change logging/management with a bunch of ingestion scripts and outage reports
  • timeserieswidget:
    jquery plugin to easily get highly interactive graphite graphs onto html pages (dashboards)
  • structured_metrics:
    python library to convert graphite metrics tree into a tag space with clearly defined units and target types, and arbitrary metadata.
  • graph-explorer:
    dashboard that provides a query language so you can easily compose graphs on the fly to satisfy varying information needs.
All tools are designed for integration with other tools and each other. Timeserieswidget gets data from anthracite, graphite and elasticsearch. Graph-Explorer uses structured_metrics and timeserieswidget.

Future work

There's a whole lot going on in the monitoring space, but I'd like to highlight a few things I personally want to work more on:
  • I spoke with Michael Leinartas at Monitorama (and there's also a launchpad thread). We agreed that native tags in graphite are the way forward. This will address some of the pain points I'm already fixing with structured_metrics but in a more native way. I envision submitting metrics would move from:
    stats.serverdb123.mysql.queries.selects 895 1234567890
    to something more along these lines:
    host=serverdb123 service=mysql type=select what=queries target_type=rate 895 1234567890
    host=serverdb123 service=mysql type=select unit=Queries/s 895 1234567890
    h=serverdb123 s=mysql t=select queries r 895 1234567890
  • switch Anthracite backend to ElasticSearch for native integration with logstash data (and allow you to use kibana)

Sun, 24 Mar 2013

Hi Planet Devops and Infratalk

This blog just got added to planet devops and infra-talk, so for my new readers: you might know me as Dieterbe on irc, github or twitter. Since my move from Belgium to NYC (to do backend stuff at Vimeo) I've started writing more about devops-y topics (whereas I used to write more about general hacking and arch linux release engineering and (automated) installations). I'll mention some earlier posts you might be interested in: FWIW, I'm attending Monitorama next weekend in Boston.

Mon, 21 Jan 2013

Profiling and behavior testing of processes and daemons, and Devopsdays NYC

Profiling a process run

I wanted the ability to run a given process and get
a plot of key metrics (cpu usage, memory usage, disk i/o) throughout the duration of the process run.
Something light-weight with minimal dependencies so I can easily install it on a server for a one-time need.
Couldn't find a tool for it, so I wrote profile-process
which does exactly that in <100 lines of python.

black-box behavior testing processes/daemons

I wrote simple-black-box to do this.
It runs the subject(s) in a crafted sandbox, sends input (http requests, commands, ...)
and allows to make assertions on http/statsd requests/responses, network listening state, processes running, log entries,
file existence/checksums in the VFS/swift clusters, etc.
Each test-case is a scenario.
It also can use logstash to give a centralized "distributed stack trace" when you need to debug a failure after multiple processes interacting and acting upon received messages; or to compare behavior across different scenario runs.
You can integrate this with profile-process to compare runtime behaviors across testcases/scenarios.

Simple-black-box talk @ Devopsdays NYC

I did a quick 5min talk, despite some display/timing issues it was well received. (in particular I got some really positive feedback from one person and still wonder if that was a recruiter attempting to hire me -but being shy about it...- it was quite awkward)
raw uncut video. Go to 'New York, January 18th, 2013' from 02:36:25 to 02:41:15

More random thoughts about Devopsdays NYC

  • I'm getting tired of people on stage making a big revelation out of adding an index to a database column. This happens too often at web-ops/devops conferences, it's embarrassing. But at least it's not like the "how we made our site 1000x faster"-style Velocity talks that should have been named "caching and query optimization for newbies"
  • Paperless post confirms again they got their act together and keeps us up to date with their great work. Follow them.
  • Knights of the Provisioning Round Table - Bare Metal Provisioning was mostly (to my surprise) 4 individuals presenting their solution instead of a real round-table, but (to my surprise again) they were not as similar/repetitive as I expected and the pros/cons of all solutions were compared more in depth than I dared to hope. I covered dell crowbar before and like it, though I wonder when this thing is actually gonna be reliable.
  • Dave Zwieback and John Willis gave hilarious talks
  • Tried to start an open space discussion around collaboration patterns and anti-patterns, which I think is a very interesting subject, because how individuals in a team collaborate is crucial to success, but yet very little is written about it (that I could find). I would hope we can distill the years of aggregate experience of people into concise patterns and anti-patterns and document how (not) well they work for development styles (such as agile/scrum), team size, company structure/hierarchy, roadmap/technical debt pressure, etc. And especially in light of any of these things changing, because I've found people can be very change-resistive.
  • DevOps At Obama for America & The Democratic National Committee was good, I thought it would be a rehash of what was said at Coding Forward New York City: Meet the Developers Behind the Obama Campaign but there were a bunch of interesting insights about state of the art technology in here (mostly Amazon stuff)
  • A bunch of talks where the same could have been said in half the time, or less
Random thoughts about some sponsors:
  • Librato is quite cool. It's basically how my open source tool graph-explorer would look like after finishing a bunch of TODO's, combining it with graphite, polishing it all up, and offering it as a hosted solution. I find it interesting if this is a successful business with only such a limited scope
  • Even cooler is datadog. It goes beyond just metrics and doesn't just provide hosted graphing, it provides a solution for a philosophy that aims for a centralized insight of all your operational data, related collaboration and prioritized alerts that are to the point. They get a lot of things right, the open source world has some catching up to do
Interesting that both use Cassandra and free-form tags for flexibility, validating the approach I'm taking with graph-explorer. Now Graphite could use a distributed metrics storage backend over which one can do map-reduce style jobs to gather intelligence from metrics archives (maybe based on Cassandra too?), but that's another story.

Anyway, living in NYC with its vibrant ecosystem of devops people and companies organizing plenty of meet-ups and talks on their own makes it less pressing to have an event like Devopsdays, though it was certainly a good event, thanks to the sponsors and the volunteers.

Wed, 09 Jan 2013

Graph-Explorer: A graphite dashboard unlike any other

The above sounds like a marketing phrase and I'm just as skeptical of them as you, but I feel it's in place. Not because GE is necessarily better, but it's certainly different.

In web operations, mostly when troubleshooting but also for capacity planning, I often find myself having very specific information needs from my time-series, and these information needs vary a lot over time. This usually means I need to correlate or compare things that no one anticipated. Things that relate to specific machines, specific services across machines, or a few specific metrics of which only the ops team knows how they are related and cross various scopes (application, network, system, etc).
I should have an easy way to filter metrics by any information in the metric's name or values.
I should be able to group metrics into graphs the way I want. (example: when viewing filesystem usage of servers, I should be able to group by server (one graph per server listing the filesystems, but also by mountpoint to compare servers on one graph).
I should be able -with minimal effort- to view metrics by their gauge/count value, but also by their rate of change and where appropriate, as a percentage of a maximum (like diskspace used).
It should be trivial to manipulate the graph interactively (toggling things on/off, switching between lines/stacked mode, inspecting datapoints, zooming, ...).
It should show me all events, colorcoded by type, with text description, and interactive so that it can use hyperlinks.
And most of all, the code should be as simple as possible and it should be easy to get running.

Dashboards which show specific predefined KPI's (this covers most graphite dashboards) are clearly unsuitable for this use case. Template-based "metric exploration" dashboards like cacti and ganglia are in my experience way too limited. Graph composing dashboards (like the stock graphite one, or graphiti) require much manual work to get the graph you want. I couldn't find anything even close to what I wanted, so I started Graph-Explorer.

The approach I'm taking is using plugins which add metadata to metrics (tags for server, service, mountpoint, interface name, ...), having them define how to render as a count, as a rate, as a percent of some max allowed value (or a metric containing the max), and providing a query language which allow you to match/filter metrics, group them into graphs by tag, and render them how you want them. The plugins promote standardized metric naming and reuse across organisations, not in the least because most correspond to plugins for the Diamond monitoring agent.

Furthermore, because it uses my graphitejs plugin (which now btw supports flot as a backend for fast canvas-based graphs and annotated events from anthracite) the manual interactions mentioned earlier are supported or at least on the roadmap.

Graph Explorer is not yet where I want it, but it's already a very useful tool at Vimeo.

Wed, 14 Nov 2012

Client-side rendered graphite charts for all

Client-side rendering of charts as opposed to using graphite's server side generated png's allows various interactivity features, such as:

  • interactive realtime zooming and panning of the graph, timeline sliders
  • realtime switching between various rendering modes (lines, stacked, etc)
  • toggling certain targets on/off, reordering them, highlighting their plot when hoovering over the legend, etc
  • basic data manipulation, such as smoothing to see how averages compare (akin to movingAverage, but now interactive)
  • popups detailing the exact metrics when hoovering over the chart's datapoints
  • popups for annotated events. (a good use for anthracite).

Those are all features of charting libraries such as flot and rickshaw, the only remaining work is creating a library that unleashes the power of such a framework, integrates it with the graphite api datasource, and makes it available over a simple but powerful api.

That's what I'm trying to achieve with
It's based on rickshaw.
It gives you a JavaScript api to which you specify your graphite targets and some options and it'll give you your graph with extra interactivity sauce. Note that graphite has a good and rich api, one that is widely known and understood, that's why I decided to keep it exposed and not abstract any more than needed.

There are many graphite dashboards with a different focus, but as far as plotting graphs, what they need is usually very similar: a plot to draw multiple graphite targets, a legend, an x-axis, 1 or 2 y-axis, lines or stacked bands, so I think there's a lot of value in having many dashboard projects share the same code for the actual charts, and I hope we can work together on this.

Mon, 12 Nov 2012

Anthracite, an event database to enrich monitoring dashboards and to allow visual and numerical analysis of events that have a business impact


Graphite can show events such as code deploys and puppet changes as vertical markers on your graph. With the advent of new graphite dashboards and interfaces where we can have popups and annotations to show metadata for each event (by means of client-side rendering), it's time we have a database to track all events along with categorisation and text descriptions (which can include rich text and hyperlinks). Graphite is meant for time series (metrics over time), Anthracite aims to be the companion for annotated events.
More precisely, Anthracite aims to be a database of "relevant events" (see further down), for the purpose of enriching monitoring dashboards, as well as allowing visual and numerical analysis of events that have a business impact (for the latter, see "Thoughts on incident nomenclature, severity levels and incident analysis" below)
It has a TCP receiver, a database (sqlite3), a http interface to deliver event data in many formats and a simple web frontend for humans.

design goals:

  • do one thing and do it well. aim for integration.
  • take inspiration from graphite:
    • simple TCP protocol
    • automatically create new event types as they are used
    • run on port 2005 by default (carbon is 2003,2004)
    • deliver events in various formats (html, raw, json, sqlite,...)
    • stay out of the way
  • super easy to install and run: install dependencies, clone repo. the app is ready to run
I have a working prototype on

About "relevant events"

I recommend you submit any event that has or might have a relevant effect on:

  • your application behavior
  • monitoring itself (for example you fixed a bug in metrics reporting. it shouldn't look like the app behavior changed)
  • the business (press coverage, viral videos, etc), because this also affects your app usage and metrics.

Formats and conventions

The TCP receiver listens for lines in this format:

<unix_timestamp> <type> <description>

There are no restrictions for type and description, other than that they must be non-empty strings.
I do have some suggestions which I'll demonstrate through fictive examples;
but note that there's room for improvement, see the section below)

# a deploy_* type for each project
ts "deploy e8e5e4 initiated by Nicolas --"
ts puppet "all nodes of class web_cluster_1: modified apache.conf; restart service apache"
ts incident_sev2_start "mysql2 crashed, site degraded"
ts incident_sev2_resolved "replaced db server"
ts incident "hurricane Sandy, systems unaffected but power outages among users, expect lower site usage"
# in those exceptional cases of manual production changes, try to not forget adding your event
ts manual_dieter "i have to try this firewall thing on the LB"
ts backup "backup from database slave vimeomysql22"

Thoughts on incident nomenclature, severity levels and incident analysis

Because there are so many unique and often subtle pieces of information pertaining to each individual incident, it's often hard to map an incident to a simple severity level or keyword. When displaying events as popups on graphs I think no severity levels are needed, the graphs and event descriptions are much more clear than any severity level could convey.
However, I do think these levels are very useful for reporting and numerical analysis.
On slide 53 of the metametrics slidedeck Allspaw mentions severity levels, which can be paraphrased in terms of service degradation for the end user: 1 (full), 2 (major), 3 (minor), 4 (no).
I would like to extend this notion into the opposite spectrum, and have similar levels on the positivie scale, so that they represent positive incidents (like viral videos, press mentions, ...) as opposed to problematic ones (outages).
For incident analysis we need a rich nomenclature and schema: incidents can (presumably) have a positive or negative impact, can be self-induced or not, and can be categorized with severity levels; they can also be planned for (maintenance, release announcements) or not. While we're at it, how about events to mark point in time where the cause was detected, as well as resolved, so we can calculate TTD and TTR (see Allspaw slidedeck)?
Since basically any event can have a positive or negative impact, an option is to leave out the type 'incident' and give a severity field for every event type.
I'm thinking of a good nomenclature and a schema to express all this. (btw, notice how in common ops literature the word incident is usually associated with outages and bad things; while actually an incident can just as well be a positive event) as well as UI features to support this analysis.

I need your help

The tcp receiver works, the backend works, i have a crude (but functional) web app, and a simple http api to retrieve the events in all kinds of formats. Next up are:
  • monitoring dashboard for graphite that gathers events from anthracite, can show metadata, and can mark a timeframe between start and stop events
  • plugings for puppet, chef to automatically submit their relevant events
  • a better web UI and actually provide features to do statistics on events and analysis such as TTD, TTR, with colors for severity levels etc

Wed, 07 Nov 2012

Histograms in statsd, and graphing them over time with graphite.

I submitted a pull request to statsd which adds histogram support. Example histogram, from Wikipedia
(refresher: a histogram is [a visualization of] a frequency distribution of data, paraphrasing your data by keeping frequencies for entire classes (ranges of data). histograms - Wikipedia)
It's commonly documented how to plot single histograms, that is a 2D diagram consisting of rectangles whose

  • area is proportional to the frequency of a variable
  • whose width is equal to the class interval
Class intervals go on x-axis, frequencies on y-axis.

Note: histogram class intervals are supposed to have the same width.
My implementation allows arbitrary class intervals with potentially different widths, as well as an upper boundary of infinite.

Plotting histograms.. over time

We want to plot histograms over time, and not just for a few select points in time (in which case you can just make several histograms), but a contiguous range of time, preferably through graphite's 2D graphs cause graphite is neat and common enough.
Time goes on x-axis, that's pretty much a given. So I'm trying to explore ways to visualize both class intervals as well as frequencies on the y-axis.

The example I'll use are page rendering timings, condensed into classes with upper boundaries of 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 50 and infinite seconds

Tips and notes:

  • the histogram implementation stores absolute frequencies, but it's easy to get relative frequencies in percent, like so:
  • I'll be using relative frequencies here because it normalizes the scale of the y-axis
  • In this use case each class has a notion of desirability (low render time good, high render time bad),
    I think it makes sense to use color to represent this. This extends to a lot of operational metrics which one would be using histograms for.
    (unlike non-software histograms that represent demographics or tree heights, where classes usually have nothing to do with desirability or quality).
    As it turns out, it's fairly easy to programmatically compute colors between green and red in order to have mathematically correct "steps" of color.
    However, Looks like HSV values are more suited than RGB but graphite doesn't support HSV (yet) (although one could convert HSV to RGB). Also it looks like green-purple would be a better choice for people with color blindness. I haven't gone too far in this topic.
  • Since I choose to go with color gradients, it means I better use stacked graphs, otherwise it would be too hard to distinguish which graph is what
  • None of this is restricted to timing data. The metric type under which histograms are (and should be) implemented is called "timing", which is a misleading name but we're working on renaming it.

First version

width=740&from=-24h&title=Render time histogram&
vtitle=relative frequency in %&yMax=100&
rendertime histogram
Turns out we mainly see the vast majority that performs well, simply because with this way of rendering, the higher the frequency of a class, the more prominent. Bad values are hard to see because there's not many of them, despite being more interesting. A thought I had at this point was to make all "class bands" equally wide and use a green-to-red gradient to denote the frequency values, or even just keep the current color assignments but rely on something like opacity to express frequencies. Alas, none of this is currently possible with graphite, as far as I can tell. Though I would like to explore this further. Especially because I think it wouldn't be hard to implement in graphite.

So, let's see what can be done right now.

Leaving out the smallest class

This adaption is basically the same as before, but leaves out the smallest class (which took most space), this way the other bands are a bit more visible but the effect isn't as clear as we want.
width=740&from=-24h&title=Render time histogram&
vtitle=relative frequency in %, leaving out first class&
rendertime histogram leaving out first class

Per-band scaling

Finally, the bigger the values represented by each class the more we inflate the band, so the more problematic cases become more visible, despite having a lower frequency.
width=740&from=-24h&title=Render time histogram&
vtitle=rel. freq with scale adjustment per band&
rendertime histogram with higher focus for higher class interval
I started off by scaling each band by the width of the class interval. This is actually more arbitrary than it may seem.
The point is that now it's easier to spot acute as well as long-standing problems, but note you can't really read statistics from this graph because of the per-band scaling.
Note also that outliers contribute to the outer band(s) and are given as much focus as non-outliers in the same bands. In a system over which you have no complete control (i.e. if you were graphing histograms of time until first byte or page loaded at client, where you rely on the internet as a transport) it makes sense to give less attention to outliers and focus on optimizing for as many users as possible, I think it there's no reliable way to subtract outliers from the upper bands and you should also graph averages and percentiles and understand what each graph does. But anyway here I want to include outliers, because they represent latencies we can fix.

Final notes

While the tools we have are by no means perfect, I'm seeing gradual improvement in the monitoring space. This work is only a small piece of the puzzle. The rendering of histograms can be improved but at this point I think they are good enough to be usable. The real challenge is putting in place automated trending, anomaly detection and alerting. If we can figure that out, there's less need to be looking at graphs in the first place.

Sun, 19 Aug 2012

Moving to New York City

I have a one-way ticket to NYC on Sept. 21st. Vimeo HQ is in Manhattan, and practically the whole team works in the building so it makes sense for me to relocate and join them locally. I'm looking forward to working with the colleagues face to face, but mainly I'm looking forward to the experience of living in such a different place, and exploring the US. In fact, I already have some small trips planned (Hamptons NY, camping in Pennsylvania, skiing in New York this winter) with some friends I met last year in NY. We're also looking into possibly going to California or Mexico for the holidays.
I've been living in Ghent for about a year now and absolutely loved it. I'll definitely miss Ghent, I think NYC might be a bit too hectic for my taste, but I just have to try this and see :) There's so much to learn and explore. I especially have high expectations of the music scene, which I'ld like to get involved in. (btw my ex-band is recording an EP and looking for a new drummer)

As for things in Belgium: selling furniture, moving out of my apartment next weekend, having a goodbye drink friday sept. 14th, flying a week later and returning in the summer 2013 for the gentse feesten. Maybe I'll visit sooner, depending on how homesick I get :)

Sat, 21 Jul 2012

Resigning as Arch Linux developer

A few days ago, I resigned as Arch Linux developer.
I'm sad to go, but I felt like my work on Arch became a drag, so it was time I officialized my decreased interest. The Releng team we started more than 3 years ago is now dead, but other developers are showing interest in iso building and installer scripts, so as long as they don't burn out, you'll see new isos again. More information in my resignation mail linked above.


Wed, 02 May 2012

Dell crowbar openstack swift

Learned about Dell Crowbar the other day. It seems to be (becoming) a tool I've wanted for quite a while, because it takes automating physical infrastructure to a new level, and is also convenient on virtual.

::Read more

Tue, 27 Mar 2012

Joining Vimeo

Working on scalable information retrieval systems at the university of Ghent has been very fun: interesting and challenging work, smart team, and an environment that fosters growth and innovation. I could definitely see myself continuing there...
However, Vimeo got in touch and told me about their plans... specifically what's going into the new version and what other stuff they have on their roadmap. I can honestly say vimeo is the most beautiful web property I've ever seen [*], not just that, they also provide a top product/service, and host a great community of passionate people who create some of the most beautiful online videos I've ever seen. (examples: Vietnam travel report, Sabian cymbals taking advertising to a whole new level in this video with my musician hero Mike Portnoy, a city time lapse video)
From what I can tell, they also do product management well: they know what their territory is, and how to cultivate it through stellar community management. They are not a general purpose video site and hence do not compete directly with YouTube or Facebook.

And now, I have the opportunity to be a part of that. After much pondering I decided to go for it. Resigning at the university was hard but smooth, I felt I had to take this chance and they were very supportive.
I'll be working on the infrastructure/backend side of things, I'm actually working on transcoding infrastructure right now. Working from my place in Ghent, a move to NYC at some point in the future might happen, but we'll see...

[*] When they told me the new version would be more appealing than the old, I couldn't believe that's possible. but to my own surprise they succeeded.

Sun, 25 Mar 2012

Lighttpd socket Arch Linux /var/run tmpfs tmpfiles.d

On Arch Linux, and probably many other distros /run is a new tmpfs, and /var/run symlinks to it. With Lighttpd you might have a fastcgi socket defined something like "/var/run/lighttpd/sockets/mywebsite.sock". This won't work anymore as after each reboot /var/run is an empty directory and lighttpd won't start, /var/log/lighttpd/error.log will tell you:
2012-03-16 09:21:34: (log.c.166) server started 
2012-03-16 09:21:34: (mod_fastcgi.c.977) bind failed for: unix:/var/run/lighttpd/sockets/mywebsite.sock-0 No such file or directory 
2012-03-16 09:21:34: (mod_fastcgi.c.1397) [ERROR]: spawning fcgi failed. 
2012-03-16 09:21:34: (server.c.945) Configuration of plugins failed. Going down.
That's where this new tool tmpfiles.d comes in. It creates files and directories as described in the configs, and gets invoked on boot. Like so:
$ cat /etc/tmpfiles.d/lighttpd.conf 
d /run/lighttpd/sockets 0700 http http

Sun, 08 Jan 2012

Thailand, Berlin Velocity EU, NYC, Ghent and more metal

I've been meaning to write about a lot of stuff in separate posts, but they kept getting delayed, so I'll just briefly share everything in one post.


In July I did a 3-week group journey through Thailand arranged by Explorado, but organized by ("outsourced to") the 2 guides of roots of Asia, who did an amazing job. The whole concept was exploring "the real Thailand" by means of Eco-tourism. We've been in Bangkok (twice), Chiang Mai city, a mountain village in the province of Chiang Mai, through the jungle, at a fisherman village in Phuket and at the tropical island of Koh Pha ngan. The latter was at the end of the trip and was timed perfectly to get some deserved rest in a more touristy (although not too busy) area, but the majority of the trip was spent far away from the typical touristy areas so we could be submerged in honest, authentic Thai culture and visit authentic locations, often we were at locations where seeing a group of white folks is not common. We've been at unique authentic temples, stayed with various locals and hill tribes, shared meals with them, took the same transport they did (at one point an entire village collected their bikes so we could borrow them to do a bike trip through rice fields and some of the most beautiful lakes I've ever seen). We've had plenty of beautiful moments during those 3 weeks. Like visiting the home of one of the local Thai who built his entire house out of clay, by himself and some friends, or visiting the ancient temple where our guide got married, in a forest in the hills, it was the most beautiful temple of the entire trip, but no other tourists go there because it's not really known (and should probably be kept that way). Or going to a bar in Chiang Mai city (one evening on my own, the next I brought a fellow traveler) to have some good times with some locals. The Eco-conscious part of the travel means:
  • green travel (minimize impact on the environment, "leave no trace"). Other than taking the plane over there and back we did a pretty good job, we've used public buses, night trains, biodegradable soap, etc
  • local foods (no import, minimal packaging, wrap in banana leaves, etc)
  • supporting Eco-conscious projects (like elephant nature park, which is an entire volunteer-based reserve to give mistreated elephants (which has been a big problem in Thailand) a better life, where we washed and fed elephants)
This has been a great experience, and although I found the culture in the South disgustingly based on profiting from tourists, and the cities are too polluted and dirty, I've seen cultures so respectful of nature and each other, living by values I've been trying to apply at home - but being frowned upon in our western society because we're so brainwashed by consumerism, which was beautiful and heartwarming.

Photo album


Berlin Velocity EU conference Berlin Reichstag building I've been in Berlin for the first Velocity conference in the EU, which was quite good. The best part was probably the "Velocity Birds of feather" (whatever that means) unconference the day before at betahaus, which was great for meeting some folks such as the guys (which BTW, is the site we host our music on), although lots more interesting folks attended the conference itself (and it was packed). Berlin itself was nice too. Lots of history (Berlin wall, world war(s)), lots of impressive architecture (old and new), very cheap (albeit mediocre in quality) food, lots of Italian food, a bit cold though.

New York city

Brooklyn bridge Manhattan harbor I'm still recovering from the awesome time I just had in NYC. I've been way more busy over there than I anticipated. I should have stayed 2 or 3 weeks instead of 1 :). I've met various locals (one of whom who'd love to become a city guide as 2nd job because she just loves showing people around, so that just turned out great!). I didn't go for the typical touristy things (I skipped things like the WTC memorial, empire state building, statue of liberty, to the extent you can skip them, as they are very visible from pretty much all over the place). Instead, I wanted to get a feel of the real city and the people inhabiting it. I've seen parts of Queens, central and North-West Brooklyn, lots of areas (but not enough) in Manhattan and even Staten Island, been to a rock concert, comedy, improv and cabaret shows, the movies, more bars than I can count and mostly ate out with company (just as real new yorkers do, of course, though for breakfast that feels a bit weird). I even went shopping (not mall-shopping, but groceries in the supermarket, the Williamsburg Foodtown - that's what it's called - clerk advised me to enjoy every last second in the US, phrased in a way as if any other place in the world sucks in comparison, which is ridiculous, but turns out I followed his advice anyway) because I stayed at an apartment in Williamsburg, I also had 2 roommates, with whom I ironically couldn't spend as much time as I wanted to as I was so busy meeting up with all those other people, I also visited the Etsy and Vimeo offices (both are awesome) and met up with Dave Reisner (who is one of our latest Arch Linux devs, and who lives in NJ, but don't tell anyone) and who forgot to show me around in the Google office ;-) And I realize some of the past sentences are a bit long and busy but that's one of the things I learned at New York I guess. For one week, I almost lived like a real New Yorker, and it was interesting (but exhausting).

Move to Ghent

Ghent, bridge of Sint-Michiels Ghent, coupure Enough about the trips. Back to daily life. I moved to the city of Ghent. Riding by bike to work every day along the scenic Coupure is fun. I am quite proud to say nearly all of my stuff in this apartment is second hand and I've been lucky to receive some free stuff as well (thanks Bram!). Not (only) because I'm cheap money conscious but I like to give things a second life instead of buying something new, lowering the impact on the environment. Even if it doesn't look too well, as long as it's functional. And this is exactly one of those values I mentioned above which is often not understood in our Western society but I was pleased to find out this philosophy is the standard in large parts of Thai culture.

Death metal

Promo shoot Live @ Frontline, Ghent We've done 3 gigs (which had great reception, luckily) and we've got planned a few already for 2012, one of which will be at the From Rock Till Core festival in Merelbeke. We also did a semi-professional photo-shoot, and I made a website (you can tell I'm not a designer).

That wraps up 2011 for me. Good times.. Happy new year everybody!

Sun, 02 Oct 2011

Luamail: a mail client built into luakit

Similarly to how back in 2009 there was no browser that works in a way I find sane, and I started solving that with uzbl, now I'm fed up with the lack of an email client that works in a way I find sane. Uzbl turned out to be a bit cumbersome for my taste, so I switched to the uzbl-inspired but more pragmatic luakit browser, which is much in the same vein, except that all configuration, extensions, event handling, programmatic input etc are done by interfacing with lua API's. Now I want to build the "luakit of email clients". Let me explain what's that all about...

Basically the story is pretty much the same as it was with uzbl. There are no mail clients which offer a high level of customization and interfacing possibilities. There are some mail clients aimed at "power users" and "lightweight mail clients" like mutt/alpine/nmh etc but those are also restricted in extensibility and often crippled in terms of features. Currently I'm using Claws-mail, which is the least sucky client I found, but it's also nowhere near what I want.

::Read more

Wed, 06 Jul 2011

Hitchhiking.. try it.

For the last few months, I've started to actively use hitchhiking as a means to travel between home and work. What started as a "I'm not sure about this, it seems a bit awkward, but I do want to know how it goes and feels, so I'll try it out once" ended up being "this is great, I'm doing it every day and loving it". Here's why you should try it and why it may make your life more awesome.

::Read more

Thu, 16 Jun 2011

Poor mans pickle implementations benchmark

Gensim is a very cool python2, numpy-based vector space modelling (information retrieval) framework. It does the job in a straightforward way, and it has been a great project for me to learn python with because it uses some nice tricks in real life scenarios (like Generators) and is AFAICT elegantly coded. Sometimes I find it hard to believe how much functionality can be crammed in so few lines of (readable) code.

But anyway we're having some issues in it with cPickle (it breaks when saving large matrices, it breaks with some objects). For now I worked around it by using jsonpickle but I wonder how viable this alternative really is.

To give at least a crude idea of performance characteristics of different pickle methods, I wrote a very simple benchmark program - picklebench - to compare pickle, cPickle and jsonpickle. The script fills a dictionary which gets bigger and bigger, and for certain sizes of dictionary it is saved to, and loaded from disk again. We measure some metrics of each step. We continue until memory is exhausted.

limitations of this benchmark:

::Read more

Tue, 17 May 2011

Where are the new Arch Linux release images?

This is a question I get asked a lot recently. The latest official images are a year old. This is not inherently bad, unless you pick the wrong mirror from the outdated mirrorlist during a netinstall, or are using hardware which is not supported by the year old kernel/drivers. A core install will yield a system that needs drastic updating, which is a bit cumbersome. There are probably some other problems I'm not aware of. Many of these problems can be worked around (with 'pacman -Sy mirrorlist' on the install cd for example), but it's not exactly convenient.

Over the past years (the spare time in between the band, my search for an apartment in Ghent and a bunch of other things) I've worked towards fully refactoring and overthrowing how releases are being done. Most of that is visible in the releng build environment repository. Every 3 days, the following happens automatically:

  • packages to build images (archiso) and some of which are included on the images (aif and libui-sh) get rebuilt. They are actually git versions, the latter two have a separate develop branch which is used. Normal packages get updated the normal way.
  • the images are rebuilt, and the dual images get generated
  • the images, the packages and their sources are synced to the public on
Actually things are bit more involved but this is the gist of it. All of this is now run on a dedicated VPS donated by airVM.

I never really completed the aif automatic test suite, somewhere along the way I decided to focus on crowdsourcing test results. The weight of testing images (and all possible combinations of features) has always been huge, and trying to script tasks would either get way complicated or insufficient. So the new approach is largely inspired by the core and testing repositories: we automatically build testing images, people report feedback, and if there is sufficient feedback for a certain set of images (or a bunch of similar sets of images) that allows us to conclude we have some good material, we can promote the set to official media.
The latest piece of the puzzle is the new releng feedback application which Tom Willemsen contributed. (again: outsourcing FTW). It is still fairly basic, but should already be useful enough. It lists pretty much all features you can use with archiso/AIF based images and automatically updates the list of isos based on what it sees appearing online, so I think it will be a good indicator on what works and what doesn't, and that for each known set of isos.

So there. Bleeding edge images for everyone, and for those who want some quality assurance: the more you contribute, the more likely you'll see official releases.

While contributing feedback is now certainly very easy, don't think that only providing feedback is sufficient, it takes time to maintain and improve aif and archiso as well and contributions in that department are still very welcome. I don't think we'll get to the original plan of official archiso releases for each stable kernel version, that seems like a lot of work despite all the above.

As for what is new: again too much to list, here is a changelog but I stopped updating it at some point. I guess the most visible interesting stuff is friendlier package dialogs (with package descriptions), support for nilfs, btrfs and syslinux (thanks Matthew Gyurgyik), and an issues reporting tool. Under the hood we refactored quite a bit, mostly blockdevice related stuff, config generation and the "execution plan" (like, how each function calls each other and how failures are tracked) in AIF has been simplified considerably.

Sat, 14 May 2011

My metal band

Since the audience of this blog is largely technical, I don't post much about other topics, but I feel it's time for a short summary about one of my "real life projects".
In the spring of 2009 I joined a progressive death metal band. I've been drumming since I was 17, but during the last 2 years I've been practising and rehearsing like never before.[1]
When you hear yourself on tape for the first time, it's a bit of disillusionment as you suddenly hear every imperfection, many of which you didn't realise you had (or didn't think were very noticeable).
So 2 years of practicing, rehearsing, test recordings, real recordings, mixing sessions (where you really grow a good ear towards imperfections) later we are now getting to the point where we can nail our stuff and are looking very forward to our first gig, which will be June 3rd in jh sjatoo in Kalken. We've written about 7 songs, of which at this point we play 5. I wish we had proper recordings of all of them, but "Total Annihilation" captures several aspects of our style:
In early 2010 I treated myself (found a nice 2nd hand deal) to a new pdp birch kit with Zildjian Z custom cymbals (that was actually at the time I was in the interview process for a Palo-Alto position at Facebook so I might have needed to sell it again soon after, but that didn't happen).
Here are some pics: 1, 2, 3, 4. More info about the band:

[1] Hence the need I had to find a new maintainer for Uzbl)

Mon, 28 Mar 2011

Thank you Google!

Google, you get a lot of bad words over you lately. "Evil", "big brother", "dangerous", ....
But I just wanted to say: thank you.
You provide us some nice services. Google search, Gmail, analytics, google maps, ...
All of these products are/were game changers and made the life of people all over the world easier.
Many people take them for granted and don't realise what it takes to design, engineer and operate these applications.
That you provide them for free makes it even more amazing. And as far as advertisements are concerned; many business models rely on them and I don't see that changing any time soon.
I'll take a personalized ad over a generic ad any day. The more you can optimize targetted ads on my screen, the more useful I'll find them.
Just don't overdo them, but you know that already.


Sat, 26 Mar 2011

Dvcs-autosync: An open source dropbox clone... well.. almost

I found the Dvcs-autosync project on the vcs-home mailing list, which btw is a great list for folks who are doing stuff like maintaining their home directory in a vcs.
In short: Use cases:

::Read more

Mon, 07 Mar 2011

Let's make the world a better place. Let's stop the abuse of SI prefixes

This has been on my mind for a while. But now I actually took some time to launch a little project to do something about it.
1024 just ain't 1000. stop abusing SI prefixes!

Sat, 05 Mar 2011

Why rewriting git history? And why should commits be in imperative present tense?

There are tons of articles describing how you can rewrite history with git, but they do not answer "why should I do it?". A similar question is "what are the tradeoffs / how do I apply this in my distributed workflow?".
Also, git developers strongly encourage/command you to write commit message in imperative present tense, but do not say why. So, why?
I'll try to answer these to the best of my abilities, largely based on how I see things. I won't get too detailed (there are enough manuals and tutorials for the exact concepts and commands).

::Read more

Sun, 06 Feb 2011

Can we build a simple, cross-distribution installation framework?

Today at Fosdem 2011 I did my talk Can we build a simple, cross-distribution installation framework? Basically, using the Arch Installation Framework as a starting point, along with the notion that most of the code is actually not Arch-specific I adressed other distros to check if there was any interest in sharing workload on the distribution-agnostic aspects of the framework. If other distros with a similar philosophy of little-abstractions/KISS would join, we would all reap the benefits of a simple, yet quite featureful installer. There was some interest, so we'll see what happens.

Tue, 25 Jan 2011

Dir 2011, Fosdem 2011

On February 4, I'll be in Amsterdam at DIR 2011, the 11th Dutch-Belgian Information Retrieval Workshop. After that, I'm going to the devopsdinner and Fosdem beer event in Brussels. On february 5/6 of course, Fosdem itself. Looking forward to the systemd talk. On sunday I'll do a talk about simple shell based Gnu/Linux installers, like mentioned earlier I hope devs from other "lightweight"/kiss-style distro's will be present (Gentoo and other *too's, Crux, *ppix, ... You know who you are) It would be interesting to share some common codebase for distribution independent topics (like filesystems), or at least discuss how feasible it would be.

Sat, 22 Jan 2011

Building a search engine

I started working at IBCN, the research group of the university of Ghent. I was looking to get back to the challenging world of high-performance and large-scale (web) applications, but I also wanted something more conceptual and researchy, rather then the highly hands-on dev- and ops work I've been doing for a few years now.
The Bom-vl project is pretty broad: it aims to make the Flemish cultural heritage media more useable by properly digitizing, archiving and making public the (currently mostly analog) archives from providers such as TV stations.

Currently, I believe there's some >100TB of media in our cluster (mostly from VRT, afaik), along with associated textual descriptions/metadata, with more to follow. The application is currently for a selected audience but the goal is to make it public in the near future. I'm part of the search engine team, we aim to provide users with the most relevant hits for their queries, by using existing technology (think Lucene, hadoop, etc) or devising our own where needed. As I'm charged with a similarity search problem ("other videos which might also interest you"), I'm studying information retrieval topics such as index and algorithm design and various vector models. Starting next week, I'll probably start implementing and testing some approaches.

Wed, 19 Jan 2011

Blog moved

This blog now runs on Pyblosxom on Lighttpd on my new Linode machine.
The moving/conversion process wasn't as smooth as I thought it would be as I needed to work quite a bit on pyblosxom and implement some new plugins to get certain features working (like syntax highlighting).
Also, my previous hosting provider removed my account before the contract expired.
But luckily I managed to restore everything and all should work pretty much as before, in particular:
Rss feeds should still be working on the old urls and the same GUID's are used to avoid spamming anyone, syntax highlighting of all old entries and comments works, but not yet posting highlighted code in new comments.

In related news: pyblosxom has improved quite a bit and is nearing a 1.5 release.
The drupal-to-pyblosxom tool is now quite polished and comes with a bunch of "check whether everything is ok" scripts.

If you notice anything funny, let me know.

Tue, 28 Dec 2010

Libui-sh: a library providing UI functions for shell scripts

== A library providing UI functions for shell scripts ==

When you write bash/shell scripts, do you write your own error/debug/logging/abort functions?
Logic that requests the user to input a boolean, string, password, selection out of a list,
date/time, integer, ... ?

Libui-sh is written to take care of all that.
libui-sh is meant to a be a general-purpose UI abstraction library for shell scripts.
Low impact, easy to use, but still flexible.
cli by default, can optionally use ncurses dialogs as well.

::Read more

Sun, 19 Dec 2010

Migrating blogs from Drupal to Pyblosxom

pyblosxom is a pretty cool blogging platform written in python.
Like many of the modern minimal blog engines it works with plaintext files only (no database), has a relatively small codebase, supports many plugins (like markdown support), is written in a proper scripting language, has a simple and clean file structure, is seo-friendly, and so on.
The one feature that sets it apart from other minimal blog engines is that it supports comments, and doesn't just rely on an external service like disqus, but stores comments as plaintext files as well.
Some features seem a bit overengineered (like, multiple possible locations to store themes (known as "flavours") and templates; I'm a fan of convention over configuration and keeping things simple), but discussing this with the maintainer revealed this is because pyblosxom is meant as a reimplementation of the original perl-based bloxsom project. Over time features could be simplified and/or redesigned.
So I plan to migrate this blog from drupal to pyblosxom.
To do this, I'm building the tool drupal-to-pyblosxom.
The goal is to convert posts, associated metadata (publish time, tags) and comments from the drupal database to pyblosxom files. Source code display should be converted too (merely a matter of converting between different plugin conventions), and images shown should be downloaded. Currently I'm about halfway, if there's anyone out there with a similar use case, help is welcome ;)

::Read more

Thu, 16 Dec 2010

Checking if a git clone has any unique content, git/svn scripts

When cleaning up a system and going over git repositories I often wonder if a git repo contains any interesting, but unpushed work. (i.e. "unique" content)
I heard bzr (or was it hg...) can do it out-of-the-box, but I couldn't find any existing solution for git.
So I wrote a script to do this. It checks a repo for unique commits, tags, branches, dirty files/index, added files, or stashed states. In comparison to a specific remote, or all of them, and uses an appropriate exitcode.
The script is part of a bigger git-scripts repo (most of the scripts written by random people). Although the original repo creator hasn't gotten back to me this seems like a good starting point to have some sense of order in the wildspread of git scripts.

Here are some other scripts I find pretty useful:

::Read more

Wed, 08 Dec 2010

Filesystem code in AIF

In light of the work and discussions around supporting Nilfs2 and Btrfs on Arch Linux and its installer AIF,
I've shared some AIF filesystem code design insights and experiences on the arch-releng mailing list.
This is some hard to understand code. Partly because it's in bash (and I've needed to work around some limitations in bash),
partly because there is some complex logic going on.

I think it's very useful material for those who are interested (it can also help understanding the user aspect),
so I wanted to share an improved version here.
On a related topic: I proposed to do a session at Fosdem 2011/"distro miniconf" about simple (console based) installers for Linux,
and how multiple distributions could share efforts maintaining installation tools, because there are a lot of cross-distribution concerns
which are not trivial to get right (mostly filesystems, but I also think about clock adjustments, bootloaders, etc).
Already several distro's use the (or a fork of) the Arch installer, for example Pentoo,
but I think cooperation could be much better and more efficient.


::Read more

Mon, 22 Nov 2010

Handing off uzbl to a new project leader

As of yesterday, Brendan 'bct' Taylor is the new Uzbl project leader / maintainer.
Yesterday I did the newspost on which explains the reasoning. I can add it feels pretty weird "giving away" and "leaving behind" a project you spent so much time on and which grew a large (well, for a FOSS side project with a hacker audience) base of users and contributors, and which served as inspiration for various other projects.

::Read more

Fri, 15 Oct 2010

Rsyncbench, an rsync benchmarking tool

Background info:
I'm currently in the process of evaluating (V)PS hosting providers and backup solutions. The idea being: I want a (V)PS to run my stuff, which doesn't need much disk space,
but in the meantime it might be a good idea to look for online backup solutions (oops did I say "online"? I meant "cloud"), like on the (V)PS itself, or maybe as a separate solution.
But I've got some diverse amount of data (my personal data is mostly a lot of small plaintext files, my mom has a windows VM for which I considered syncing the entire vdi file)
At this point the biggest contenders are Linode (which offers quite some flexibility and management tools, but becomes expensive when you want extra disk space (2$/month*GB), Rackspace backup gives you 10GB for 5$/month, but they have nice backup tools so I could only backup the important files from within the windows VM (~200MB), and then there's Hetzner, which offers powerful physical private servers with a lot of storage (160GB) for 29eur/month, but less flexibility (I.e. kvm-over-ip costs an extra 15eur/month)

Another issue, given the limited capacity of Belgian internet connections, I needed to figure out how much bandwith rsync really needs, so I can calculate if the duration of a backup run including syncing the full vdi file is still reasonable.

I couldn't find an rsync benchmarking tool, so I wrote my own.


  • simple
  • non invasive: you specify the target and destination hosts (just localhost is fine too), and file locations
  • measures time spent, bytes sent (measured with tcpdump), and data sent (rsync's statistics which takes compression into account)
  • supports plugins
  • generates png graphs using Gnuplot
  • two current plugins: one using files of various sizes, both randomly generated (/dev/urandom) and easily compressable (/dev/zero), does some use cases like initial sync, second sync (no-op), and syncing with a data block appended and prepended. The other plugin collects vdi files from rsnapshot directories and measures the rsyncing from each image to the next

::Read more

Sat, 25 Sep 2010

An rss2email fork that sucks less

Rss2email is a great tool. I like getting all my news messages in my mailbox and using smtp to make the "news delivery" process more robust makes sense.
However, there are some things I didn't like about it so I made a github repo where I maintain an alternative version which (imho) contains several useful improvements, both for end users and for developers/downstreams.
Also, this was a nice opportunity for me to improve my python skills :)

Here is how it compares:

::Read more

Fri, 03 Sep 2010

What the open source community can learn from Devops

Being active as both a developer and ops person in the professional life, and both an open source developer and packager in my spare time, I noticed some common ground between both worlds, and I think the open source community can learn from the Devops movement which is solving problems in the professional tech world.

For the sake of getting a point across, I'll simplify some things.

First, a crash course on Devops...

::Read more

Wed, 25 Aug 2010

the "Community Contributions" section on the Arch Linux forums is a goldmine

The Community contributions subforum of the Arch Linux forums is awesome.
It is the birthplace of many applications, most of them not Arch Linux specific.
File managers, media players, browsers, window managers, text editors, todo managers, and so on. Many shell scripts, urxvt extensions and dwm patches aswell.
Most of the apps are designed after suckless/KISS principles, but there are also some GUI programs.

If you like to discover new apps and tools, check it out.

Mon, 23 Aug 2010

Review of "Python 3 Object Oriented Programming"

Dusty Phillips, Arch Linux shwag guy, Archcon co-organizer, (python) consultant and since recently buddy of mine wrote his first book: Python 3 Object Oriented Programming.

I got the opportunity to get a free pdf copy in exchange for a review on my blog, so here we go.
Mind you, my Python knowledge is very limited. I have done some python programming, and I once read (most of)

::Read more

Sat, 31 Jul 2010

Back from Canada, Archcon

I'm back from Canada/Archcon, and it was great. I've been in Toronto for 11 days, and visited Montreal for 3 days.


Archcon was small (20-ish people). (That's what you get for doing it in Canada ;), but very nice.
Interesting talks, informal, good vibe, decent logistics and catering.
This year it happened because Dusty and Ricardo actually just wanted to have a conference without worrying too much about the attendance,
next year we should do it again because Arch (conferences) rock(s), and because we need more visitors. More central locations such as Seattle and Europe have been suggested.
Either way, next year both Judd (founder) and Aaron (current overlord) should be there. (this year they both had lame excuses like family reunions and "almost getting married". Congrats btw, Aaron!)

It was an absolute pleasure to meet some more of my fellow devs, and users.
Here is a pic from the group (unfortunately, a few are missing)

::Read more

Sun, 04 Jul 2010

Off to Toronto July 14-28, Archcon

As mentioned earlier, I'll be at Archcon in Toronto in a few weeks.
It's a very small conference, and the first of its kind. At the last FrOSCon we have been playing with the idea to hold an informal Arch conference in Europe, but those were just ideas. Dusty and Ricardo beat us with an actual implementation.
This is great, and one of the milestones in Arch Linux history. Which is why I want to be there and help making it better.

::Read more

Wed, 16 Jun 2010

Restoring ssh connections on resume

I use pm-utils for hibernation support.
It has a hooks system which can execute stuff upon hibernate/suspend/thaw/resume/..., but they run as root.
If you want to run stuff as a regular user you could do something like

su $user -c <command>

..but these commands have no access to your user environment.
In my user environment I have a variable which I need access to, namely SSH_AUTH_SOCK, which points to my agent which has some unlocked ssh keys. Obviously you don't want to reenter your ssh key passwords everytime you resume.
(In fact, I started using hibernate/resume because I got tired of having to enter 4 passwords on boot. - 1 for dm_crypt, 1 for login, 2 for ssh keys, not because it is much faster)

The solution is very simple. Use this:

sudo pm-hibernate &&

This way, will be executed when you resume, after the complete environment has been restored.
Ideal to kill old ssh processes, and setup tunnels and ssh connections again.
I'm probably gonna integrate this into my microDE

Sun, 07 Mar 2010

Uzbl, monitoring, AIF talks

I recently did two talks, for which the videos are now online.

If all goes well, I'll be at ArchCon this summer, where I'll be doing these talks:

We're not sure yet if those talks will get videotaped.

::Read more

Fri, 12 Feb 2010

facebook usrbincrash php implementation

Implementation for Facebook usr bin crash puzzle. (how/why)

I haven't touched the code for a few months, but better to put it online then to let it rot.

2 branches:

  • master: basically what I submitted to FB, and what just works
  • withpruning: an attempt for futher optimalisation (it only improves the runtime in some cases) but I didn't finish that version and there's a bug in it somewhere

In the repo you'll also find various test input files supplied by the community on the forums and a script to benchmark the implementation on all inputfiles.

Not working for Facebook

In november last year, I was contacted by Facebook HR.
They found my background interesting and thought I might be a good
fit for an "application operations engineer" position in Palo Alto, California. (it is
basically the link between their infrastructure engineering and operations/support

::Read more

Sun, 24 Jan 2010

Fosdem 2010

I'll be at fosdem - 10th edition - again this year.
I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting

I'll be presenting a lightning talk about uzbl.
Also, Arch Linux guys Roman, JGC, Thomas and me will hang out at the distro miniconf. We might join the infrastructure round-table panel, but there is no concrete information yet.

More stuff I'm looking forward to:

::Read more

Mon, 11 Jan 2010

Arch Linux interview and Uzbl article

Apologies for only informing you about the second article now. I assumed most of you follow LWN (you probably should) or found the article anyway.
Of all the articles written about uzbl, no one came close to the quality of Koens work. So even though it's a bit dated it's still worth a read.

Wed, 09 Dec 2009

RRDtool: updating RRA settings and keeping your collected data

When you use rrdtool, it can happen that you first create your databases, then collect a whole bunch of data and decide later you want more accuracy/longer periods.
Especially when using zenoss (the monitoring solution I mostly work with at Kangaroot), which uses very conservative RRD settings by default (i.e. 5-minute intervals for only the first 50 hours). Zenoss provides a way for you to change the way RRD's are created, but not to apply those settings to already existing RRD files, which I found out *after* I started monitoring everything ;)

rrdresize can help: it (just) adds or removes locations for rows.
In my case it was not good enough because zenoss uses a variety of resolutions (step sizes), and so if you add rows to all of them rrdtool - when graphing - will often pick a higher resolution RRA that just had rows added (and hence contain unknown values), even though you have the values, albeit at a lower resolution.

So you need a way to update all rows in the RRA's.
I found a perl tool that does just that. (I think, I didn't study all details). So, you install that in your /home/zenoss for instance and then you run the following script, which creates new rrd files with the new settings and uses the perl script to copy all data into it.

# invoke me like this:
# find /usr/local/zenoss/zenoss/perf/ -name '*.rrd' -exec ./ {} \; >> newrrd-logfile

[ -d "$backupdir" ] || mkdir -p "$backupdir" || exit 2
[ -d "$newdir"    ] || mkdir -p "$newdir" || exit 2
[ -f "$file"      ] || exit 3

echo "Processing $file .."
base="`basename "$file"`"
[ ! -f "$backupdir/$base" ] || mv "$backupdir/$base" "$backupdir/$base".old || exit 4
cp "$file" "$backupdir/$base"
cd "$newdir" && rrdtool create "$base" \
--step '300' \
--start '1230768000' \
'DS:ds0:GAUGE:900:U:U' \
'RRA:AVERAGE:0.5:1:122640' \
'RRA:AVERAGE:0.5:6:55536' \
/home/zenoss/ "$backupdir/$base" "$base" | grep -v 2009 # hide some output
cp "$base" "$file" || exit 5
echo "Done"

Oh and btw, rrdwizard is a cool webapp when you're feeling too lazy/have forgotten how to write rrdtool commands

Sun, 01 Nov 2009

ext3 logical partition resizing

You probably know you can resize primary partitions by deleting them and recreating them, keeping the starting block the same but using a higher block as ending point. You can then increase the filesystem.
But what about logical partitions? A while back I had to resize an ext3 logical partition which ended at the end of the last logical partition. I learned some usefull stuff but I only made some quick scratch notes and I don't remember all details so:
Do not expect a nice tutorial here, it's more of a commented dump of my scratch notes and some vague memories.
The information in this post is not 100% accurate

I wondered if I could just drop and recreate the extended partition (and if needed, recreating all contained logical partitions, the last one being bigger of course) but nowhere I could find information about that.

::Read more

Mon, 12 Oct 2009

About the maemo summit 2009 and the nokia n900

So I'm back from the 3-day maemo summit in Amsterdam. It was very nice. Very well organized, and Nokia definitely invested enough in catering, fancy-suited people and such to please all 400 of us. I met several interesting people, both from the community, as well as Nokia guys.
The talks were diverse, but interesting (duh?). I will especially remember the kickoff with its fancy visual effects and loud music that set the mood straight for the entire weekend.
The best moment was, of course, when it was announced that every summit participant would receive a n900. Uncontrolled hapiness all around.

::Read more

Fri, 02 Oct 2009

nokia n900 & maemo summit 2009

I have been looking for the "perfect mobile companion device" already for a while. Basically I want a "pocket PC that can do as much as possible over which i have as much control as possible so I can do things my way, but still fits in a pocket and which can do gsm and such"
So, something like a netbook, but really portable, and that can also do telephony stuff.
Nokia's recently announced n900 seems to be very close to what I'm looking for.
It could have been a tad bigger (to make typing easier) but other then that it looks perfect: powerful, high resolution display, Linux with a "usual" userspace (unlike Android) to give me all freedom I'm looking for, keyboard, plenty of space and many goodies such as wifi, a-gps, fm receiver/transmitter, IR, bluetooth, digital camera, tv-out and so on.

This device has ignited my interest in Maemo and all things related so I'll be in Amsterdam on October 9-10-11, at the Maemo summit 2009. I was lucky enough to score a place in the ibis hotel Amsterdam, as Nokia has reserved more rooms then they could fill with invited speakers and own personnel ;-)
I'm hoping it will be possible to buy a device at the conference. The timing would be perfect. Nokia seems to be a really cool company and so far, they haven't disappointed yet...

Somewhat related: a Mer (maemo alternative) developer told me he was very interested in making uzbl available on Mer, so he did just that. I'm curious myself how usable uzbl will be on the n900. Only one way to find out :)

Tue, 22 Sep 2009

Opening files automatically on mainstream Linux desktops

Xfce/Gnu/Linux works amazingly well on my moms workstation, with one exception: opening files automatically with the correct program.

The two biggest culprits are:

  • Gtk's "open file with" dialog: if any Gtk program doesn't know how to open a file it brings up this dialog that is horrible to use. You can search through your entire VFS for the right executable. No thumbnails, no usage of .desktop files, $PATH, autocompletion and not even limiting the scope to directories such as /usr/bin
  • Mozilla software such as Firefox and Thunderbird: they only seem to differentiate files by their mimetype, not by extension. There are add-ons to make it easier to edit these preferences, but eventually you're in a dead end because you get files with correct extensions but unuseful mimetimes (application/octet-stream)

Luckily the fd.o guys have come up with .desktop files.

::Read more

Mon, 14 Sep 2009

Snip: a dead-simple but quite powerful text expander and more

Inspired by Snippy and snippits, I wrote a simple tool called snip.
It helps you to automatically fill in text for you (which can be dynamically created) and/or to perform custom keypresses and operations.

::Read more

Fri, 04 Sep 2009


I'm starting to keep track of some things I want. I've picked Amazon because they have many items in their database.

Fri, 28 Aug 2009

Froscon 2009 afterthoughts

Froscon was great.

  • It was smaller then I thought, I especially assumed more activity in the devrooms (I'm too used to fosdem which is much bigger), but hey nice conference anyway.
  • It was great to meet (some of) my Arch Linux colleagues. Mostly the guys from Germany showed up, and Roman who is from Ukraine.
  • We all got an Arch t-shirt/polo of which we could pick the color ourselves. Woohoo. Thanks donators.
  • A few people also told me they had tried / were using uzbl (my browser project) so that was really cool.
  • I had an interesting chat with Thomas Lange, the author of FAI (a tool to automatically mass-install systems, mostly Debian). I've used it in the past, liked it and was somewhat inspired by it when building AIF. (but some design goals are different).

Here is a picture of my "Arch releng partner" Gerhard (right) and I (left). We've done a lot of work together and it was great to talk face to face for once. Here we're showing (proudly) an Arch Linux Froscon disc (which is a slightly modified version of the 2009.08 core images):

Team photo:

Image gallery

Tue, 18 Aug 2009

A script that pulls photos from facebook

Fbcmd is pretty cool.
I quickly hacked this script together which pulls all photo albums from friends on facebook, so I have them available where I want. (It should also pull your own albums, but I don't have any so I can't check that)

::Read more

Mon, 10 Aug 2009

Arch Linux 2009.08 & Froscon 2009

So, the Arch Linux 2009.08 release is now behind us, nicely on schedule.
I hope people will like AIF because it was a lot of work and we didn't receive much feedback. I personally like it to apply my fancy backup restoration approach.
But I'm sure if more people would look at the code we would find quite some design and implementation things that could be improved. (With uzbl I was amazed how much difference it can make if many people all have ideas and opinions about every little detail)

Later this week I'm off to the Counting Cows festival in France, and the week after that (august 22-23) I'm going to FrOSCon in Germany where I will meet some of my Arch Linux colleagues in real life, which I'm really looking forward to.

If anyone wants a ride to froscon let me know. But note I'll try to maximize my time there (leave saturday early and come back late on sunday. I even took a day off on monday so I might stay a day longer if I find more interested people to hang out there)

Wed, 22 Jul 2009

AIF automatic lvm/dm_crypt installations and test suite

We're working hard on a new Arch release. (should be done by froscon)

Amongst the slew of fixes and improvements there are also some cool new things I'm working on.
First of all, I worked more on the automatic installations. Now you can easily install an LVM based Arch system on top of dm_crypt for example.
You type this command:

aif -p automatic -c /usr/share/aif/examples/fancy-install-on-sda

And bam you have a complete working system with LVM, dm_crypt etc all set up. You just need to change your keymap, hostname, network config and such (or, configure that on the beforehand in the config file for AIF)

Another thing I started working on is a very simple test suite.
Basically, when launching a test, the following steps are invoked

  • installation of an arch system with aif's automatic procedure using a certain config file
  • installation of a verification script onto the target system and configuration of the target to run the script on boot (DAEMONS variable in /etc/rc.conf)
  • if aif ended succesfully: automatic reboot.. and tada!

The verification script will check things like availability (and size) of LVM volumes, amount of swap space, keyboard layout, network and so on.
Here's a picture of a rough first version:

Stay tuned!

Sat, 06 Jun 2009

Mysql status variables caveats

While setting up Zenoss and reading Mysql documentation about status variables I learned:

  • All select_* variables ("Select statistics" graph in Zenoss) are actually about joins, not (all) selects. This also explains why there is no clear relation to com_select (which shows the amount of selects). ("Command statistics:selects" graph in Zenoss)
  • Com_select does not denote all incoming select commands. If you have a hit on your query cache, com_select is not incremented. So I thought we were doing less qps while in fact we were just getting more cache hits. Qcache_hits gets incremented on cache hits (but is not monitored by Zenoss)

Zenoss & Mysql monitoring

I've been playing with Zenoss (2.4) for the first time. Here are my thoughts:

::Read more

Mon, 27 Apr 2009

Uzbl. A browser that adheres to the unix philosophy.

I need a browser that is fast, not bloated, stores my data (bookmarks, history, account settings, preferences, ...) in simple text files that I can keep under version control, something that does not reinvent the wheel, something that I can control.

Well, I could not find it.
So I started the uzbl browser project.

::Read more

Sat, 25 Apr 2009

Poor mans dmenu benchmark

I wanted to know how responsive dmenu and awk, sort, uniq are on a 50MB file (625000 entries of 80 1-byte chars each).

generate file:

echo "Creating dummy file of 50MB in size (625000 entries of 80chars)"
echo "Note: this takes about an hour and a half"
for i in `seq 1 625`
        echo "Iteration $i of 625 ( $entries_per_iteration each )"
        for j in `seq 1 $entries_per_iteration`
                echo "`date +'%Y-%m-%d %H:%M:%S'` `date +%s`abcdefhijklmno`date +%s | md5sum`" >> ./dummy_history_file

measure speed:

echo "Plain awk '{print \$3}':"
time awk '{print $3}' dummy_history_file >/dev/null

echo "awk + sort"
time awk '{print $3}' dummy_history_file | sort >/dev/null
echo "awk + sort + uniq"
time awk '{print $3}' dummy_history_file | sort | uniq >/dev/null

echo "Plain dmenu:"
dmenu < dummy_history_file
echo "awked into dmenu:"
awk '{print $3}' dummy_history_file | dmenu
echo "awk + sort + uniq into dmenu:"
awk '{print $3}' dummy_history_file | sort | uniq | dmenu

I ran the test twice about an half hour after generating the file, so in the first run, the first awk call may have been affected by a no longer complete Linux block cache.
(I also edited the output format a bit)
Run 1:

Plain awk '{print $3}':
real	0m1.253s
user	0m0.907s
sys	0m0.143s

awk + sort:
real	0m3.696s
user	0m1.887s
sys	0m0.520s

awk + sort + uniq:
real	0m15.768s
user	0m12.233s
sys	0m0.820s

Plain dmenu:
awked into dmenu:
awk + sort + uniq into dmenu:

Run 2

Plain awk '{print $3}':
real	0m1.223s
user	0m0.923s
sys	0m0.107s

awk + sort:
real	0m2.799s
user	0m1.910s
sys	0m0.553s

awk + sort + uniq:
real	0m16.387s
user	0m12.019s
sys	0m0.787s
Plain dmenu:
awked into dmenu:
awk + sort + uniq into dmenu:

Not too bad. It's especially uniq who seems to cause a lot of slowdown. (in this dummy test file, are entries are unique. If there were lots of dupes, the results would probably be different, but I suspect that uniq always needs some time to do its work, dupes or not). The real bottleneck seems to be raw cpu power. Not storage bandwidth at all since Linux caches it. If uncached, I estimate the sequential read would take 1.5 seconds or so. (about 30MB/s on common hard disks)

Once the stuff gets piped into dmenu, there is a little lag but it's reasonably responsive imho.
Test performed on an athlon xp @ 2GHz. 1 GB of memory. There were some other apps running, not a very professional benchmark but you get the idea :)

Sat, 14 Mar 2009

Automatic installations with AIF

Yesterday I finished the first working version of AIF's automatic procedure, along with a sample config for a basic install..

For me personally this means I can start working on the next step towards my goal of having all my systems "metadata" centrally stored (along with my real "data"), and the possibility to reconstruct all my systems in a deployment-meets-backup-restore fashion ( see rethinking_the_backup_paradigm_a_higher-level... )

The last few weeks I/we've been quite busy fixing up many things related to aif and the arch release stuff in general. The kernel guys are now at 2.6.29-rc8, so a stable release won't be long anymore and I want to get as much things fixed before 2.6.29 is out, so we can release soon after. (with /arch/setup and /arch/quickinst replaced by aif)

Misc notes:

  • Working together with Aaron and Gerhard is great. Our interests and knowledge are diverse, so we each have our place in the releng team, but there is some overlap too. This works out great. Input from them (and the other developers) often brings fixes for stupid things that I've spent too much time on and was still looking over.
  • I'm also a "real" Arch developer (this actually means package maintainer) now: I maintain 1 package :)
  • The 2009.02 release was a success. I was suprised by the amount of attention it got (maybe because we were the first distro to ship a release that supports ext4). I realised that all those delays to fix some (imho) details was probably the best move after all.
  • I've been looking at nixOS, which is a very interesting project. The way I see it, they combine the "functional description" part (which is on many systems handled by puppet/cfengine/kickstart/aif/...) right into their OS core. That, and some other features. It was a bit too unstable when I tried it though.

Thu, 05 Feb 2009

Fosdem 2009

I'm going to FOSDEM, the Free and Open Source Software Developers' European Meeting

I'm particulary interested in:

Thu, 22 Jan 2009

Arch Linux release engineering

I don't think I've ever seen so much anxiety/impatience/hope/buzz for a new Arch Linux release. (this is because of 2.6.28 with ext4 support).
The last release was 6 months ago, which is not so good.. also the arch-installer project has been slacking for a while. But the Arch devs have been very busy and many things going on. You know how it goes...

That's why some new people have stepped up to help out on a new release:
Today, we are on the verge of a 2009-01 release (though that has been said so many times lately ;-) and together with Aaron we have started a new project: the Arch Linux Release Engineering team.
Members of this team are Aaron himself, Gerhard Brauer and me.

Our goals:

  • coordinated releases following the rhythm of kernel releases (that's a release every 2-3 months baby!).
  • anticipate the availability of the kernel in the testing repo's instead of having to wait for it to go to core before building alpha/beta images
  • migration to AIF as the new Arch installer (woot!)
  • testing. Leveraging the possibilities of AIF as an unattended installer, we should be able to script full installations and health checking of the resulting system.
  • involving the community more? releasing "testing iso's" ? not sure about that. we'll see...

We also have:

Oh yeah AIF is mirrored @ and available packaged in the official repo's!

Mon, 19 Jan 2009

CakePHP and a paradigm shift to a code generation based approach?

At my new job, I'm writing a quite full-featured web application.
I've choosen to use CakePHP.
Why? Well, it may be 2 years since I last used it, but I've followed the project and it's planet, and it seems to have matured and gained even more monumentum.
I want to use something that is widely used so there is plenty of stuff available for it, it's RAD, it's flexible and powerful.
I noticed things such as CLI support and documentation have improved tremendously too.

However, I find that still, the recommended (or at least "most commonly used") practices are not as efficient as they could be, and that emphasis is placed on the wrong aspects.
See, even though the bake tool has come a long way since I last used it, it's still used to "generate some standard models/controllers/views" and the developer can take it from there [further editing the resulting files himself].
Finetuning generated code by editing the templates (in fact, only views have templates; the php code of models and controllers is hardcoded in the scripts that generate them), is still an obscure practice...
Also, there are very few commandline switches (Right now you can choose your app dir, whether you want to bake a model,controller or view, and it's name.)
All other things (validation rules, associatons, index/view/edit/add actions/views, which components, overwrite yes/no etc) are all handled interactively.
There are also some smaller enoyances such as when you specify one option like the name of the model, it assumes you don't want interactivity and produces a model containing nothing more then the class definition and the membervariable $name, which is usually worthless.
One thing that is pretty neat though, If you update $this->recursive in a model, the baked views will contain stuff for the associated things. But so much more could be done...

::Read more

Wed, 24 Dec 2008

Jobhunt over.

What better way to launch the new year then starting to work as a System Engineer/Developer for a consulting firm where everyone breathes Linux and Open Source?
Next week I'll start at Kangaroot. Woohoo.

Wed, 17 Dec 2008

new AIF release

My holidays present for Arch devs and users: AIF alpha-0.6 !

* Changes since alpha 0.5:

::Read more

Sat, 13 Dec 2008

#1 productivity tip: showers

When you're stuck on a problem, or not even stuck but you just want to boost your creative/out-of-the-box thinking...
Take a shower. When I'm thinking about a problem and I take a shower, the ideas and thoughts just start popping up, one after each other, or sometimes even two at the same time. It's amazing. And it works every time.

::Read more

Mon, 24 Nov 2008

Looking for a new job

The adventure at Netlog didn't work out entirely, so I'm looking for a new challenge!

My new ideal (slightly utopic) job would be:

  • Conceptual engineering while still being close to the technical side as well, most notably system engineering and development.
  • Innovative: go where no one has gone before.
  • Integrated in the open-source world. (Bonus points for companies where open source is key in their business model)

To get a detailed overview of my interests and skills, I refer to:

Mon, 17 Nov 2008

AIF: the brand new Arch Linux Installation Framework

Recently I started thinking about writing my own automatic installer that would set up my system exactly the way I want.
(See rethinking_the_backup_paradigm_a_higher-level...)

I looked at the official Arch install scripts to see if I could reuse parts of their code, but unfortunately the code was just one big chunk of bash code with the main program and "flow control" (you must first do this step, then that), UI-code (dialogs etc) and backend logic (create filesystems, ...) all mangled up and mixed very closely together.
Functionality-wise the installer works fine, but I guess the code behind it is the result of years of adding features and quick fixes without refactoring, making it impossible to reuse any of the code.

So I started to write AIF: the Arch Linux Installation Framework (actually it had another name until recently), with these 3 goals in mind:

::Read more

Handling a remote rename/move with Git

I recently had to rename a repo on my Github account. Github has made this very easy but it's just one side of the issue. Obviously you must also update any references to this remote in other clones, otherwise pushes, fetches etc won't work anymore.

You can do this in two ways:

  • open .git/config and modify the url for the remote manually
  • git remote rm origin && git remote add origin$user/$project.git

That's it! All will work fine again.

Sun, 09 Nov 2008

Muse ... wow

Weird as it might sound, I've never bothered to listen to Muse songs.. until now. Some people have recommended the band to me so I really had to stop ignoring this band someday. And wow.. what have I been missing al that time :/
Songs like Butterflies and Hurricanes and Citizen Erased are among the most beautiful songs I've ever heard now.

Sun, 19 Oct 2008

dautostart, a standalone freedesktop-compliant application starter

I couldn't find a standalone application/script that implements freedesktop compliant (XDG based) autostarting of applications, so I decided to write my own.
The project is at .

Right now, all the basics seem to work (except "Autostart Of Applications After Mount" of the spec).
It's probably not bugfree. I hacked it together in a few hours (but it works for me :-). Bugreports welcome!

::Read more

Tue, 14 Oct 2008

Visual feedback of the exit status of the previous command in bash

Put this in your .bashrc, and the current directory in your PS1 will be printed green if the previous command had exit state 0, red otherwise. No more typing 'echo $?', ' && echo ok', '|| echo failed' etc on the command line.

::Read more

Mon, 29 Sep 2008

I'm done with Gnome/Gconf

I'm managing my ~ in svn but using gnome & gconf makes this rather hard.
They mangle cache data together with user data and user preferences and spread that mix over several directories in your home (.gconf, .gnome2 etc).
The .gconf directory is the worst. This is where many applications store all their stuff. User preferences but also various %gconf.xml files, which seem to be updated automatically everytime 'something' happens: They keep track of timestamps for various events such as when you press numlock or become available on pidgin.
I'm fine with the fact they do that. I'm sure it enables them to provide some additional functionality. But they need to do it in clearly separated places (such as xdg's $XDG_CACHE_HOME directory)

::Read more

Tue, 23 Sep 2008

DDM v0.4 released

DDM v0.4 has been released.
Since the last release many, many things have been changed/fixed/added.

::Read more

Tue, 09 Sep 2008

My projects are on github now

I've put my (somewhat interesting) projects on GitHub.
Git is a damn cool VCS for distributed development, and I think Github integrates with it really nicely, adding some useful aspects for following and collaborating on projects.
The projects I have migrated to my GitHub profile are:

::Read more

Wed, 13 Aug 2008

A fast way to get stuff out of your head and into your GTD inbox

Often while you're occupied with something, some thought pops into your head. Something that you want to remember/do something about.

::Read more

Sat, 09 Aug 2008

Requirements for the perfect GTD tool

I've been reading GTD lately and it's absolutely a great and inspiring book.
Having made my home office space into a real Zen I want to start implementing GTD in my digital life but it seems very hard to find a good GTD tool that fully implements GTD. (even though there are a lot of tools out there)

The most interesting ones (each for different reasons) I've looked at so far are Thinkingrock, tracks and yagtd (the latter requiring most work before it does everything I need, but it's also the most easy to dive into the code base). I'm keeping my eyes open because there are certainly more things to discover.

Even though there are probably no applications out there that can do everything I want, I just wanted to share my feature-wishlist. These are the requirements I find that a really good tool should comply with:

::Read more

Mon, 21 Jul 2008

Rethinking the backup paradigm: a higher-level approach

In this post I explain my vision on the concepts of backups and how several common practices are in my opinion suboptimal and become unnecessary or at least can be done more easily by managing data on a higher level by employing other patterns such as versioning important directories and distributed data management.

::Read more

Sun, 15 Jun 2008

Dump your azerty and querty because the only keyboard layout that makes sense is Dvorak!

For a while now I am typing using solely the Dvorak keyboard layout. I roughly estimate it has been 4 or 5 months now - with the first month being a pain in the ass because i had to relearn typing pretty much from scratch - but now my typing speed is starting to exceed what it used to be in querty, and I still have much headroom to improve.

For those who have no clue what I'm talking about: think for 30 seconds which characters you type the most and which the least (eg: which characters occur the most/least in the language you type?).

Ok you got them? Now look at your keyboard and spot where these characters are. Now consider where your fingers are most of the time (if you've never learned to type: the 'base position' for your fingers is on the middle row). Notice anything strange?

::Read more

Sat, 24 May 2008

Announcing the Netlog Developer Pages

At work, we've setup the Netlog Developer Pages

It is the place where you can/will find all information around our OpenSocial implementation, our own API, skin development, sample code and so on.
We've also launched a group where you can communicate with fellow developers and Netlog employees.
The page also features a blog where you can follow what is going on in the Netlog Tech team.

PS: We've also updated our jobs page

Thu, 01 May 2008

Windows sucks

I had to fix a problem at my dad's company...
"The network was broken."

It was a NetBEUI network connecting some windows stations - it has been running for years - and now suddenly the nodes couldn't find eachother.
One of the boxes (windows 2000 iirc) had 2 network cards, one for the network, the other not used for anything (not even connected). Disabling the latter - not even touching the former - fixed half of the network.

There was another box that couldn't find any other node in the network. This happened to be a box of which the ps/2 mouse broke. It had a usb mouse connected but since it was windows 95 it was not supported. I removed the usb mouse and attached another ps/2 mouse. This fixed not only the mouse but also the box could suddenly find the other boxes again....

Windows really does suck.

And the worst part is: even though all is fine now, I have no clue for how long it will work, and what will be the cause the next time it will be broken?

Sat, 19 Apr 2008

I survived LCL 31-3-2008

On 31-3-2008 LCL, one of the most used datacenters in Belgium - and the only one with a 0% downtime record in Belgium - had major power issues with their datacenter in Diegem, bringing lots of Belgian parties offline. (more specifics on the net).

If you're one of the sysadmins of a website with 35M members and 150M hits per day this means you're in for an exciting night ...

::Read more

Sat, 29 Mar 2008

DDM : a Distributed Data Manager

UPDATE: this information is outdated. See for latest information.


If you have multiple sets of data (e.g.: music, images, documents, movies, ...) and you use these on more then one system ( e.g. a laptop and a file server) then you probably also have some 'rules' on how you use these on your systems. For example after capturing new images you maybe put them on your laptop first but you like to sync them to your file server frequently. On the other hand you also want all your high-res images (stored on the server) available for editing on the laptop, and to make it more complicated you might have the same images in a smaller format on your server (for gallery programs etc.) and want these (or a select few albums of them) available on the road.

The more different types of data you have and the more you have specific work flows the harder it becomes to keep your data as up to date as possible and consistent on your boxes. You could manually rsync/(s)cp your data but you end up in having a mess (at least that's how it turned out on my boxes). Putting everything under version control is great for text files and such, but it's not an option for bigger (binary) files.

I wanted to keep all my stuff neatly organised in my home directories and I want to create good work flows with as minimum hassle as possible, so I decided to write DDM: the Distributed Data Manager.

::Read more

Mon, 03 Mar 2008

Tweaking Lighttpd stat() performance with fcgi-stat-accel

If you serve lots of (small) files with Lighttpd you might notice you're not getting the throughput you would expect. Other factors (such as latencies because of the random read patterns ) aside, a real show stopper is the stat() system call, which is a blocking system call ( no parallelism ). Some clever guys thought of a way to solve this : a fastcgi program that does a stat(), so when it returns Lighty doesn't have to wait because the stat information will be in the Linux cache. And in the meanwhile your Lighty thread can do other stuff.

::Read more

Tue, 12 Feb 2008

I'm not going to Fosdem 2008

I wish I could put this on my webpage :

I’m going to FOSDEM, the Free and Open Source Software Developers’ European Meeting

::Read more

Wed, 30 Jan 2008

Per-directory bash history

I've been thinking about how a specific bash history for each directory could improve productivity, and unlike what I feared it was actually pretty easy to find a solution on the net.

::Read more

Sat, 19 Jan 2008

The key to mastering a musical instrument ...

The key to mastering a musical instrument is learning an other.

Wed, 28 Nov 2007

Hacking into my router by brute-forcing http authentication

I forgot the username and password to access the web panel of my router.
Luckily I knew some possible usernames and some patterns that I could have used to construct my password, so I just had to try all the combinations... Too much work to do manually but easily done when scripted.

Here is the php script that I came up with. (obviously stripped of my personal stuff). It got my account in less then a second :)

::Read more

Sun, 25 Nov 2007

gtk dialogs for (shell)scripts with zenity and the ask-pass gui tools for ssh-add

Phew! where to start? Probably at this blogpost. It's about making it very easy to work with external encrypted volumes. I'm not going to talk about the article itself but about a great tool i discovered thanks to it: Zenity. It's an LGPL-licensed program written in C by some guys from Gnome and Sun. You can call it from any script and present a user with a gtk widget such as a password-dialog, filechooser, calendar, ... It has

::Read more

Sat, 15 Sep 2007

Simple command to retrieve stuff from RAM after closing/crashing an application

After an app is closed or crashed, the data is still in your RAM and you can very easily get it back by grepping /proc/kcore.
thanks Martin for this tip!

Sun, 26 Aug 2007

Emulating two-dimensional (or even multi-dimensional) arrays in bash

Ever needed to use arrays of two or more dimensions but got stuck on Bash limited array support which provides only 1 dimension?

There is a trick that let's you dynamically create variable names. Using this, you can emulate additional dimensions.

::Read more

Sun, 05 Aug 2007

Nagios monitoring in your desktop panel aka Xfce Genmon panel plugin rules!

FOSS is written by users, for users, and what I've been doing/experiencing this afternoon is a perfect example of that.

::Read more

Sun, 29 Jul 2007

Upgrading Drupal the easy way

I just upgraded this site to Drupal 5.2. The package came with upgrading instructions consisting of 11 steps to complete the upgrade proces, but after reading it a few times I realized it could be done easier.

::Read more

Thu, 19 Jul 2007

Bye CakePHP, bye dAuth... Hello Drupal!

I'm afraid the time has come to say goodbye to CakePHP, and to the projects I've been working on for it.
I still like Cake ... In fact, the further development of 1.2 goes the more I like it (well, generally spoken that is ... because there are some minor things I don't like but that's not important now). The truth of the matter is I like to develop, I like the php language and I enjoy working with Cake.
But .. all the sites I currently work on are all community sites or blogs, and although some of them have some specific requirements, in the end it's all very generic and a full blown content management system like Drupal proves much more useful and feature full then developing my own application in a web application framework such as Cake. (even if that's becoming easier and easier to do)

::Read more

Sat, 14 Jul 2007

Assymetric keys instead of passwords for SSH authentication to increase security and convenience

I've been using OpenSSH already for a while and although I've seen mentions of "public key authentication" and "RSA encryption" several times in it's config files, I never decided to figure out what it did exactly, and stuck to password authentication. But now the guys at work explained how it works and after reading more about it, I'm totally hooked on it!

It's a feature in ssh protocol version 2 (thus it's around for already a while, e.g. we can all use it without updating something) which essentially comes down to this: you generate an asymmetric key pair and distribute the public key to all remote hosts. When logging in to that host, the host will encrypt a random integer, which only you can decrypt (using the private key) and hence prove your identity. Too secure your private key you'll store it encrypted with a password. Ssh-agent (which is bundled with openssh) is the tool that interacts with ssh to perform this task: when logging in to a host, ssh-agent will open the private key for you automatically if it can decrypt it with the password it receives from you But the problem is you'll have to load (enter your password and decrypt the key) each time again.

This is where keychain comes in, or you can use SSH Agent (don't confuse this with the ssh agent that comes with openssh) if you're a Mac user and like gui's. These tools basically ask you your passwords for all private keys you wish to use in a session (with session I mean "the whole time of using your computer"), decrypt the encrypted key on your hard disk and cache the decrypted key in ram, so it can be used in all terminals you open.

For more information:
OpenSSH key management, Part 1: Understanding RSA/DSA authentication
OpenSSH key management, Part 2: Introducing ssh-agent and keychain
OpenSSH key management, Part 3: Agent forwarding and keychain improvements (freaks only ;-))

Have fun

Sun, 08 Jul 2007

The perfect GTK music player: can Exaile replace Amarok?

I've always liked Amarok: it does everything I always wanted, and more. It looks perfect in every way ...
But .. it uses the QT library, and although there are tricks to make QT applications more fit in with your gtk desktop/theme it will never fit in perfectly, not only graphically but also because you still need to load the qt libraries when you want to listen to some music and it is built to interact with the KDE desktop environment.

So, I've been looking for an alternative, a GTK application strong enough to actually be able to replace Amarok, the king of all software music players.

::Read more

Webpages should not contain "add to Digg / / Technorati /..." links

I don't like pages / articles / blog posts /.. accompanied by "Digg this", "add to" or "add to Technorati" links.
Why not? Because this is meta level functionality. Not functionality of the blog/article/page in question, but on a higher level. And thus this should be handled on a higher level: the web browser. Just like we can create and manage bookmarks (I mean the old fashioned ones, not the delicious ones) in our browser: this is not the task of a web page. (we all know how silly "bookmark this" links look on a page, right?)

Whether you like these kind of services or not is up to you (personally I think the most popular content is often the most subjective, biased and close minded, not to mention too mainstream for "real" geeks but that's another story) but people who are serious about it should just make sure they can do it for any page they visit (eg use a Firefox extension to enable digging and adding to delicious) so we can get rid of this ugly clutter that is put on some pages.

I know not every browser supports this already (either by default or by extending it with (3rd party) plug-ins) and even if it is, not everyone enables this functionality, so it's a bit a chicken and egg problem.

But then again, most web2.0 people already use a browser that supports it, or will support it in the very near future, so let's get rid of this inappropriately placed meta-functionality !

Wed, 04 Jul 2007

Afgestudeerd - graduated

Foreign visitors: yeay I graduated today \o/

Dutchies: joepie, afgestudeerd...
Vandaag proclamatie gehad, ben geslaagd met voldoening, zelfs geen enkele buis \o/
En een 12/20 voor de Masterproef :-)

Sun, 01 Jul 2007

PhpDeliciousClient, a php cli client to administer accounts

PhpDeliciousClient is a console based client for doing maintenance on accounts.
I wrote it because - to my knowledge - there currently is no good program (including the personalized web page itself) that lets you make changes to your data in a powerful, productive manner. (with data I primarily mean tags. Posts and bundles are considered less important).

You probably are familiar with the fact that a Delicious account (or any tag based meta data organizing system, for that matter) can soon become bloated: It gets filled with way too many tags. Among those tags several of them mean the same (fun, funny, humor, ...) or include the other (humor, jokes, ...) You can group them in bundles but even then you need to add all the tags to a post if you want it to appear in the results for that tag. Not very convenient. Also, if you have your bookmarks available in Firefox, you'd have a menu with several hundreds of entries (one for each tag), each menu containing usually just a few (or worse: just one) entry.

When I got in this situation I tried to fix it, but it was a hell of a task to do this on the Delicious webpage itself, and I although I found some other tools they were far to basic, outdated, dependent on other stuff or just not meant for this kind of task, so I decided to write my own.

The result is a php command line program called PhpDeliciousClient (as you can see, I added it to the menu on the left too), which uses the PhpDelicious library to access the api.

The primary focus of the program is to help you to bring your tags in balance, in an as efficient way as possible. Other stuff, which can be done just fine on the delicious page (editing single posts, changing your password, ...) is not implemented.

It's a bit hacky, I don't give any guarantees but I can tell I used it to edit my own page, going from about 400 tags to about 80 without any problems.

That said, head over to the PhpDeliciousClient project page for some more information, and to download it ;-)

Sun, 24 Jun 2007

Getting statistics about events that don't trigger page requests with Google Analytics

You probably already heard of Google Analytics. It's a pretty nice program that (basically) gathers data about visits of your site and creates reports of it. It works by including some JavaScript code on your page, so that each page request triggers a call to the Analytics tracker sending along some data such as which page is requested and which resolution was used. (no personal or other privacy-sensitive data is sent). But here is the deal! I just discovered that you can also track events that don't require page requests!
Think of links to files or to external locations, JavaScript events (Ajax anyone?) or even Flash events (but who is crazy enough to use Flash anyway?).

::Read more

Mon, 18 Jun 2007

Bye, Google sandbox!

Today I'm finally out of Google's Sandbox.

Google has this system called the sandbox where new pages go into for 6 months, in order to prevent scammers/spammers from resurrecting dummy pages - and scoring well in Google - all the time.
During these 6 months a page will score very bad in search results, even if it should rate very well for the specific keywords.

Smart people will look at my first post, dated 03/04/2007, but keep in mind that before this blog existed I already had a dummy page with my name on (the keywords I want to score on) as soon as I could because back then I already knew I wanted to put a blog here and I wanted to get out of the sandbox as soon as possible.

I know some people who kept forgetting my URL and didn't find it in Google, well, you will find me now!

Sat, 02 Jun 2007

You can't make bits harder to copy

I just watched Cory Doctorow's talk which is part of the Authors@Google series on youtube.

He made some great points about where the (music) industry gets/does it wrong and about some fundamental flaws in our law systems (especially with regards to copyright). All of which are of course results of the challenges imposed by the "information age". (which I also introduced in

::Read more

Tue, 29 May 2007

Drag 'n drop tutorial with the CakePHP 1.2 Ajax helper, Prototype framework and Scriptaculous library


During the development of my thesis I wanted to create a drag 'n drop interface. But I never did anything like that, I never used CakePHP's Ajax helper and neither made I ever use of more advanced functionalities of Scriptaculous/Prototype. Hell I even never touched Ajax before this!

Although there are some basic CakePHP/Ajax tutorials out there, I still had a hard time because some knowledge about Ajax (in CakePHP) was assumed in all of those. After a lot of googling I even found a tutorial called CakePHP: Sortable AJAX Drag & Drops - The Basics
"Perfect!" I thought, until after staring at the article for a long while and I started to notice nowhere in the article "$ajax->drag", "$ajax->drop" or "$ajax->dropRemote" is used. (those are calls on the CakePHP Ajax helper to enhance objects to become draggable, or to become a dropbox where draggables can be dropped into). So the only more or less suited tutorial about drag 'n drop was actually about sorting and didn't use the drag/drop function calls at all. Even though it contains very useful information.

Long story short: I finally got it working (thanks to Krazylegz and kristofer and possibly others too, it has been a while so I may forget someone ;-), and learned a lot in the process. I will share what I learned with you guys so that hopefully it's a bit easier for you then what I had to go through.

::Read more

Open source en softwarepatenten vanuit een ethisch perspectief

Voor school moest ik een ethisch verslag schrijven.
Hier moesten 3 elementen in voorkomen:

  • Inleidende tekst / Persoonlijke reflectie over beroeps- en bedrijfsethiek
  • Ethische beschouwingen bij het eindwerk
  • Uitgebreidere verhandeling met ethische beschouwingen over een vraagstuk naar keuze uit het domein van wetenschap, techniek, beroeps- en bedrijfsleven. In mijn geval is dit vraagstuk naar keuze "open source en softwarepatenten" geworden

Zoals sommigen onder jullie weten vind ik ethiek een heel belangrijk aspect in het leven en ik ben dan ook blij dat ik het issue van open source en softwarepatenten verder heb kunnen uitdiepen, want dit is iets waar ik in geinteresseerd ben.

Mijn werkje kan je hier downloaden. Lees het gerust door en laat me weten wat je ervan vindt!

Een heel interessant artikel dat ik ben tegengekomen (wel, eigenlijk mailde de docent van Ethiek het me door ;-) is Community Ethics and Challenges to Intellectual Property geschreven door Kaido Kikkas. Neem ook dit zeker door want die kerel slaat de nagel op de kop!
Bovenstaande tekst heeft ook het levensverhaal van Edward Howard Armstrong aangehaald. Toen ik hierover las heb ik direct zijn hele biografie op wikipedia doorgenomen. In het kort komt het er op neer dat die persoon enkele geniale uitvindingen heeft gedaan, maar het bedrijf waarvoor hij werkte heeft via dure rechtzaken hem in de zak gezet en geruineerd. Uiteindelijk ging zijn huwelijk en zijn hele leven eraan kapot. En dan heeft hij zelfmoord gepleegd.
Ongelooflijk hoe ver het egoisme van een bedrijf kan gaan. Laten we dan ook leren uit de geschiedenis en kritisch staan tegenover de nieuwe elementen waar we tegenover staan zoals intellectuele eigendommen en patenten. We willen toch niet dezelfde fouten maken?

In mijn zoektocht naar meer info over deze man stootte ik op deze pagina: doomed engineers. Neem deze even door!

Last but not least wil ik bij deze ook iedereen de film Revolution OS aanraden. Het is al lang geleden dat ik hem gezien heb, maar hij geeft een prima beeld over de opgang van vrije software (en Linux in het bijzonder), en je leert er vooral ook bij over enkele onethische praktijken van softwaregiganten, you know which one I mean...

Thu, 24 May 2007

Thesis finished

Yesterday, after a night of searching and fixing spelling errors, things that could be better explained and other small details,
I got my thesis printed and delivered the six books to my school.

::Read more

Tue, 17 Apr 2007

I just became a "System & Network Architect"

I just signed my contract at Incrowd, the company behind sites such as redbox and facebox.

I will be working there in a team of all young, enthusiastic people. Among those, some people are already familiar to me: my old friend Lieven (we've played in a band together but kept in touch afterwards) and my ex-classmate Jurriaan. Both of them love their jobs btw :-).

My official title is "System & Network architect".
Things I will be doing there is keeping the "lower level" (hardware, network, databases) secure, stable and performing well.

::Read more

Tue, 10 Apr 2007

Debian changing route: the end of the perfect server linux distribution?

From the very little experience I have with Debian, and from the stuff I've been reading about it, I think I can safely say Debian has always been a special distribution: packages always take very long to get into the stable tree, because Debian wanted to be a rock solid system where packages go through a lot of testing. ("We release it when it's done") The end result is a distro where you don't have the latest software, neither as much flexibility as, say Gentoo or Arch: You'd many times need to adapt your way of doing things to the "Debian way" (or be prepared to look for help in really obscure places and probably break things) but the end result is a stable distro where everything works very decently. That, combined with no licensing fees (unlike for example Red hat), make it the perfect choice for a server in small companies, where money is more important then features such as professional support or official certifications.

However, it seems like Debian is taking a route that will make it lose it's advantages over other distributions in the server market:

::Read more

Sat, 07 Apr 2007

Figuring out CakePHP's new AuthComponent

In the Cake community, there has always been much interest in authentication/authorization systems. The issue of authentication has been addressed in several add-ons provided by the community, such as DAuth (written by me), OthAuth (written by Crazylegs) and many others.

However, one of the additions to the 1.2 branch which is currently in active development , is a built-in auth module. A module that isn't finished yet but it sure is worth it looking at. (In fact I'm thinking about making a new dAuth version built on cake's own auth system.). As most bakers know, there is very little information about the 1.2 branch in general, and the auth component in specific. So what I will try to do, is delve in the code, mess with it, and explain my findings in this post. For this first post it will be more trying to decipher the source code, messing with it will probably for a little later on.

::Read more

Mon, 19 Mar 2007

Kwartee 4 2007 verslag

Dit weekend (17-18 maart) ben ik naar Kwartee 4 geweest.
Kwartee weekends worden georganiseerd door Formaat (vroeger bekend als VFJ) en ging door in vormingscentrum destelheide te Dworp (dichtbij Halle, ten zuiden van Brussel).
Twee man sterk (Steven en ik) vertegenwoordigden we jeugdhuis SjaTOo.

::Read more

Thu, 15 Mar 2007

Fosdem 2007 review

Every year, during a special weekend in February, the University Libre of Brussels suddenly becomes a little more geeky.
It's that time of the year when many European (and some inter-continental) colleagues join us at
Fosdem: the Free and Open source Software Developers' European Meeting (more info here).

::Read more

Wed, 14 Mar 2007

My favorite bash tricks

Hello everyone.
This post is about bash, the shell providing so many users easy access to the underlying power of their system.
(not bash the quote database, although i really like that website too ;-) )
Most people know the basics, but getting to know it better can really increase your productivity. And when that happens, you might start loving bash as much as I do ;-)

I assume you have a basic knowledge of bash, the history mechanism, and ~/.bash* files.
So here they are, my favorite tricks, key combo's and some bonus stuff:


  • "cd -" takes you back to the previous directory, wherever that was. Press again to cycle back.
  • putting arguments between braces {like,so} will execute the command multiple times, once for each "argument". Bash will make the cartesian product when doing it multiple times in 1 expression. Some less-obvious tricks with this method are mentioned here
  • HISTIGNORE : with this variable you have control over which things are being saved in your history. Here is a nice explication. Especially the space trick is very useful imho.
  • CD_PATH : Here is a great explanation ;-)
  • readline (library used by bash) trick: put this in your ~/.inputrc (or /etc/inputrc) :
    "\e[5~": history-search-backward
    "\e[6~": history-search-forward

    This way you can do *something*+pageup/pagedown to cycle through your history for commands starting with *something*
    You can use the up/down arrows too, their codes are "\e[A" and "\e[B"

  • for more "natural" history saving behavior (when having several terminals open): put this in .bash_profile:

    PROMPT_COMMAND='history -a'

    (write each command separately in a new entry, instead of all at shell exit).
    And type

    shopt -s histappend

    to append instead of overwrite. (this might be default on some distro's. I think it was on Gentoo)


  • ctrl+r : search through your history. Repeatedly press ctrl+r to cycle through hits.
  • ctrl-u : cut everything on the current line before the cursor.
  • ctrl-y : paste text that was cut using ctrl-u. (starting at the cursor)
  • !$: equals the last word of the previous command. (great when performing several operations on the same file)

Bonus material

  • Bash completion, an add-on for bash which adds completion for arguments of your commands. It's pretty smart and configurable. (it's in portage, and probably in repos of all popular distros out there)
  • This script provides you an interface to the rafb pastebin!
  • Recursively delete all .svn folders in this folder, and in folders below. "find . -name .svn -print0 | xargs -0 rm -rf"
  • Recursively count number of files in a directory: "find -not -type d | wc -l"


Those were all important tricks I'm currently using. On the web you'll find lots more useful tips :-).
If that still isn't enough, there is also man bash :o

With aliases and scripts (and involving tools like sed or awk) the possibilities become pretty much endless. But for that I refer to and your favorite web search engine.

Sun, 04 Mar 2007

Hello world!

Finally, my own website...
I already wanted to get this up for a long time. My initial idea was writing (and styling) it all from scratch using the marvelous CakePHP framework along with an authentication system i wrote, dAuth.
However, due to my lack of time I decided to use the excellent drupal platform, of which I'm quite sure will get the job done equally well, while drastically liberating my time, so I can invest it in other projects :-)
Dries Buytaert's talk on fosdem this year really helped on making that decision ;-)

So, what will this site be about?

  • me
  • my interests
    • Free & Opensource software, and the thoughts/ideals behind it
    • PHP scripting/programming (I like C(++), bash and j2se too but I'm not as good at it as I am at php)
    • Audio recording/mixing/production
    • Drumming, one of my greatest hobbies
    • Music, bands, movies,... I like
    • productivity (TDD, automation scripts, shell/DE tweaks, ...)
    • ethics and philosophy, these aspects are really important in my life
    • Jeugdhuis SjaTOo, our local youth club

Now let's get started ;-)