Dieter on the web

Profiling a process run

black-box behavior testing processes/daemons

Simple-black-box talk @ Devopsdays NYC

More random thoughts about Devopsdays NYC

• I'm getting tired of people on stage making a big revelation out of adding an index to a database column. This happens too often at web-ops/devops conferences, it's embarrassing. But at least it's not like the "how we made our site 1000x faster"-style Velocity talks that should have been named "caching and query optimization for newbies"
• Paperless post confirms again they got their act together and keeps us up to date with their great work. Follow them.
• Knights of the Provisioning Round Table - Bare Metal Provisioning was mostly (to my surprise) 4 individuals presenting their solution instead of a real round-table, but (to my surprise again) they were not as similar/repetitive as I expected and the pros/cons of all solutions were compared more in depth than I dared to hope. I covered dell crowbar before and like it, though I wonder when this thing is actually gonna be reliable.
• Dave Zwieback and John Willis gave hilarious talks
• Tried to start an open space discussion around collaboration patterns and anti-patterns, which I think is a very interesting subject, because how individuals in a team collaborate is crucial to success, but yet very little is written about it (that I could find). I would hope we can distill the years of aggregate experience of people into concise patterns and anti-patterns and document how (not) well they work for development styles (such as agile/scrum), team size, company structure/hierarchy, roadmap/technical debt pressure, etc. And especially in light of any of these things changing, because I've found people can be very change-resistive.
• DevOps At Obama for America & The Democratic National Committee was good, I thought it would be a rehash of what was said at Coding Forward New York City: Meet the Developers Behind the Obama Campaign but there were a bunch of interesting insights about state of the art technology in here (mostly Amazon stuff)
• A bunch of talks where the same could have been said in half the time, or less

• Librato is quite cool. It's basically how my open source tool graph-explorer would look like after finishing a bunch of TODO's, combining it with graphite, polishing it all up, and offering it as a hosted solution. I find it interesting if this is a successful business with only such a limited scope
• Even cooler is datadog. It goes beyond just metrics and doesn't just provide hosted graphing, it provides a solution for a philosophy that aims for a centralized insight of all your operational data, related collaboration and prioritized alerts that are to the point. They get a lot of things right, the open source world has some catching up to do

In web operations, mostly when troubleshooting but also for capacity planning, I often find myself having very specific information needs from my time-series, and these information needs vary a lot over time. This usually means I need to correlate or compare things that no one anticipated. Things that relate to specific machines, specific services across machines, or a few specific metrics of which only the ops team knows how they are related and cross various scopes (application, network, system, etc).
I should have an easy way to filter metrics by any information in the metric's name or values.
I should be able to group metrics into graphs the way I want. (example: when viewing filesystem usage of servers, I should be able to group by server (one graph per server listing the filesystems, but also by mountpoint to compare servers on one graph).
I should be able -with minimal effort- to view metrics by their gauge/count value, but also by their rate of change and where appropriate, as a percentage of a maximum (like diskspace used).
It should be trivial to manipulate the graph interactively (toggling things on/off, switching between lines/stacked mode, inspecting datapoints, zooming, ...).
It should show me all events, colorcoded by type, with text description, and interactive so that it can use hyperlinks.
And most of all, the code should be as simple as possible and it should be easy to get running.

Dashboards which show specific predefined KPI's (this covers most graphite dashboards) are clearly unsuitable for this use case. Template-based "metric exploration" dashboards like cacti and ganglia are in my experience way too limited. Graph composing dashboards (like the stock graphite one, or graphiti) require much manual work to get the graph you want. I couldn't find anything even close to what I wanted, so I started Graph-Explorer.

The approach I'm taking is using plugins which add metadata to metrics (tags for server, service, mountpoint, interface name, ...), having them define how to render as a count, as a rate, as a percent of some max allowed value (or a metric containing the max), and providing a query language which allow you to match/filter metrics, group them into graphs by tag, and render them how you want them. The plugins promote standardized metric naming and reuse across organisations, not in the least because most correspond to plugins for the Diamond monitoring agent.

Furthermore, because it uses my graphitejs plugin (which now btw supports flot as a backend for fast canvas-based graphs and annotated events from anthracite) the manual interactions mentioned earlier are supported or at least on the roadmap.

Being active as both a developer and ops person in the professional life, and both an open source developer and packager in my spare time, I noticed some common ground between both worlds, and I think the open source community can learn from the Devops movement which is solving problems in the professional tech world.

For the sake of getting a point across, I'll simplify some things.

First, a crash course on Devops...

::Read more

metrics often seem to lack details, such as units and metric types

• the unit a metric is measured in (bits, queries per second, jiffies, etc)
• the "type" (a rate, an ever increasing counter, gauge, etc)
• the scale/prefix (absolute, relative, percentage, mega, milli, etc)

• what

  what is being measured?: bytes, queries, timeouts, jobs, etc

• target_type

  must be one of the existing clearly defined target_types (count, rate, counter, gauge)
  These match statsd metric types (i.e. rate is per second, count is per flushInterval)

tree navigation/querying is cumbersome, metrics search is broken. How do I organize the tree anyway?

no interactivity with graphs

screenshot:

events lack text annotations, they are simplistic and badly supported

• anthracite is designed specifically to manage events.
  It brings extra features such as different submission scripts, outage annotations, various ways to see events and reports with uptime/MTTR/etc metrics.
• timeserieswidget displays your events on graphs along with their metadata (which can be just some text or even html code).
  this is where client side rendering shines

screenshots:

cumbersome to compose graphs

• interactive composing: with the graphite composer, you navigate through the tree and apply functions. This is painfull, dashboards like descartes and graphiti can make this easier
• use a dashboard that uses predefined templates (gdash and others) They often impose a strict navigation path to reach pages which may or may not give you the information you need (usually less or way more)

• GE gives you a query interface on top of structured_metric's tag space. this enables a bunch of things (see above)
• you can yield arbitrary targets for each metric, to look at the same thing from a different angle (i.e. as a rate with `derivative()` or as a daily summary), and you can of course filter by angle
• You can group metrics into graphs by arbitrary tags (e.g. you can see bytes used of all filesystems on a graph per server, or compare servers on a graph per filesystem). This feature always results in the "wow that's really cool" every time I show it
• GE includes 'what' and 'target_type' in the group_by tags by default so basically, if things are in a different unit (B/s vs B vs b etc) it'll put them in separate graphs (controllable in query)
• GE automatically generates the graph title and vertical title (always showing the 'what' and the unit), and shows all metrics' extra tags. This also gives you a lot of inspiration to modify or extend your query

limited options to request a specific time range

Referenced projects

• anthracite:
  event/change logging/management with a bunch of ingestion scripts and outage reports
• timeserieswidget:
  jquery plugin to easily get highly interactive graphite graphs onto html pages (dashboards)
• structured_metrics:
  python library to convert graphite metrics tree into a tag space with clearly defined units and target types, and arbitrary metadata.
• graph-explorer:
  dashboard that provides a query language so you can easily compose graphs on the fly to satisfy varying information needs.

Future work

• I spoke with Michael Leinartas at Monitorama (and there's also a launchpad thread). We agreed that native tags in graphite are the way forward. This will address some of the pain points I'm already fixing with structured_metrics but in a more native way. I envision submitting metrics would move from:

  stats.serverdb123.mysql.queries.selects 895 1234567890
  

  to something more along these lines:

  host=serverdb123 service=mysql type=select what=queries target_type=rate 895 1234567890
  host=serverdb123 service=mysql type=select unit=Queries/s 895 1234567890
  h=serverdb123 s=mysql t=select queries r 895 1234567890
  

• switch Anthracite backend to ElasticSearch for native integration with logstash data (and allow you to use kibana)

Introduction

Graphite can show events such as code deploys and puppet changes as vertical markers on your graph. With the advent of new graphite dashboards and interfaces where we can have popups and annotations to show metadata for each event (by means of client-side rendering), it's time we have a database to track all events along with categorisation and text descriptions (which can include rich text and hyperlinks). Graphite is meant for time series (metrics over time), Anthracite aims to be the companion for annotated events.
More precisely, Anthracite aims to be a database of "relevant events" (see further down), for the purpose of enriching monitoring dashboards, as well as allowing visual and numerical analysis of events that have a business impact (for the latter, see "Thoughts on incident nomenclature, severity levels and incident analysis" below)
It has a TCP receiver, a database (sqlite3), a http interface to deliver event data in many formats and a simple web frontend for humans.

design goals:

• do one thing and do it well. aim for integration.
• take inspiration from graphite:
  • simple TCP protocol
  • automatically create new event types as they are used
  • run on port 2005 by default (carbon is 2003,2004)
  • deliver events in various formats (html, raw, json, sqlite,...)
  • stay out of the way
• super easy to install and run: install dependencies, clone repo. the app is ready to run

About "relevant events"

I recommend you submit any event that has or might have a relevant effect on:

• your application behavior
• monitoring itself (for example you fixed a bug in metrics reporting. it shouldn't look like the app behavior changed)
• the business (press coverage, viral videos, etc), because this also affects your app usage and metrics.

Formats and conventions

The TCP receiver listens for lines in this format:

<unix_timestamp> <type> <description>

There are no restrictions for type and description, other than that they must be non-empty strings.
I do have some suggestions which I'll demonstrate through fictive examples;
but note that there's room for improvement, see the section below)

# a deploy_* type for each project
ts deploy_vimeo.com "deploy e8e5e4 initiated by Nicolas -- github.com/Vimeo/main/compare/foobar..e8e5e4"
ts puppet "all nodes of class web_cluster_1: modified apache.conf; restart service apache"
ts incident_sev2_start "mysql2 crashed, site degraded"
ts incident_sev2_resolved "replaced db server"
ts incident "hurricane Sandy, systems unaffected but power outages among users, expect lower site usage"
# in those exceptional cases of manual production changes, try to not forget adding your event
ts manual_dieter "i have to try this firewall thing on the LB"
ts backup "backup from database slave vimeomysql22"

Thoughts on incident nomenclature, severity levels and incident analysis

I need your help

• monitoring dashboard for graphite that gathers events from anthracite, can show metadata, and can mark a timeframe between start and stop events
• plugings for puppet, chef to automatically submit their relevant events
• a better web UI and actually provide features to do statistics on events and analysis such as TTD, TTR, with colors for severity levels etc

Client-side rendering of charts as opposed to using graphite's server side generated png's allows various interactivity features, such as:

• interactive realtime zooming and panning of the graph, timeline sliders
• realtime switching between various rendering modes (lines, stacked, etc)
• toggling certain targets on/off, reordering them, highlighting their plot when hoovering over the legend, etc
• basic data manipulation, such as smoothing to see how averages compare (akin to movingAverage, but now interactive)
• popups detailing the exact metrics when hoovering over the chart's datapoints
• popups for annotated events. (a good use for anthracite).

There are many graphite dashboards with a different focus, but as far as plotting graphs, what they need is usually very similar: a plot to draw multiple graphite targets, a legend, an x-axis, 1 or 2 y-axis, lines or stacked bands, so I think there's a lot of value in having many dashboard projects share the same code for the actual charts, and I hope we can work together on this.

• Profiling and behavior testing of processes and daemons, and Devopsdays NYC
• Graph-Explorer: A graphite dashboard unlike any other
• Client-side rendered, highly interactive graphite charts for all
• Dell crowbar openstack swift
• Anthracite, an event/change management database to track what has a business and/or operational impact and show annotated events on dashboards
• Histograms in statsd, and graphing them over time with graphite
• stop abusing SI prefixes cause 1000 ≠ 1024
• What the open source community can learn from devops