arch bash cakephp conf dauth devops drupal foss git golang information age life linux lua mail monitoring music mysql n900 netlog openstack perf photos php productivity python thesis travel uzbl vimeo web2.0

RRDtool: updating RRA settings and keeping your collected data

When you use rrdtool, it can happen that you first create your databases, then collect a whole bunch of data and decide later you want more accuracy/longer periods.
Especially when using zenoss (the monitoring solution I mostly work with at Kangaroot), which uses very conservative RRD settings by default (i.e. 5-minute intervals for only the first 50 hours). Zenoss provides a way for you to change the way RRD's are created, but not to apply those settings to already existing RRD files, which I found out *after* I started monitoring everything ;)

rrdresize can help: it (just) adds or removes locations for rows.
In my case it was not good enough because zenoss uses a variety of resolutions (step sizes), and so if you add rows to all of them rrdtool - when graphing - will often pick a higher resolution RRA that just had rows added (and hence contain unknown values), even though you have the values, albeit at a lower resolution.

So you need a way to update all rows in the RRA's.
I found a perl tool that does just that. (I think, I didn't study all details). So, you install that in your /home/zenoss for instance and then you run the following script, which creates new rrd files with the new settings and uses the perl script to copy all data into it.

#!/bin/sh
# invoke me like this:
# find /usr/local/zenoss/zenoss/perf/ -name '*.rrd' -exec ./newrrd.sh {} \; >> newrrd-logfile

file=$1
backupdir=/home/zenoss/rrds-backup
newdir=/home/zenoss/rrds-new
[ -d "$backupdir" ] || mkdir -p "$backupdir" || exit 2
[ -d "$newdir"    ] || mkdir -p "$newdir" || exit 2
[ -f "$file"      ] || exit 3

echo "Processing $file .."
base="`basename "$file"`"
[ ! -f "$backupdir/$base" ] || mv "$backupdir/$base" "$backupdir/$base".old || exit 4
cp "$file" "$backupdir/$base"
cd "$newdir" && rrdtool create "$base" \
--step '300' \
--start '1230768000' \
'DS:ds0:GAUGE:900:U:U' \
'RRA:AVERAGE:0.5:1:122640' \
'RRA:AVERAGE:0.5:6:55536' \
'RRA:MAX:0.5:6:55536'
/home/zenoss/rrdremove.pl "$backupdir/$base" "$base" | grep -v 2009 # hide some output
cp "$base" "$file" || exit 5
echo "Done"

Oh and btw, rrdwizard is a cool webapp when you're feeling too lazy/have forgotten how to write rrdtool commands

Comments

After modifying the script for my need, adding multiple ds(s), i did some comparison of the graphs of before and after.

i see the graph after is much more smoother than before and there are noticeable difference in average and max. I guess this is kind of expected ?
do you agree?

Well, the point was indeed to have higher resolution data, as Zenoss is very conservative (low-res after 50 hours) by default.
There shouldn't be big differences in average/max.
Although i have to add when i ran the script i noticed there were some graphs for which the data had become incorrect. But most of the graphs were fine.
Rely on your own judgement to see if it still looks correct afterwards. And make a backup. ;)

Btw, I can integrate any useful patch to rrdmove you might send me :-)

Has anybody been able to get this to work on a DERIVE rrd type? Everytime I try, the rrd gets created, but with either blanked-out values, or the following values zenoss reads are WAY off-base (by at least a factor of ten... definitely not the right readings).
On the off-chance that somebody else is looking for an answer to the same question as mine, I found the solution.

Use this tool to generate the rrds. Let them get created as GAUGE types. Then, once they're created, use the 'tune' option of rrdtool to change the type to DERIVE or COUNTER or whatever else you want.

In all the RRDs I updated, I have a 5min gap in the graphs (300s step time), which I suspect is just the time needed for things to start making sense to rrdupdate again.

Other than that, it works flawlessly.


Name:


E-mail:


URL:


Comment:


What is the first name of the guy blogging here?


This comment form is pretty crude. Make sure mandatory fields are entered correctly.
Basic html tags (a,i,b, etc) are allowed, others are sanitized