arch bash cakephp dauth devops drupal fosdem foss git golang information age linux lua mail monitoring music n900 netlog openstack photos php productivity python real life thesis travel uzbl vimeo web2.0
In light of the work and discussions around supporting Nilfs2 and Btrfs on Arch Linux and its installer AIF,
I've shared some AIF filesystem code design insights and experiences on the arch-releng mailing list.
This is some hard to understand code. Partly because it's in bash (and I've needed to work around some limitations in bash),
partly because there is some complex logic going on.
I think it's very useful material for those who are interested (it can also help understanding the user aspect),
so I wanted to share an improved version here.
On a related topic: I proposed to do a session at Fosdem 2011/"distro miniconf" about simple (console based) installers for Linux,
and how multiple distributions could share efforts maintaining installation tools, because there are a lot of cross-distribution concerns
which are not trivial to get right (mostly filesystems, but I also think about clock adjustments, bootloaders, etc).
Already several distro's use the (or a fork of) the Arch installer, for example Pentoo,
but I think cooperation could be much better and more efficient.
my acronyms for this text:
"Normal" FS'es ("do something on the BD represented by DF /dev/foo, so that you can then call `mount /dev/foo $somedir`") are trivial to add to aif.
Basically you just need to tell aif the name of the filesystem, and which commands and arguments it needs to invoke to create it and add a label to it.
Nilfs2 falls in this category. So do ext2/3/4, xfs, jfs, reiserfs, etc. (Nilfs is now supported and new archiso testbuilds are available)
"Complex" FS'es (which yield a new DF for a DM BD, which can span multiple underlying BD's, etc) are more difficult, and I'll explain why.
Btrfs falls in this category. So do LVM, dm_crypt and softraid. In aif terminology anything you put on top of a BD is a FS. This is not always technically correct, but it's not far-fetched and avoids needless complication. For example softraid would be a FS you put on top of one or more BD's, and which itself yields a new BD on top of which you can put something else.
In the same way I call a VG a FS being applied on top of a PV BD, which itself results in a new BD which can host multiple LV FS'es, which in return yield new BD's which can host other FS'es. Currently AIF provides support for lvm and dm_crypt, but not softraid or Btrfs.
How aif works is this: it uses a "model" that represents how your DF/FS structure will look like.
I personally usually configure my hard disk like this:
a boot partition, and a partition on which i do dm_crypt, which results in a DM BD, which I make a PV, then put a VG on
it, which contains multiple LV's, one for swap, and two containing
some FS'es which get mounted as / and /home.
You can see that model on the bottom ($BLOCKDATA) of the example file "fancy-install-on-sda" for automatic installations, included with aif.
You might have noticed in the installer - if you do it interactively - how you first configure all
your filesystems in the dialog interface, but only after confirming, it
does all the required actions, step by step. Actually the dialog-based configuration helper will just generate a textfile in the same format as $BLOCKDATA and the processing code is the same for interactive and automatic installs.
Since in the config file for an automatic install you could define your FS's in abitrary order, aif
figures out the dependencies and processes things in the right order.
With the example given above aif will parse the text and will figure out the order of creation: first partition (obviously), then create the dm_crypt, then the PV, then the VG,
then the LV's, then the FS'es on those LV's)
then mount all mountpoints in the right order (first /, then /home and /boot)
On rollback (when user changes his mind, an error occured - usually because of misconfiguration like a too big LV) everything I just explained happens in reverse.
This allows users to tune their config (or try something completely different), without the installer breaking with errors like "This device is already marked as encrypted" or "VG already exists"
I choose this model-based approach initially because I wanted to get rid
of the ugly, hacky original installer code, but still provide a lot of
control through the nice dialog interfaces. And interfaces that work fast (not causing you to wait between every step because it's creating the FS you just defined).
Perhaps most importantly, since my main goal was automautic installs where you could just specify how you wanted your
FS/BD structure to look like (not a series of commands), relatively little extra code was needed. (this is also what separates AIF from FAI&debian installer. On Debian the interactive and automatic installers are different projects. On Arch they are just different extensions for the same codebase)
Because of all this, I have sweared quite a bit over the last few
years, wondering if bash really was a good choice. But using an external tool to work with a text-base graph/tree datastructure (where nodes can have some properties) will probably remove the big nuisance. Also, Because bash v4 has associative arrays (finally), I can clean up a bunch of other code as well.
I've pondered whether we should just let users do everything on the commandline (like Gentoo), or provide a minimal layer of
abstraction, like provide some scripts which they can modify that setup
a system in a certain way. (for example, basically a series of mkfs; mount;pacman; calls)
Another alternative would be to just make the user choose between a series of "common setups" and make them answer some questions for the choosen setup.
This is how the old installer did it, and afaik archboot still does it, but it doesn't scale well with more and more possibilities.
Actually, aif still contains the optional "autoprepare" method, which is in fact a simple wizard for a simple setup.
I guess it's a tradeoff between making it easy for users and not overloading the brain of people who want to hack on the installer.
The approach which afaik has always been taken by Arch is to make the installer hold the hands of the user.
When creating aif, I've choosen to stick to that concept. And frankly I still like the idea. It may be hard to maintain,
but as a user you can finish all your installs and have a lot of options, but still use very minimal keystrokes (and mental energy).
Like mentioned earlier, softraid hasn't been implemented yet in aif, nor btrfs.
I would need to know the most common/recommended use cases, and figure out the best way to implement them.
Btrfs might be relatively easy, if i can implement it like i did lvm. But it seems like a great FS and I don't want to provide only-half-assed support for it.
Either way, refactoring the datastructure is something that will needs to be done at some point, especially for softraid.
I hope this article was a bit understandable. And if you have any advice, please share :)
The actual blockdevices-filesystems library of AIF is here:
And the menu's are here:
https://github.com/Dieterbe/aif/blob/2ee11ef334983fc68352364025ba5a97d795b95e/src/core/libs/lib-ui-interactive.sh" (the most interesting functions are interactive_filesystems for the main FS menu, and interactive_filesystem which handles definition of a specific FS)