Dieter on the web

I'm back from Canada/Archcon, and it was great. I've been in Toronto for 11 days, and visited Montreal for 3 days.

Archcon

Archcon was small (20-ish people). (That's what you get for doing it in Canada ;), but very nice.
Interesting talks, informal, good vibe, decent logistics and catering.
This year it happened because Dusty and Ricardo actually just wanted to have a conference without worrying too much about the attendance,
next year we should do it again because Arch (conferences) rock(s), and because we need more visitors. More central locations such as Seattle and Europe have been suggested.
Either way, next year both Judd (founder) and Aaron (current overlord) should be there. (this year they both had lame excuses like family reunions and "almost getting married". Congrats btw, Aaron!)

It was an absolute pleasure to meet some more of my fellow devs, and users.
Here is a pic from the group (unfortunately, a few are missing)

::Read more

As mentioned earlier, I'll be at Archcon in Toronto in a few weeks.
It's a very small conference, and the first of its kind. At the last FrOSCon we have been playing with the idea to hold an informal Arch conference in Europe, but those were just ideas. Dusty and Ricardo beat us with an actual implementation.
This is great, and one of the milestones in Arch Linux history. Which is why I want to be there and help making it better.

::Read more

I'll be at fosdem - 10th edition - again this year.

I'll be presenting a lightning talk about uzbl.
Also, Arch Linux guys Roman, JGC, Thomas and me will hang out at the distro miniconf. We might join the infrastructure round-table panel, but there is no concrete information yet.

More stuff I'm looking forward to:

::Read more

We're working hard on a new Arch release. (should be done by froscon)

Amongst the slew of fixes and improvements there are also some cool new things I'm working on.
First of all, I worked more on the automatic installations. Now you can easily install an LVM based Arch system on top of dm_crypt for example.
You type this command:

And bam you have a complete working system with LVM, dm_crypt etc all set up. You just need to change your keymap, hostname, network config and such (or, configure that on the beforehand in the config file for AIF)

Another thing I started working on is a very simple test suite.
Basically, when launching a test, the following steps are invoked

• installation of an arch system with aif's automatic procedure using a certain config file
• installation of a verification script onto the target system and configuration of the target to run the script on boot (DAEMONS variable in /etc/rc.conf)
• if aif ended succesfully: automatic reboot.. and tada!

The verification script will check things like availability (and size) of LVM volumes, amount of swap space, keyboard layout, network and so on.
Here's a picture of a rough first version:

Stay tuned!

Recently I started thinking about writing my own automatic installer that would set up my system exactly the way I want.
(See http://dieter.plaetinck.be/rethinking_the_backup_paradigm_a_higher-level...)

I looked at the official Arch install scripts to see if I could reuse parts of their code, but unfortunately the code was just one big chunk of bash code with the main program and "flow control" (you must first do this step, then that), UI-code (dialogs etc) and backend logic (create filesystems, ...) all mangled up and mixed very closely together.
Functionality-wise the installer works fine, but I guess the code behind it is the result of years of adding features and quick fixes without refactoring, making it impossible to reuse any of the code.

So I started to write AIF: the Arch Linux Installation Framework

::Read more

So, the Arch Linux 2009.08 release is now behind us, nicely on schedule.
I hope people will like AIF because it was a lot of work and we didn't receive much feedback. I personally like it to apply my fancy backup restoration approach.
But I'm sure if more people would look at the code we would find quite some design and implementation things that could be improved. (With uzbl I was amazed how much difference it can make if many people all have ideas and opinions about every little detail)

Later this week I'm off to the Counting Cows festival in France, and the week after that (august 22-23) I'm going to FrOSCon in Germany where I will meet some of my Arch Linux colleagues in real life, which I'm really looking forward to.

If anyone wants a ride to froscon let me know. But note I'll try to maximize my time there (leave saturday early and come back late on sunday. I even took a day off on monday so I might stay a day longer if I find more interested people to hang out there)

I don't think I've ever seen so much anxiety/impatience/hope/buzz for a new Arch Linux release. (this is because of 2.6.28 with ext4 support).
The last release was 6 months ago, which is not so good.. also the arch-installer project has been slacking for a while. But the Arch devs have been very busy and many things going on. You know how it goes...

That's why some new people have stepped up to help out on a new release:
Today, we are on the verge of a 2009-01 release (though that has been said so many times lately ;-) and together with Aaron we have started a new project: the Arch Linux Release Engineering team.
Members of this team are Aaron himself, Gerhard Brauer and me.

Our goals:

• coordinated releases following the rhythm of kernel releases (that's a release every 2-3 months baby!).
• anticipate the availability of the kernel in the testing repo's instead of having to wait for it to go to core before building alpha/beta images
• migration to AIF as the new Arch installer (woot!)
• testing. Leveraging the possibilities of AIF as an unattended installer, we should be able to script full installations and health checking of the resulting system.
• involving the community more? releasing "testing iso's" ? not sure about that. we'll see...

We also have:

• arch-releng mailing list
• Release Engineering bugtracker

Oh yeah AIF is mirrored @ http://projects.archlinux.org/ and available packaged in the official repo's!

Today at Fosdem 2011 I did my talk Can we build a simple, cross-distribution installation framework? Basically, using the Arch Installation Framework as a starting point, along with the notion that most of the code is actually not Arch-specific I adressed other distros to check if there was any interest in sharing workload on the distribution-agnostic aspects of the framework. If other distros with a similar philosophy of little-abstractions/KISS would join, we would all reap the benefits of a simple, yet quite featureful installer. There was some interest, so we'll see what happens.

• slides
• video

This is a question I get asked a lot recently. The latest official images are a year old. This is not inherently bad, unless you pick the wrong mirror from the outdated mirrorlist during a netinstall, or are using hardware which is not supported by the year old kernel/drivers. A core install will yield a system that needs drastic updating, which is a bit cumbersome. There are probably some other problems I'm not aware of. Many of these problems can be worked around (with 'pacman -Sy mirrorlist' on the install cd for example), but it's not exactly convenient.

Over the past years (the spare time in between the band, my search for an apartment in Ghent and a bunch of other things) I've worked towards fully refactoring and overthrowing how releases are being done. Most of that is visible in the releng build environment repository. Every 3 days, the following happens automatically:

• packages to build images (archiso) and some of which are included on the images (aif and libui-sh) get rebuilt. They are actually git versions, the latter two have a separate develop branch which is used. Normal packages get updated the normal way.
• the images are rebuilt, and the dual images get generated
• the images, the packages and their sources are synced to the public on http://releng.archlinux.org

Actually things are bit more involved but this is the gist of it. All of this is now run on a dedicated VPS donated by airVM.

I never really completed the aif automatic test suite, somewhere along the way I decided to focus on crowdsourcing test results. The weight of testing images (and all possible combinations of features) has always been huge, and trying to script tasks would either get way complicated or insufficient. So the new approach is largely inspired by the core and testing repositories: we automatically build testing images, people report feedback, and if there is sufficient feedback for a certain set of images (or a bunch of similar sets of images) that allows us to conclude we have some good material, we can promote the set to official media.
The latest piece of the puzzle is the new releng feedback application which Tom Willemsen contributed. (again: outsourcing FTW). It is still fairly basic, but should already be useful enough. It lists pretty much all features you can use with archiso/AIF based images and automatically updates the list of isos based on what it sees appearing online, so I think it will be a good indicator on what works and what doesn't, and that for each known set of isos.

So there. Bleeding edge images for everyone, and for those who want some quality assurance: the more you contribute, the more likely you'll see official releases.

While contributing feedback is now certainly very easy, don't think that only providing feedback is sufficient, it takes time to maintain and improve aif and archiso as well and contributions in that department are still very welcome. I don't think we'll get to the original plan of official archiso releases for each stable kernel version, that seems like a lot of work despite all the above.

As for what is new: again too much to list, here is a changelog but I stopped updating it at some point. I guess the most visible interesting stuff is friendlier package dialogs (with package descriptions), support for nilfs, btrfs and syslinux (thanks Matthew Gyurgyik), and an issues reporting tool. Under the hood we refactored quite a bit, mostly blockdevice related stuff, config generation and the "execution plan" (like, how each function calls each other and how failures are tracked) in AIF has been simplified considerably.

Yesterday I finished the first working version of AIF's automatic procedure, along with a sample config for a basic install..

For me personally this means I can start working on the next step towards my goal of having all my systems "metadata" centrally stored (along with my real "data"), and the possibility to reconstruct all my systems in a deployment-meets-backup-restore fashion ( see http://dieter.plaetinck.be/rethinking_the_backup_paradigm_a_higher-level... )

The last few weeks I/we've been quite busy fixing up many things related to aif and the arch release stuff in general. The kernel guys are now at 2.6.29-rc8, so a stable release won't be long anymore and I want to get as much things fixed before 2.6.29 is out, so we can release soon after. (with /arch/setup and /arch/quickinst replaced by aif)

Misc notes:

• Working together with Aaron and Gerhard is great. Our interests and knowledge are diverse, so we each have our place in the releng team, but there is some overlap too. This works out great. Input from them (and the other developers) often brings fixes for stupid things that I've spent too much time on and was still looking over.
• I'm also a "real" Arch developer (this actually means package maintainer) now: I maintain 1 package :)
• The 2009.02 release was a success. I was suprised by the amount of attention it got (maybe because we were the first distro to ship a release that supports ext4). I realised that all those delays to fix some (imho) details was probably the best move after all.
• I've been looking at nixOS, which is a very interesting project. The way I see it, they combine the "functional description" part (which is on many systems handled by puppet/cfengine/kickstart/aif/...) right into their OS core. That, and some other features. It was a bit too unstable when I tried it though.

In light of the work and discussions around supporting Nilfs2 and Btrfs on Arch Linux and its installer AIF,
I've shared some AIF filesystem code design insights and experiences on the arch-releng mailing list.
This is some hard to understand code. Partly because it's in bash (and I've needed to work around some limitations in bash),
partly because there is some complex logic going on.

I think it's very useful material for those who are interested (it can also help understanding the user aspect),
so I wanted to share an improved version here.
On a related topic: I proposed to do a session at Fosdem 2011/"distro miniconf" about simple (console based) installers for Linux,
and how multiple distributions could share efforts maintaining installation tools, because there are a lot of cross-distribution concerns
which are not trivial to get right (mostly filesystems, but I also think about clock adjustments, bootloaders, etc).
Already several distro's use the (or a fork of) the Arch installer, for example Pentoo,
but I think cooperation could be much better and more efficient.

Anyway:


::Read more

My holidays present for Arch devs and users: AIF alpha-0.6 !

* Changes since alpha 0.5:

::Read more

• Arch Linux team interview @ OSNews
• Uzbl @ LWN.net

Apologies for only informing you about the second article now. I assumed most of you follow LWN (you probably should) or found the article anyway.
Of all the articles written about uzbl, no one came close to the quality of Koens work. So even though it's a bit dated it's still worth a read.

I recently did two talks, for which the videos are now online.

• Uzbl lightningtalk @ fosdem 2010
• Open Source Monitoring tools lightning talk @ Kangaroot showcase 2009. This one is password protected. If you were a participant, you should have received the pass

If all goes well, I'll be at ArchCon this summer, where I'll be doing these talks:

• AIF: The Arch Installation Framework
• Uzbl – web interface tools which adhere to the unix philosophy. Whereas in the fosdem talk I had to focus a lot of information into a short timeslot, here I'll elaborate a bit more

We're not sure yet if those talks will get videotaped.


::Read more

Froscon was great.

• It was smaller then I thought, I especially assumed more activity in the devrooms (I'm too used to fosdem which is much bigger), but hey nice conference anyway.
• It was great to meet (some of) my Arch Linux colleagues. Mostly the guys from Germany showed up, and Roman who is from Ukraine.
• We all got an Arch t-shirt/polo of which we could pick the color ourselves. Woohoo. Thanks donators.
• A few people also told me they had tried / were using uzbl (my browser project) so that was really cool.
• I had an interesting chat with Thomas Lange, the author of FAI (a tool to automatically mass-install systems, mostly Debian). I've used it in the past, liked it and was somewhat inspired by it when building AIF. (but some design goals are different).

Here is a picture of my "Arch releng partner" Gerhard (right) and I (left). We've done a lot of work together and it was great to talk face to face for once. Here we're showing (proudly) an Arch Linux Froscon disc (which is a slightly modified version of the 2009.08 core images):

Team photo:

Image gallery