I survived LCL 31-3-2008

On 31-3-2008 LCL, one of the most used datacenters in Belgium - and the only one with a 0% downtime record in Belgium - had major power issues with their datacenter in Diegem, bringing lots of Belgian parties offline. (more specifics on the net).

If you're one of the sysadmins of a website with 35M members and 150M hits per day this means you're in for an exciting night ...

Long story short: LCL couldn't bring the power back up fast enough so we rented our own generator. After rushing to Brussels we ( The Netlog ITS-team) could sit in the offices upstairs. But after a while they had to close it down but we could stay in a room nearby (without furniture). After that we went to Steven's appartment in Machelen. We worked about 30 hours in a row, getting our sites back up and running (mostly fixing corrupted databases). Our friends from the devteam also did some great achievements the next day, by using the redundancy of denormalized database layouts they could restore nearly all missing content that had vanished because of table repairs.

Here is some footage: