I survived LCL 31-3-2008

On 31-3-2008 LCL, one of the most used datacenters in Belgium - and the only one with a 0% downtime record in Belgium - had major power issues with their datacenter in Diegem, bringing lots of Belgian parties offline. (more specifics on the net).

If you're one of the sysadmins of a website with 35M members and 150M hits per day this means you're in for an exciting night ...

Long story short: LCL couldn't bring the power back up fast enough so we rented our own generator. After rushing to Brussels we ( The Netlog ITS-team) could sit in the offices upstairs. But after a while they had to close it down but we could stay in a room nearby (without furniture). After that we went to Steven's appartment in Machelen. We worked about 30 hours in a row, getting our sites back up and running (mostly fixing corrupted databases). Our friends from the devteam also did some great achievements the next day, by using the redundancy of denormalized database layouts they could restore nearly all missing content that had vanished because of table repairs.

Here is some footage:

Netlog homepage


hi-res version

At LCL



Generator


Toon & Nicolas


Kenny


Steven - Jakob - Lorenz


Jakob - Steven - me - Kenny

At Steven's appartment

No pics made here, just a silly video ;)

Trackback URL for this post:

http://dieter.plaetinck.be/trackback/41
Submitted by Dieter_be on Sat, 04/19/2008 - 12:10. categories [ ]

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>. Beside the tag style "<foo>" it is also possible to use "[foo]". PHP source code can also be enclosed in <?php ... ?> or <% ... %>.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Security question, designed to stop automated spam bots