• Home
  • Search
    •  
  • Login
    • Username: Password:

      Did you miss your activation email?

Author Topic: Getting Ready to move servers - Sunday July 24th  (Read 9858 times)

Nolefan

  • Lord of Avalon
  • Benevolent Despot
  • *****
  • Posts: 2334
  • 八九不离十
    • BeijingDaze
Getting Ready to move servers - Sunday July 24th
« on: July 24, 2011, 01:14:35 PM »
We’re sure you noticed we suffered a bit of a disaster with the server over the past few months, and more specifically 4 weeks ago!

That time, monitors alerted us of a failed drive in this server, we contacted the Host to have this swapped and the replacement was scheduled but a second drive also started producing errors before the drive was replaced.

This server runs a RAID10 array which means two drives could fail and it should stay online however in this instance the bad drive quickly corrupted the RAID array.

Both drives were replaced by Softlayer and we attempted a repair using fsck but this failed, leaving us with no choice but to restore the server from a backup which took 10 hours to complete then another filesystem check took a few more hours.

Moving forward we’ll be making a number of changes to ensure this doesn’t happen again and to improve restoration times and communication on any future issues.

1. This is the 6th drive failure we’ve had on this server despite having replaced each drive more than once and the RAID controller card. We no longer trust this hardware and will be replacing it with a new server as soon as possible - an exact date and time will be emailed to you when we receive and setup the hardware.

2. Another server will be bought online in the next hour in Dallas so files are restored faster. The current backup servers are hosted in Los Angeles so restores to Dallas take slightly longer than they should.

3. We're getting ready to move everything over into the new machine hoping it's more stable than the last piece of crap we called a server that made me lose even more hair than I'm already losing.

4. The new server is now online and has been setup. The transfer is being started in the next hour and there are a few points to note. There will be no downtime during the transfer

so pretty much it's business as usual and keep posting. This is pretty much just a heads up agagagagag agagagagag

alors régressons fatalement, eternellement. Des débutants, avec la peur comme exutoire à l'ignorance et Alzheimer en prof d'histoire de nos enfances!
- Random food, music and geek tales from the 'Jing: http://beijingdaze.com

Granny Mae

  • Barfly
  • *
  • Posts: 3479
Re: Getting Ready to move servers - Sunday July 24th
« Reply #1 on: July 24, 2011, 08:03:35 PM »
Thanks Nolefan and to the other folk involved. I can't really understand it all, but I appreciate that it takes a lot of work. agagagagag

mlaeux

  • Barfly
  • *
  • Posts: 1799
  • How's the water?
    • Fukushima has changed everything.
Re: Getting Ready to move servers - Sunday July 24th
« Reply #2 on: July 25, 2011, 07:11:47 AM »
Ditto what Granny said.

ericthered

  • Global Moderator
  • *****
  • Posts: 6498
Re: Getting Ready to move servers - Sunday July 24th
« Reply #3 on: July 25, 2011, 07:52:44 AM »
Errr...ok..soo...you sprayed raid into the server to get rid of some gremlins, right??  agagagagag agagagagag agagagagag All hail the Techno-Wizard who makes the Saloon work at uptimum spiffiness
"Anyone who lives within their means suffers from a lack of imagination." Oscar Wilde.

"It's all oojah cum spiffy". Bertie Wooster.
"The stars are God's daisy chain" Madeleine Bassett.

piglet

  • Barfly
  • *
  • Posts: 1667
    • Piglet's House
Re: Getting Ready to move servers - Sunday July 24th
« Reply #4 on: July 25, 2011, 04:26:42 PM »
Hear Hear.well done gentlemen. Top Hole!  agagagagag
For people who like peace and quiet - a phoneless cord

AMonk

  • Global Moderator
  • *****
  • Posts: 7334
Re: Getting Ready to move servers - Sunday July 24th
« Reply #5 on: July 26, 2011, 06:52:18 AM »

 agagagagag Noles akakakakak
Moderation....in most things...

solongtinik

  • Barfly
  • *
  • Posts: 458
  • tryin to complete my life's journey
Re: Getting Ready to move servers - Sunday July 24th
« Reply #6 on: July 26, 2011, 11:48:52 PM »
echoing granny! agagagagag
thanks noles! akakakakak
if he does not struggle against what is wrong-even if it seems beyond his strength-he will bever find the right road

Ruth

  • Barfly
  • *
  • Posts: 3349
Re: Getting Ready to move servers - Sunday July 24th
« Reply #7 on: July 27, 2011, 05:20:43 AM »
Wo bu ming bai, other than that it takes a whole lot of dedication and effort to keep this place running.

Thank you Noles for all the time and effort you put in behind the scenes to keep the Saloon gremlin free and accessible to us.

(Sorry about the hair loss. Have you tried the civet juice? I'm told it works wonders.)
If you want to walk on water, you have to get out of the boat.