Some Minor Outages Coming Up

Last week's disk crash managed to cause low-level corruption in the RAID array (which is not supposed to happen) as well as bringing down the server (which is not supposed to happen). The only way to fix it is to wipe the array clean and rebuild it, which kind of defeats the purpose of having a RAID array in the first place.

So we're going to be taking a short outage soon to add a new array. Then I'm going to migrate everyone onto it, and then we're going to wipe and rebuild the old array. This is a lot of work for me, but it means that we'll be down for 10 minutes here and there instead of six or eight hours all at once.

I've also done a complete backup to another server, because I've seen too many things that aren't supposed to happen, happen.

This never happened with my 3Ware controllers. :(

Update: New RAID array addded without anything actually catching fire! Some of the virtual machines didn't come up quite right, and it took several seconds to reboot them, but all good apart from that.

posted by Pixy Misa at 01:11 AM

| Access Comments