Server crashed

Rack
Rack

Yay, woke up this morning and wanted to quickly check a few things on this website. Fired up the computer and tried to access habschned.com when i noticed that the server cannot be reached. Immediately proceeded to check other sites i host in the same data center (different servers though) and found them not working neither.

That’s not exactly a very good start into a Sunday morning. Opened a support ticket with the data center and what the guys came back to me with was quite shocking. I expected it to be some sort of “small” issue, like the VM migrating to a different node or something, boy was i wrong. This is what the guys came back with:

 

Hi Toto,

We’re going to be upfront about this issue:

There has been a catastrophic failure on our systems. Data from many VMs have been lost. If your server is offline at the moment, it is most likely affected.

We are trying to restore data from backups made about 1-2 weeks ago.

As we are spending all of our human resources into service restoration, I’m unable to give further specific details into the issue at this time. Full details will be posted once we’ve resolved this issue.

We will be here all of Sunday fixing this issue. It is estimated to take a few hours to restore the VMs.

We appreciate your patience during this time.

Yeah, that’s not a reply you want to read. Especially not if you configured backups on a weekly basis. This site is backed up every sunday afternoon, and since the outage occured early sunday morning i was pretty much screwed. The most recent backup i had dated back to July 12 2014 and i kept my fingers crossed that not all data would have been lost. Later this evening (about an hour ago) i got a phone call letting me know that the box is up and running again, but all data was gone. So i had to restore the backup and lost about 4 or 5 posts in the process which i will have to write again. Anyway, good to be back online with this site and i hope the other sites will be online shortly as well.

Be the first to comment

Leave a Reply

Your email address will not be published.


I confirm

*


This site uses Akismet to reduce spam. Learn how your comment data is processed.