On Dec 3, 2013 a server crash wiped out the database...
- webmaster
- Site Admin
- Posts: 1102
- Joined: Jul 24, 2004 10:25pm
- Gender: Male
- Location: Redding, CA
- Contact:
On Dec 3, 2013 a server crash wiped out the database...
Some astute individuals out there may have noticed that the website hasn't been accessible since about late night Sunday... well, funny thing, turns out there was a monumental system failure which corrupted just about everything possible, so we pretty much lost all email and all database tables. Backups? Yes, well... normally we're very good at keeping backups, but the backups that were on the disc also got corrupted and the external backup was scheduled to be hooked up the next morning.
It's likely there is a semi-recent backup on some hard drive somewhere, but it's going to take time to find. So I figured there probably wouldn't be any better time to start fresh with the forums, and when we can locate a backup of all the forum threads then we can post it as an archive.
I'll let Gary explain in more detail when he has the time.
It's likely there is a semi-recent backup on some hard drive somewhere, but it's going to take time to find. So I figured there probably wouldn't be any better time to start fresh with the forums, and when we can locate a backup of all the forum threads then we can post it as an archive.
I'll let Gary explain in more detail when he has the time.
-
- Visitor
Re: What a week!
Starting fresh! Whether you can get the forums back to normal matters not to me. It's kinda nice not having all the dead threads anymore.
Re: What a week!
^That was me. Forgot I had to activate my account first xD
"Life is pain, highness. Anyone who says differently is selling something."
"If you want to view paradise simply look around and view it. Anything you want to, do it. You want to change the world? There's nothing to it."
"If you want to view paradise simply look around and view it. Anything you want to, do it. You want to change the world? There's nothing to it."
-
- Host Admin
- Posts: 276
- Joined: Aug 27, 1973 12:03am
- Gender: Male
- Location: CA, USA, Earth, Milkyway
Re: What a week!
First off, let me apologize for everything that has happened. As Ken can probably contest, “protect the data” is a motto that I’ve been living by for some years. My second job had that written on a white board with permanent marker.
So here is the situation as it sits. About 3 weeks ago I took one of our two iSCSI server’s offline. It was the backup. The purpose of this was to upgrade the drive capacity of this unit and to them make it the primary, and then doing the same to the old one. When I say old, I don’t mean 10 year old hardware. The upgrade was actually schedule to go in place that next morning.
At exactly 2:00 PST Sunday morning, the virtual server I had running as the iSCSI target initiated a check and immediately caused the box to freeze. At around 5:00 am PST I received a call about email being down from a client. Checking into it I had one of the techs from the colo reboot the box. It came back just fine. Mind you, this wasn’t a hardware failure (as this is a very redundant machine) but rather a software hiccup.
I spent the next several hours attempting to fix the issue. The iSCSI is built on Sun Solaris 11 running zpool and zfs. The underlying disk drives are virtual (4 of them, each 512mb – VMware limitation) creating a 2TB iSCSI storage system. It sound’s small, but the underlying drives are high speed very redundant drivers to prevent failures. The problem is the 4 “virtual” drives that make up the 2TB share were marked as bad. Again, the underlying disks were 100%.
I worked with a Solaris tech who guided me through attempting to fix this problem. In the end the commands we worked through managed to get the drive working again, just with no data. So we stopped at that point as to ensure that we don’t cause any additional damage to the unit and we have contacted a data recovery specialist.
We are in the process of arranging terms of contract and shipping of the hardware to the consultant. At the end of this, regardless of data recovery, we’re looking at about $6,000 in total fees with no guarantee of data recovery. It will also take 10 days or so to find out whether the data is viable when it’s finished.
Besides the web site, I myself had amassed 10+ years’ worth of data, emails, etc. My data was on a second virtual disk array (so we’re actually talking about 2 x 2TB iSCSI shares worth of data).
This is a series of unfortunate events in that we’re been very diligent about leasing redundant hardware (high end sun boxes, commercial raid solutions, etc.), software, and using as much underlying protections as we can, and yet we still failed you, our community.
The hosting site of this business has been generating us little revenue over the years. We have kept this in place mostly because it breaks even, and we have a lot of friends that utilize out services (as well as a few valued customers). In 2013 alone we’re upgrade the bulk of our hardware to plan for the future of the business and to better support the existing clients that we have. Unfortunately that’s put us in a tough situation. We will continue to attempt record data until it is deemed that we cannot recover it. There is no ETC for that at this time. In 15 years we’re never lost any significant data and have been able to recover from my outages within less than a day (worst case scenario) with 100% of that data intact. This is Murphy ’s Law; the small period of time when you’re not protected is when you will fail. That was a 3 week span out of 15 years.
I have been working painstakingly to get everything up to at least a usable state (I’m literally 100 hours in to a 6 day week right now). For the AA site, there will need to be some performance tweaking, and I’m going to have one of my guys on it once we get the remainder of our clients back online.
Ken has done an outstanding job with AA over the years and I know he’s put his heart and soul into it, and as a personal friend since college (a lifetime ago) I hits me hard to know that I’ve impacted him (and his users) in this way. I was watching the site it received it 1 millionth his after 2 years, I was there when it hit 1 million hits per day (2005 ish ??).
With all of this you have my heart filled apology.
From the Hold Stead perspective, we will be winding down the commercial hosting, my friends sister company will continue to run this site (as a few others) as they are the ones that leased us the hardware. So this will go on…
As we will rebuild, bigger and better, we must…
--Theo
AKA Gary Smith
So here is the situation as it sits. About 3 weeks ago I took one of our two iSCSI server’s offline. It was the backup. The purpose of this was to upgrade the drive capacity of this unit and to them make it the primary, and then doing the same to the old one. When I say old, I don’t mean 10 year old hardware. The upgrade was actually schedule to go in place that next morning.
At exactly 2:00 PST Sunday morning, the virtual server I had running as the iSCSI target initiated a check and immediately caused the box to freeze. At around 5:00 am PST I received a call about email being down from a client. Checking into it I had one of the techs from the colo reboot the box. It came back just fine. Mind you, this wasn’t a hardware failure (as this is a very redundant machine) but rather a software hiccup.
I spent the next several hours attempting to fix the issue. The iSCSI is built on Sun Solaris 11 running zpool and zfs. The underlying disk drives are virtual (4 of them, each 512mb – VMware limitation) creating a 2TB iSCSI storage system. It sound’s small, but the underlying drives are high speed very redundant drivers to prevent failures. The problem is the 4 “virtual” drives that make up the 2TB share were marked as bad. Again, the underlying disks were 100%.
I worked with a Solaris tech who guided me through attempting to fix this problem. In the end the commands we worked through managed to get the drive working again, just with no data. So we stopped at that point as to ensure that we don’t cause any additional damage to the unit and we have contacted a data recovery specialist.
We are in the process of arranging terms of contract and shipping of the hardware to the consultant. At the end of this, regardless of data recovery, we’re looking at about $6,000 in total fees with no guarantee of data recovery. It will also take 10 days or so to find out whether the data is viable when it’s finished.
Besides the web site, I myself had amassed 10+ years’ worth of data, emails, etc. My data was on a second virtual disk array (so we’re actually talking about 2 x 2TB iSCSI shares worth of data).
This is a series of unfortunate events in that we’re been very diligent about leasing redundant hardware (high end sun boxes, commercial raid solutions, etc.), software, and using as much underlying protections as we can, and yet we still failed you, our community.
The hosting site of this business has been generating us little revenue over the years. We have kept this in place mostly because it breaks even, and we have a lot of friends that utilize out services (as well as a few valued customers). In 2013 alone we’re upgrade the bulk of our hardware to plan for the future of the business and to better support the existing clients that we have. Unfortunately that’s put us in a tough situation. We will continue to attempt record data until it is deemed that we cannot recover it. There is no ETC for that at this time. In 15 years we’re never lost any significant data and have been able to recover from my outages within less than a day (worst case scenario) with 100% of that data intact. This is Murphy ’s Law; the small period of time when you’re not protected is when you will fail. That was a 3 week span out of 15 years.
I have been working painstakingly to get everything up to at least a usable state (I’m literally 100 hours in to a 6 day week right now). For the AA site, there will need to be some performance tweaking, and I’m going to have one of my guys on it once we get the remainder of our clients back online.
Ken has done an outstanding job with AA over the years and I know he’s put his heart and soul into it, and as a personal friend since college (a lifetime ago) I hits me hard to know that I’ve impacted him (and his users) in this way. I was watching the site it received it 1 millionth his after 2 years, I was there when it hit 1 million hits per day (2005 ish ??).
With all of this you have my heart filled apology.
From the Hold Stead perspective, we will be winding down the commercial hosting, my friends sister company will continue to run this site (as a few others) as they are the ones that leased us the hardware. So this will go on…
As we will rebuild, bigger and better, we must…
--Theo
AKA Gary Smith
- webmaster
- Site Admin
- Posts: 1102
- Joined: Jul 24, 2004 10:25pm
- Gender: Male
- Location: Redding, CA
- Contact:
Re:
Yeah, I'd really like that as well! I just wish it were possible. All the comments we're in the database, so for now they are gone and we'll just have to wait and see how the data recovery goes. Once we get phpMyAdmin installed I'll be able to setup the database so we can get the comment feature going again.I've commented. wrote:I'd like it if you'd restore Absolute Anime to normal. The 'anime profiles section' and the 'character profiles' section, the comments are missing, and so is the comment box. I hope you fix Absolute Anime back to normal.
-
- Master Otaku
- Posts: 64
- Joined: Jul 07, 2011 3:50am
-
- Master Otaku
- Posts: 64
- Joined: Jul 07, 2011 3:50am
-
- Master Otaku
- Posts: 64
- Joined: Jul 07, 2011 3:50am
-
- Master Otaku
- Posts: 64
- Joined: Jul 07, 2011 3:50am
FIND A BACKUP AND RESTORE ABSOLUTE ANIME THE FORUMS+COMMENTS
I hope there is a working backup to fully restore Absolute Anime and that Absolute Anime can be fully restored. I'd like it if you'd restore Absolute Anime to normal. The 'anime profiles' section' and the 'character profiles' section, the comments are missing, and so is the comment box. You can find a backup and backup the forums and then delete this version of the forums and go back to the back the forums from the backup and have them be the full forums again; and wait until you find a backup for the comments in the 'anime profiles' section and the 'character profiles' section and don't re-enable comments to those sections until you find a backup and use the backup to restore the comments. I hope you (can) fix Absolute Anime back to normal.
- Biki
- Absolute Otaku
- Posts: 2585
- Joined: Jun 09, 2005 10:09am
- Gender: Female
- Location: Drifting Through Negative Space
- Contact:
Re: What a week! A server crash wiped out the database!
I am very sorry for all that you have gone through, Theo. And you haven't failed us, though the loss of my private messages will haunt me for awhile. (That's just because I tend to keep crap like that. Ask my various old cell phones, even the completely dead ones.) And this way we can start fresh. AA prime had a good run. And if anything of it can be restored, I'm all for that, too. But don't get too down about this site. We'll remain as strong as we ever were. [: I hope you can recover all of your clients' data and such, and your emails. I know how great a loss those can be. We'll be alright around here. [: Ken, if you find a recent-ish backup and archive it, give me a heads up, won't you? Thanks. [:
ikiBBiki
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
- Biki
- Absolute Otaku
- Posts: 2585
- Joined: Jun 09, 2005 10:09am
- Gender: Female
- Location: Drifting Through Negative Space
- Contact:
Re: What a week! A server crash wiped out the database!
Theo, I'm very sorry for all that you've gone through. And you haven't failed us, though the loss of my private messages will haunt me for awhile. (That's just because I tend to keep crap like that. Ask my various old cell phones, even the completely dead ones.) But we'll be alright here. We'll remain as strong as we ever were. And this is an opportunity for us to start anew. AA prime had a good run. Though if anything of it can be restored, I'm all for that, too. I just hope you can recover your clients' data and such. And your emails as I know how the loss of them can be quite a blow. We'll be okay here. [: And Ken, if we recover any of the old forums and such and you archive them, give me a heads up, won't you? [: Thanks.
ikiBBiki
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
- Trigger Happy
- Master Otaku
- Posts: 97
- Joined: Dec 09, 2013 2:12am
Re: What a week! A server crash wiped out the database!
I hope everything turns out okay. It is a nice fresh start though and getting rid of those dead threads saves us from having to go through them all. Like I said before though, I hope everything is ok and I hope you guys find everything you need. We still think you guys are the best!!
The one and only Envy Isn't Envious
Part of the original AA gang - I sure do miss the old days
Part of the original AA gang - I sure do miss the old days