On Dec 3, 2013 a server crash wiped out the database...

After the 2013 database crash, most new topics were created in a single forum. This is that forum.
User avatar
webmaster
Site Admin
Site Admin
Posts: 1102
Joined: Jul 24, 2004 10:25pm
Gender: Male
Location: Redding, CA
Contact:

On Dec 3, 2013 a server crash wiped out the database...

Post by webmaster »

Some astute individuals out there may have noticed that the website hasn't been accessible since about late night Sunday... well, funny thing, turns out there was a monumental system failure which corrupted just about everything possible, so we pretty much lost all email and all database tables. Backups? Yes, well... normally we're very good at keeping backups, but the backups that were on the disc also got corrupted and the external backup was scheduled to be hooked up the next morning.

It's likely there is a semi-recent backup on some hard drive somewhere, but it's going to take time to find. So I figured there probably wouldn't be any better time to start fresh with the forums, and when we can locate a backup of all the forum threads then we can post it as an archive.

I'll let Gary explain in more detail when he has the time.
I've commented.
Visitor

Post by I've commented. »

I'd like it if you'd restore Absolute Anime to normal. The 'anime profiles section' and the 'character profiles' section, the comments are missing, and so is the comment box. I hope you fix Absolute Anime back to normal.
User avatar
Tyche
Absolute Otaku
Posts: 1653
Joined: Dec 10, 2006 11:16am
Location: overhill and underhill

Re: What a week!

Post by Tyche »

Starting fresh! Whether you can get the forums back to normal matters not to me. It's kinda nice not having all the dead threads anymore.
User avatar
Tyche
Absolute Otaku
Posts: 1653
Joined: Dec 10, 2006 11:16am
Location: overhill and underhill

Re: What a week!

Post by Tyche »

^That was me. Forgot I had to activate my account first xD
"Life is pain, highness. Anyone who says differently is selling something."

"If you want to view paradise simply look around and view it. Anything you want to, do it. You want to change the world? There's nothing to it."
Theo
Host Admin
Host Admin
Posts: 276
Joined: Aug 27, 1973 12:03am
Gender: Male
Location: CA, USA, Earth, Milkyway

Re: What a week!

Post by Theo »

First off, let me apologize for everything that has happened. As Ken can probably contest, “protect the data” is a motto that I’ve been living by for some years. My second job had that written on a white board with permanent marker.

So here is the situation as it sits. About 3 weeks ago I took one of our two iSCSI server’s offline. It was the backup. The purpose of this was to upgrade the drive capacity of this unit and to them make it the primary, and then doing the same to the old one. When I say old, I don’t mean 10 year old hardware. The upgrade was actually schedule to go in place that next morning.

At exactly 2:00 PST Sunday morning, the virtual server I had running as the iSCSI target initiated a check and immediately caused the box to freeze. At around 5:00 am PST I received a call about email being down from a client. Checking into it I had one of the techs from the colo reboot the box. It came back just fine. Mind you, this wasn’t a hardware failure (as this is a very redundant machine) but rather a software hiccup.

I spent the next several hours attempting to fix the issue. The iSCSI is built on Sun Solaris 11 running zpool and zfs. The underlying disk drives are virtual (4 of them, each 512mb – VMware limitation) creating a 2TB iSCSI storage system. It sound’s small, but the underlying drives are high speed very redundant drivers to prevent failures. The problem is the 4 “virtual” drives that make up the 2TB share were marked as bad. Again, the underlying disks were 100%.

I worked with a Solaris tech who guided me through attempting to fix this problem. In the end the commands we worked through managed to get the drive working again, just with no data. So we stopped at that point as to ensure that we don’t cause any additional damage to the unit and we have contacted a data recovery specialist.

We are in the process of arranging terms of contract and shipping of the hardware to the consultant. At the end of this, regardless of data recovery, we’re looking at about $6,000 in total fees with no guarantee of data recovery. It will also take 10 days or so to find out whether the data is viable when it’s finished.

Besides the web site, I myself had amassed 10+ years’ worth of data, emails, etc. My data was on a second virtual disk array (so we’re actually talking about 2 x 2TB iSCSI shares worth of data).

This is a series of unfortunate events in that we’re been very diligent about leasing redundant hardware (high end sun boxes, commercial raid solutions, etc.), software, and using as much underlying protections as we can, and yet we still failed you, our community.

The hosting site of this business has been generating us little revenue over the years. We have kept this in place mostly because it breaks even, and we have a lot of friends that utilize out services (as well as a few valued customers). In 2013 alone we’re upgrade the bulk of our hardware to plan for the future of the business and to better support the existing clients that we have. Unfortunately that’s put us in a tough situation. We will continue to attempt record data until it is deemed that we cannot recover it. There is no ETC for that at this time. In 15 years we’re never lost any significant data and have been able to recover from my outages within less than a day (worst case scenario) with 100% of that data intact. This is Murphy ’s Law; the small period of time when you’re not protected is when you will fail. That was a 3 week span out of 15 years.

I have been working painstakingly to get everything up to at least a usable state (I’m literally 100 hours in to a 6 day week right now). For the AA site, there will need to be some performance tweaking, and I’m going to have one of my guys on it once we get the remainder of our clients back online.

Ken has done an outstanding job with AA over the years and I know he’s put his heart and soul into it, and as a personal friend since college (a lifetime ago) I hits me hard to know that I’ve impacted him (and his users) in this way. I was watching the site it received it 1 millionth his after 2 years, I was there when it hit 1 million hits per day (2005 ish ??).

With all of this you have my heart filled apology.

From the Hold Stead perspective, we will be winding down the commercial hosting, my friends sister company will continue to run this site (as a few others) as they are the ones that leased us the hardware. So this will go on…

As we will rebuild, bigger and better, we must…

--Theo
AKA Gary Smith
User avatar
webmaster
Site Admin
Site Admin
Posts: 1102
Joined: Jul 24, 2004 10:25pm
Gender: Male
Location: Redding, CA
Contact:

Re:

Post by webmaster »

I've commented. wrote:I'd like it if you'd restore Absolute Anime to normal. The 'anime profiles section' and the 'character profiles' section, the comments are missing, and so is the comment box. I hope you fix Absolute Anime back to normal.
Yeah, I'd really like that as well! I just wish it were possible. All the comments we're in the database, so for now they are gone and we'll just have to wait and see how the data recovery goes. Once we get phpMyAdmin installed I'll be able to setup the database so we can get the comment feature going again.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

Post by I've commented. »

I had to re-register.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

Post by I've commented. »

That new "posting review" thing is making posting difficult.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

Post by I've commented. »

I hope there is a working backup to fully restore Absolute Anime and that Absolute Anime can be fully restored.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

Post by I've commented. »

Fix the clock on Absolute Anime. The time displayed is incorrect.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

FIND A BACKUP AND RESTORE ABSOLUTE ANIME THE FORUMS+COMMENTS

Post by I've commented. »

I hope there is a working backup to fully restore Absolute Anime and that Absolute Anime can be fully restored. I'd like it if you'd restore Absolute Anime to normal. The 'anime profiles' section' and the 'character profiles' section, the comments are missing, and so is the comment box. You can find a backup and backup the forums and then delete this version of the forums and go back to the back the forums from the backup and have them be the full forums again; and wait until you find a backup for the comments in the 'anime profiles' section and the 'character profiles' section and don't re-enable comments to those sections until you find a backup and use the backup to restore the comments. I hope you (can) fix Absolute Anime back to normal.
User avatar
Biki
Absolute Otaku
Posts: 2585
Joined: Jun 09, 2005 10:09am
Gender: Female
Location: Drifting Through Negative Space
Contact:

Re: What a week! A server crash wiped out the database!

Post by Biki »

I am very sorry for all that you have gone through, Theo. And you haven't failed us, though the loss of my private messages will haunt me for awhile. (That's just because I tend to keep crap like that. Ask my various old cell phones, even the completely dead ones.) And this way we can start fresh. AA prime had a good run. And if anything of it can be restored, I'm all for that, too. But don't get too down about this site. We'll remain as strong as we ever were. [: I hope you can recover all of your clients' data and such, and your emails. I know how great a loss those can be. We'll be alright around here. [: Ken, if you find a recent-ish backup and archive it, give me a heads up, won't you? Thanks. [:
ikiBBiki :o
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
User avatar
Biki
Absolute Otaku
Posts: 2585
Joined: Jun 09, 2005 10:09am
Gender: Female
Location: Drifting Through Negative Space
Contact:

Re: What a week! A server crash wiped out the database!

Post by Biki »

Theo, I'm very sorry for all that you've gone through. And you haven't failed us, though the loss of my private messages will haunt me for awhile. (That's just because I tend to keep crap like that. Ask my various old cell phones, even the completely dead ones.) But we'll be alright here. We'll remain as strong as we ever were. And this is an opportunity for us to start anew. AA prime had a good run. Though if anything of it can be restored, I'm all for that, too. I just hope you can recover your clients' data and such. And your emails as I know how the loss of them can be quite a blow. We'll be okay here. [: And Ken, if we recover any of the old forums and such and you archive them, give me a heads up, won't you? [: Thanks.
ikiBBiki :o
*~Color Me Impressed~*
In honor of Cold Revolver: I am not gay, I'm just colorful Betch! =-]
User avatar
Trigger Happy
Master Otaku
Posts: 97
Joined: Dec 09, 2013 2:12am

Re: What a week! A server crash wiped out the database!

Post by Trigger Happy »

I hope everything turns out okay. It is a nice fresh start though and getting rid of those dead threads saves us from having to go through them all. Like I said before though, I hope everything is ok and I hope you guys find everything you need. We still think you guys are the best!!
The one and only Envy Isn't Envious

Part of the original AA gang - I sure do miss the old days
Theo
Host Admin
Host Admin
Posts: 276
Joined: Aug 27, 1973 12:03am
Gender: Male
Location: CA, USA, Earth, Milkyway

Re:

Post by Theo »

I've commented. wrote:Fix the clock on Absolute Anime. The time displayed is incorrect.
Fixed. I setup the NTP server today.
I've commented.
Master Otaku
Posts: 64
Joined: Jul 07, 2011 3:50am

Post by I've commented. »

Thanks for fixing the time.
Post Reply