Site Status

[Dretlin]

Awesome. And didn't even need to make that phone call.

Again.

[Theo]

As you might have seem, website was offline. For some odd reason, we were seeing some problems with the web server having apache hang recently. Nothing has changed so I don't know why it's happening. I'm writing it off as hardware. If you have something that hasn't changed software wise in 18 months, it has to be something else.

We had a dual xeon SuperMicro with 8/gb laying around so I set that up. When you are able to read this, your DNS has replicated and maintenance for you is over. If you can't read this then, well, you don't know that I just said that.

This particular SuperMicro server has an interesting flaw. It was purchased to become a Windows 2003 Small Business Server but the network kept hanging. After a year not being able to use it, it was shelved. So, last year I did some research to find that there was a bug in the design. Actually, in the Intel chipset. The newer drivers will not work on it if you have more than 4GB of ram. We have 8GB. So one of the authors of the E1000 linux driver patched it to handle the case of more than 4GB of ram. So, now it's usable, but only under linux and you have to compile the driver yourself. Luckily it can be compiled without compiling the entire kernel.

So, this machine has life again. I'll pull the old one apart and play around with it. It might come back as a database/file server once I figure it out.

[Theo]

Small changes. After the server move, we maxed out connections to the server. It appears that we have leachers. Normally, I wouldn't care. Ken might though. The problem is these leachers are downloading max connections at max speed. One in particular had 200 threads going. Normal browsers have 2 threads. We have the web server set for 200 threads. So, this caused a significant slowdown for many users. AS A RESULT, we are putting a connection limiter back in place. Normally I dislike these are they have negetive affects on site performance, but the tradeoff is necessary. I have set the site for 800KB/s and 10 connections per IP. Note, this if for content. Images are housed differnetly. It should also be noted that we can tweak these limits down per IP/range. So, please repect the content, the site, the people, and the animals as well, since they can't really force your respect.

Now, with that said, if you have a particular need to want to mirror parts of the content, it would make more sense to contact Ken. I would be much easier to zip up content and have 80% of the bandwidth and servers resouces.

Time for bed now.

[Theo]

Up, Down, Up, Down, Up... This is partly caused be me. The I've had to stop/start the site several times to reload the config. I'm slowly tweaking the config to see what the best combo is for the site. The last one seemed to be working well performance wise but for some reason Apache was taking 8GB of ram + some swap. It's probably because I spawned max instances. Anyway, tweked it work to about 75% of what it was. I will check it from there. I think the magic number is probably 50% of max. Unfortunately, to make changes I have to stop and start the web service which usually takes about 10 seconds.

It looks like the site is doing very well today besides my interruptions.

[SASSZAIN]

Sounds like you've just been having fun

[BigKwell]

Heard that the site can now link to Facebook. I've got a Facebook account, but I didn't see a Facebook icon on the Absolute Anime site. Any way I can find it?

[webmaster]

For right now, it only shows up when you're logged out. It'll be up near the quick login.

[Theo]

Argh. I migrated one of the old web domains that nobody used anymore and my image icon is broken... I'll have to find it again.

[ANGEL OF HELL]

AA has taken over my life. Every. Aspect.

[Theo]

Yeah, I found my picture on one of the old hard drives I removed from the old server.

[Dretlin]

With the website name, I am really surprised we suffer from a lack of consistant users.

[SASSZAIN]

same here I think we get a lot of frequent visitors, though none of them stay..

[Theo]

My bad... I implemented a firewall change while trying to fix a problem and it blocked a bunch of people. It's been fixed. Anyway, we have been seeing a couple incidents where the Apache web service hangs. When it happens, everything looks fine. The web service just isn't serving pages anymore. We hate to have to restart it on a regular basis, but we might have to in order to solve the problem. Luckily a graceful restart solves the problem, so hopefully no users will notice a regular restart.

[Theo]

Server maintenance is being done today. I have to update some core OS files on the server. This is a must. Sicne the server is 50 miles away, if it goes bad, It will be offline for 12 hours. I don't expect this to be the case though. Server will be rebooted at 09:45 PST (about 39 minutes from the time of this posting).

[Theo]

Small problems were encountered. I had to reboot the server a couple more times than expected. There was a problem with the firewall that was fixed but the changes were never saved (plus I had to go for a walk for 15 minutes). Anyway, all should be well now. If you do encounter a problem, please email me directly (gary.smith@holdstead.com) with "absoluteanime.com" somewhere in the subject.

[Theo]

There was a problem with the database today. For some reason the mysql database decided that it would continously crash because of a bad innodb page. If this sounds greek, then you're part of the 99.99% of the world who doesn't need to know this. To fix the problem I had to stop the web services, stop mysql, add a special line to the mysql config file to allow innodb recovery, do a full export of the entire database (which wasn't that bad), renamed the mysql data folder, created a new mysql data folder, ran the mysql_install_db script, updated the permissions on the folder (and it's contents), removed the line that I added to the mysql config, started mysql, performed a full import of the data (which was slow as it's a large database), and then finally started the web services.

Why is this important. It's not, but if someone runs into the same problem, google would have indexed this page and they might just find the answer they were looking for.

So, the keywords of the day are: mysql innodb corrupt table recovery shot who in the foot and didn't know it tacobell and things you can buy theo for xmas