Site Status

[Theo]

Well, I posted a message earlier about performance on the site and then I went to the server and dumped the database to a file for a backup and for some reason my post went away. Weird... Don't know that happened. I didn't make any other changes than that.

Anyways, I had mentioned that I had noticed the site was running slow. If you notice the forums or site running slow please PM me so I can take a look into it.

[Cold Revolver]

Well whatever you did worked because now I dont have to log in twice!

[Theo]

I have to apply some fixes to the server real quick. It will require me to reboot a few machines. We might have another 15 minutes of random outages during that time. Sorry for the problem.

[Theo]

The pain should be over now. Everything should be back to normal for everyone, whatever normal may be.

[Theo]

Damned NFS mount hung. Time to create a cluster...

[Theo]

I has to reboot some servers to update some service packs. It was a short notice thing. I have a couple more to do over the next couple days. I will hopefully schedule something first next time. I might also migrate the database to a new server...

[Theo]

Some of you might be thinking “What in the holy Jesus, mother Mary of God, happened today?” Yes, I’ll be the first to admin the site was offline. But this was only the half of it. Our incoming email servers were offline as well. And damn did that hurt (as we process about 150,000 messages per pay per server x 4). I had to go to the open house for my oldest daughter tonight and then we had an outage. Let’s see how quick we can drive from Benicia to Oakland in traffic at 4:30pm. Want a challenge for the next survivor, make that drive in the fastest time possible without being stopped by the CHP. So I rebooted the big box. This is the box that has the load balancer (Linux ipvsadm for those that care). Without this nothing works. Normally it goes down occasionally when I have to restart the big box but that’s infrequent enough to not worry about (about once a quarter for patches).

Today the big box decided to say “FU” to me. Actually I ran a command I shouldn’t have, and it did what it was supposed to. Just a word of note, if you are in a terminal session window on a Windows 2003 server, don’t click shutdown when you’re 30 miles away. It just makes for a long day.

We are working to get the load balancer transferred to a cluster. When this is done it should work. We had it configured as a cluster before but a bug in the OS, load balance or the cluster caused the cluster to hang causing outages. This happened about a year ago (it’s somewhere in this thread). So we fixed it by going to a single node (i.e. a point of failure).

In the background I decided to temporarily slow down the site. . The forums are powered by 3 servers (a primary that holds the content, database and servers web pages and two others that just server web pages). I decided to do this because the disk drives in the two smaller units have been reporting some errors and since I do go to the colo every day I figured I’d grab them. I should have them rebuilt and back in a few days (I still have to buy the drives).

It’s been a bad week. Actually it’s been a real bad August for me (don’t ask, I’ll tell Ken and he can just confirm, otherwise I won’t tell). I decided (partially because I like extreme levels of stress) to rebuild my workstation. It’s helped me get my mind off the other bad things. I figured it was a good time since I have some time off work and I always have my laptop. Hey, did I tell you my friend Murphy stopped by. Yeah, he came by and said “I bet you $5 that I can fry that laptop sitting over there against the wall.” He won that bet. So I’m just now getting my desktop to work the way that I want it. It takes me upwards of 3 days to get it operational (I do a lot of development so I need to install a lot of tools and oh, World of Warcraft disks had to be located as well).

So all in all, there was an outage and it lasted for a few minutes.

I would also like to explain something else. I have received many questions about this so I will take the time to ask. I did not join the forums in 1973. In fact, PHP, Linux or online forums were even invented yet. The forums clock is based on PST (Pacific Standard Time, or California time for those that don’t get PST or Pacific Standard Time). So I calculated the number of minutes since Midnight January 1, 1970 to my birth time using a simple dateadd function in SQL. Why? Because I’m funny like that! No, not funny haha, but special ed funny. I replaced the timestamp in the database with this number. I hope that quashes this problem of “How did you join before the webmaster?” If it doesn’t please re-read this paragraph again! You might catch on that I’m not as young as some of you. Then again if you are older than me and your reading this, we need to talk about the anime convention in Anaheim!

BTW, all this is meant in good fun. If you’re offended please re-read the last sentence!

[Theo]

For those of you who know Linux geek speak then this might make sense. The load index on the primary server was through the roof. It took me several minutes to get a SSH window up and running. It's that system usage I'm worried about.

07:10:01 AM CPU %user %nice %system %iowait %idle
07:20:01 AM all 57.75 0.00 37.87 4.38 0.00
07:30:01 AM all 65.88 0.00 31.07 3.05 0.00
07:40:01 AM all 56.85 0.00 35.15 8.00 0.00
...
10:50:02 AM all 69.37 0.00 28.29 2.34 0.00
11:00:01 AM all 70.16 0.00 27.26 2.58 0.00
11:10:01 AM all 78.51 0.00 19.35 2.14 0.00
11:20:01 AM all 73.86 0.00 22.94 3.20 0.00
Average: all 65.75 0.00 28.70 5.55 0.00

I took down two of the three older servers as they were reporting drive issues. These machines were 1.5ghz P4's with 1gb ram. The main server is a 3ghz 2gb ram (running NFS for the other read only nodes and housing the master mysql database). I'm not sure if we are seeing a database issue as well. I might turn on some logging BUT this would probably cause the machine to explode and our colo expressly forbids fire inside their facilities.

I had a quad Xeon box that I run virtual servers on. We have been trying to migrate stuff off of it (since it's limited in disk space) but I'm going to throw together a couple virtual Linux instances and see if that helps with the load. I was hoping to pickup a couple new drives for the other servers but it's a budget thing. Anyone have a couple good 1U servers lying around to donate to Ken (AA)? Cash also works as well.

Hopefully these virtual servers will help out more. They have a lot more horsepower than the 1.5ghz machines.

[Theo]

Additional node has been created and added to the mix. If something breaks call me...

[Theo]

Making more changes again. For those who have been here for a while please let me know how the performance of the site is today. We've found that the database is the slowest part of the site now. Everything else is working fine. If we can fix the database now then everything should be working fine.

[Theo]

It looks like the site seems to be working much faster now. We modified a SQL statement in PHPBB to try to improve performance. There seems to be near to zero time for most of the pages now. If there is a page that takes a significant time to load please PM me the link to this so I can check it and try to fix it.

On a lighter side I would like to say that the Christmas holidays are going to be approaching soon. Yes I said Christmas, not winter holidays. You see, I'm a Christian and until any other religion openly accepts the Bacon Cheeseburger I'm going to stay that way. But that's not my I'm mentioning it. What I really want is Christmas presents. So here is my list for those feeling cheery enough to buy me something.

I would really like a Tyan or SuperMicro dual CPU Opteron MB and a couple good AMD VT enabled processors for it (preferrably dual core or quad core if you want to wait until Jan, I will). I would also like a couple GB of ram (preferebly 6gb or 8gb if they are quad core). That might sound excessive but I have a plan. It might be an evil plan containing goldfish and a USB burner but it's still a plan. Do you have a plan? Thought not! Anyway, I don't need drives as I have many of them but if you are in the mood I'd like a 4 or 6 port SATA raid controller card. I currently have 3 1TB network appliances but they are a little slow so I'll rape one of them and take their drives or if you guys are really nice we can use 6 large SATA drives to match the raid controller. We don't need a lot of space but it helps for the virtual instances

"WTF, this is overkill" you might ask. What I'd like to do is run Xen and through together all of this site on a single box that will handle the load and then some.

As many of you know I receives no gifts last year from the members. I did receive a video from Ken (I think I won it actually) but he doesn't count. So let's work up a plan to fulfill the list this year. Can I get a yeah?

So if you are truely interested in supporting this site you can PM me (or post back here). Everyone get's credit for what they provide. Now you can either provide cash (which is always great) or the actual parts. I'll probably setup a gift registry at NewEgg to make sure but I'm waiting on some answers about VT enabled Opterons.

[Theo]

My bad... Changed the IP address of one of my domain controllers for a pending upgrade of some hardware and didn't realize there were a few legacy machines using it. I swear I did a netstat prior to the change... Scouts honor...

[Theo]

As you may know there was an outage. Not sure why but both nodes of our backend cluster hung in the last 12 hours. These particular boxes have had an uptime of 400 days without failure. We are looking into the root cause.

[Theo]

It's been a bad week. First my laptop hard drive crashes (yesterday) then I try to use my wife's laptop and then it's hard drive started going klunk klink klunk and then my desktop died. 0 for 3 in one day. Damn. Luckily I have a parts store very close. But it kept me from working my real job today.

I purchased the new memory for the servers and it should be here tomorrow and the Dell replacement memory came in finally so we should be able to fix the slower server. This will cause a momentary outage. But it happens.

That's the update...

[Theo]

Well, we're almost there. I've had no sleep but I'm almost done. Things are still kind of broken, will address them tonight.

[Yodama Man]

get some sleep guy