Server Busy - Times-outs and simply overwhelmed ...

Discussion in 'Website news & discussions' started by xcel, Jun 4, 2008.

  1. xcel

    xcel PZEV, there's nothing like it :) Staff Member

    Hi All:

    ___I tried to send this via the E-Mail feature within VBulletin but apparently that failed due to the guest viewership overnight. Let me apologize up front but it is best I update you with what was going on yesterday and still today.

    ___Because of a few CNN stories about CleanMPG going live yesterday, far more users than normal attempted to access CleanMPG. By 02:00 PM yesterday afternoon, the server was absolutely overwhelmed.

    ___Previously, the server could handle roughly 1,100 simultaneous members and guests before beginning to display Server Busy messages. At 13:48, there were 1,891 members and guests online and thousands more still requesting. I spoke with the Hosting company at least 10 times yesterday to ascertain whether or not what we experienced was a Denial of Service attack. The simple fact is that there was thousands of IP's accessing our server at that very moment. The tech had to actually disconnect the server from the Internet, let the RAID drives re-synch (to restore the validity of the redundant drive), rebooted it, and reattached it to the network. Within 2 to 3 minutes, it was overwhelmed yet again. He added another Gig of RAM which is what we are using now just to stay alive. Even so, the server is on life support as there are 900+ members and guests requesting pages as I type this. Let us pray we make it through today and we should be much better off in the near future.

    ___Sean (Right_Lane_Cruiser) has shut down some automatic functions to help us get through the immediate high traffic period. I suspect we will shut down the Mileage DBase today for a time to allow page requests from Guests to go through as well? A new server is on order but due to the changing pricing policies of the host, it may be a week or two before it is finally in place -- at a much higher price than originally anticipated, unfortunately.

    ___While I am speaking about server plans, let me also introduce you to some special members and guests who have contributed toward 4 months worth of payments for the upcoming server upgrade!

    Larry Trowbridge (DiamondLarry)
    Dan Bryant (Dan)
    Mike Sirach (Msirach)
    John Loser
    Kayleigh Fischer (Lyekka)
    Nigel Burkhardt
    Doug Snyder
    Larry (HemiSync)
    James Hoffman
    Marc Wanagas (Warthog1984)
    Mike Sefton (Msefton)
    Andrew McGuckin (Palemelanesian)
    Laurie With (Laurieaw)
    Manuel Santos (MSantos)
    Reid Stuart (Atlaw4u)
    Doug Penny
    Harold Wilkie
    Kevin Hix
    Keith Pitts
    Tumkur Raghuram
    Michael Overstreet (Skwyre7)
    Charles Smith (Mrchuc)
    Bradlee Fons (Bradlee)
    Justin Fons (Citrus Insighter)
    Michael Hicks
    James Hilliard (Waldo 2001)
    Lester Rhee
    James Andrew
    Keith Gillespie
    James Epting
    Robert Hambidge
    Rich Mazzeo
    Stephen Robie
    Craig Henderson

    ___Please thank them when the opportunity arises and we are back solidly on our feet.

    ___Finally, please accept my apologies for the site not being able to handle your requests -- let alone what was thrown at it yesterday! We knew the CNN stories were coming but none of the mods, Sean or myself had any idea it would bury us like it did :(

    ___Good Luck and thank you for your understanding during these difficult times.

    ___Wayne
     
    Last edited: Jun 6, 2008
  2. xcel

    xcel PZEV, there's nothing like it :) Staff Member

    Hi All:

    ___As load is picking up again this morning; I had Sean turn off the Mileage DBase so our CleanMPG sigs will be out of action for a while to help. Sorry about the delays, fits and starts but this is nuts!

    ___Good Luck

    ___Wayne
     
  3. Right Lane Cruiser

    Right Lane Cruiser Penguin of Notagascar

    I re-enabled the sigs for the time being but I may disable again to save bandwidth and processing.
     
  4. jamtee

    jamtee Well-Known Member

    You may want to put the PayPal button in the header. When I get the server busy message I don't see the paypal button. But I always see the google ad unless the site is totally down.
     
  5. Right Lane Cruiser

    Right Lane Cruiser Penguin of Notagascar

    Good idea -- done.
     
  6. MikeN

    MikeN Well-Known Member

    Hey Wayne... are you buying a server and having it colocated, or are you basically leasing a server through the host?

    It might be too late to do anything else... but I have some ideas depending on your answers :)
     
  7. ericbecky

    ericbecky Member

    I was going to make the same paypal relocation comment yesterday... but I couldn't post. :)

    Keep manning the battlestations, Sean, and keep up the good work my friend!
     
  8. 97PROTEGE

    97PROTEGE Well-Known Member

    Hi Wayne,
    You have to see that this increased demand in server usage is the result of "the word "
    getting out. I see it as a positive sign that there is much more interest in getting the most out of whatever vehicle someone drives. Maybe the media can change peoples habits.

    I'll be sending a donation in the next few days.
    Keep up the good work.
     
  9. xcel

    xcel PZEV, there's nothing like it :) Staff Member

    Hi Mike:

    ___Can you PM me your home phone so I can get some more ideas :) I am just about to speak with the marketing guy for another round of negotiations and any help would be much appreciated.

    ___97 Protégé’, thank you! What bothers me is yesterday we could have used 10 servers of the current capabilities and we would probably still had problems :( Today is much lighter but the current server is doing its thing on the edge. Tomorrow, the current server would run great for another few months if not forever. If we don’t upgrade, it will be something else next week and the place is toast again :ccry: I can just imagine what the likes of Google, YaHoo or Microsoft must have to keep their sites up and running while we had problems with a decently fast but single little P4 server ourselves :rolleyes:

    ___Good Luck

    ___Wayne
     
    Last edited: Jun 4, 2008
  10. Ophbalance

    Ophbalance Administrator Staff Member

    Wait until one of these stories hits slashdot(./). The @!# will truly fly then ;). I've never been on the receiving end for one of those onslaughts, but my day job is administrator for all things ColdFusion (application server, like PHP, JSP, ASP, etc.) at the EPA (which I just find really amusing for some reason). Google bots, yahoo bots, other spiders play merry hell with our systems from time to time. I'm now running on 4 clustered servers to handle the traffic we generate. If ever ya'll need a volunteer, I can pitch in. I've not really touched PHP much, but I'm a fairly competent server admin.
     
  11. Kevin108

    Kevin108 Well-Known Member

    Everybody, if you can, use that big orange Donate button at the top of the page here. I know how much the things I'm learning here can save me. I don't mind donating a little of that, for sure!
     
  12. laurieaw

    laurieaw Sorceress of the North

    done. thanks for helping out.
     
  13. xcel

    xcel PZEV, there's nothing like it :) Staff Member

    Hi All:

    ___This just keeps getting better and better of course :( The techs and Admins at the Hosting site have been absolutely excellent and I have never had any trouble whatsoever with them. Even the RAM upgrade last night to the current Server was post gratis on the labor and RAM costs next to nothing of course.

    ___The tough part is knowing what apps are currently installed? After Tom left, we are now trying to figure this out manually. The Host placed a ticket in for the Admins there to do a quick search of the drives to try and get a list of the apps so that the new CleanMPG server can be built with a compatible or the same OS and apps or the updates of such. Originally it was going to be a simple transfer of the two drives from the P4 to the even faster chipped Dual P4’s but of course that doesn’t work due to the type of platforms these things are built from I was told :(

    ___Tough to do this when one does not really know what one is doing. Thanks to Sean, Dan and the rest, it sure makes the headache more of an annoyance than thousands of dollars of out of pocket expense while not knowing what is going on?

    ___Good Luck and again, thanks to you all for your support …

    ___Wayne
     
  14. Skwyre7

    Skwyre7 Well-Known Member

    Wayne,

    I was saving up for another SG, but I donated what I had saved so far to the site instead. I'd rather have the site running so that others can get help than for me to be able to watch 4 more gauges. I know every little bit helps...
     
  15. Right Lane Cruiser

    Right Lane Cruiser Penguin of Notagascar

    Thank you, Michael!!! That was amazingly selfless -- we really appreciate it!!!
     
  16. lamebums

    lamebums Member

    I swear as soon as I find a job I'm going to send something to CleanMPG.

    As it is, I get back to Kentucky on the 9th so we shall see (I hope I get a job for my own sake! I've been choosing between beer and gas over here!)
     
  17. mrchuc

    mrchuc Member

    I just sent a small donation. Wish it could be more. It amounts to what I figured I saved on just the purchase of my SG-II. Payday is coming. Maybe I can do more. Thanks everyone!

    Chuc
     
  18. Skwyre7

    Skwyre7 Well-Known Member

    That's a tough choice sometimes. ;)
     
  19. xcel

    xcel PZEV, there's nothing like it :) Staff Member

    Hi Mike and Chuck:

    ___Thank you so very much for the donations! We have had quite a few today and I will look into adding a star or something next to your name/avatar to make you all stand out because right now, we need all the help we can get ;)

    ___Now I need some Server guys advice …

    ___Here is what I am looking at currently and the $Costs are much higher than I am paying currently :(

    *******************************************

    Additionally, you could elect for the 3-5 hours of admin time at $85/hr. to migrate your data and apps to the SuperMicro server.

    SuperMicro Server with dual (2) 2.4GHz Xeon CPUs, 1GB RAM, (1) 80GB drive for $149/mo. including 800 GB per month.

    SuperMicro Server with dual (2) 2.4GHz Xeon CPUs, 1GB RAM, (2)x80GB RAID 1 drives for $159/mo. including 800 GB per month.

    SuperMicro Server with dual (2) 2.4GHz Xeon CPUs, 2GB RAM, (2)x80GB RAID 1 drives for $179/mo. including 800 GB per month.

    *******************************************


    ___(2) GB’s of ECC RAM for $20 is too much but that is what I was offered. Also, that machine uses older IDE tech with the dual P4’s vs. the currents single P4 with SATA.

    ___The next step up is really out of my price range but it is the latest stuff although way overkill for our site right now :ccry:

    *******************************************


    Dell Pro400

    Dell PowerEdge SC1425 Server

    Dual Intel 3.0GHz Hyper Threading Xeon CPUs - each with 2MB L2 Cache & 800MHz FSB, 2GB ECC DDR2 RAM, 2x80GB SATA RAID 1 HDDs (Existing drives from cleanmpg server) for $249/mo. including 800 GB per month.

    *******************************************

    ___Can anyone else add input. Mike N. offered some great advice for possibly the next generation upgrade but even he said you aren’t even close to the stops with the Dual 2.4 SuperMicro by any means :( Dbases need RAM and BUS speeds vs. a massive amount of CPU power IIRC?

    ___Good Luck and thanks in advance to any and all replies?

    ___Wayne
     
  20. PaleMelanesian

    PaleMelanesian Beat the System Staff Member

    Wayne, I agree that databases work more on ram and bus, but also depend on hard drive speeds (helped by raid). Raw processor speed is not as important. However, the duality of the processor should make a big difference.
     
    Last edited: Jun 4, 2008

Share This Page