Recent Downtime

This is a discussion on Recent Downtime within the General Discussions forums, part of the Community Boards category; It looks like we ran out of both RAM and swap space causing the system to completely lock up. This ...

  1. #1
    Administrator webmaster's Avatar
    Join Date
    Aug 2001
    Posts
    975

    Recent Downtime

    It looks like we ran out of both RAM and swap space causing the system to completely lock up. This was due to a combination of factors, I think--for one, we'd been running pretty low on RAM recently. Second, the nightly backup process kicked in, running gzip on some large (GB+) files. Finally, memcached seems to have run up to about 280MB of memory usage (despite my attempt at configuring it to us only 64MB).

    At any rate, these stresses caused some serious strain on the overall system by, I believe, slowing down DB access considerably, resulting in the DB maxing out the number of connections available. This killed the C Board and also led to a ton of open HTTP connections. I suspect that some combination of all of these activities created a cascading failure that forced a server reboot.

    It isn't clear if this would have happened had vBulletin 3 still been installed (either if it was caused memcached pushing the memory limits over the top or more processing required for vBulleitn 4), but based on what I was seeing in top, I suspect that adding additional RAM to the system is a good step to solving tis problem. We're currently running with only 2GB of RAM and a 2GB swap.

  2. #2
    Banned
    Join Date
    Aug 2010
    Location
    Ontario Canada
    Posts
    9,547
    Quote Originally Posted by webmaster View Post
    We're currently running with only 2GB of RAM and a 2GB swap.
    ... and while you're at it, you might consider replacing that trusty old 386 with one of them new fangled AMD thingies...

  3. #3
    Administrator webmaster's Avatar
    Join Date
    Aug 2001
    Posts
    975
    Actually, we're running with an AMD CPU now.

    It looks like tonight's downtime happened at the same time as last night's--my theory is that they might both have happened around the same time and been triggered by nightly backups. Fortunately, another reboot solved the issue, and I have a few other ideas about why it might have happened again--e.g. the nightly backups currently need to deal with a much larger amount of data since I have a 1.6GB dump of the C board as a backup prior to the vBulletin upgrade. I can easily move this out of the backup dir and reduce the size of the backup files a lot.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. a recent ms documentation?
    By Yarin in forum A Brief History of Cprogramming.com
    Replies: 4
    Last Post: 09-05-2008, 01:30 PM
  2. Recent Extended Downtime
    By webmaster in forum A Brief History of Cprogramming.com
    Replies: 24
    Last Post: 11-06-2004, 08:53 PM
  3. Downtime tonight!! please read
    By axon in forum A Brief History of Cprogramming.com
    Replies: 11
    Last Post: 06-16-2004, 11:41 PM
  4. Recent Extended Downtime
    By webmaster in forum A Brief History of Cprogramming.com
    Replies: 28
    Last Post: 02-21-2004, 12:54 PM
  5. Cprogramming Downtime
    By xddxogm3 in forum A Brief History of Cprogramming.com
    Replies: 3
    Last Post: 02-20-2004, 04:20 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21