Thread: Whoa, what happened?

  1. #1
    Administrator webmaster's Avatar
    Join Date
    Aug 2001
    Posts
    1,012

    Whoa, what happened?

    Over approximately the last five days (starting on Jan 4, 2009), the database backing the message board was corrupted. I believe this corruption occurred on Jan 4, as a side effect of running out of space on the tmp partition of the server.

    MySQL can recover corrupted databases, but the process is lengthy. After letting the recovery process run for the last four days, I realized that all statistics indicated that the recovery could potentially take weeks given the enormous size of the database that back the message board. (It's over 1GB total, with multiple millions of rows of sorted data--sorting is the typical technique used to recover the DB tables.)

    Rather than spending a month to try to save 5 days worth of posts (while simultaneously making the boards unavailable for posts), I decided to restore from a recent DB backup. The backup was created on 12/31. Any posts or other actions that were taken after 12/31 are GONE.

    This includes a post I made indicating that I believed I had fixed some of the database errors that were slowing down the forums. I still believe that I have fixed those errors, and that the multi-day day downtime that we experienced was a result of filling up the tmp directory as part of that investigation.

    Please report any database errors or other problems that you experience, other than lost posts from the last 5 days.

  2. #2
    Jack of many languages Dino's Avatar
    Join Date
    Nov 2007
    Location
    Chappell Hill, Texas
    Posts
    2,332
    Thanks for fixing it.
    Mainframe assembler programmer by trade. C coder when I can.

  3. #3
    Officially An Architect brewbuck's Avatar
    Join Date
    Mar 2007
    Location
    Portland, OR
    Posts
    7,396
    I administer my wife's online business server, and I've had this happen to me as well. It's the kind of mistake you only make once
    Code:
    //try
    //{
    	if (a) do { f( b); } while(1);
    	else   do { f(!b); } while(1);
    //}

  4. #4
    Registered User QuestionKing's Avatar
    Join Date
    Jan 2009
    Posts
    68
    When I log in, new unread topics are in bold (as expected), but opening any unread post will set all unread posts to the normal type (as if each one had been visited). It did not work this way before. Now every time I return to the listing after reading one new post, I have to try to figure out what is new and what is old.
    Asking a question you already know the answer to might teach you something you did not know...

  5. #5
    Registered User
    Join Date
    Jan 2008
    Posts
    290
    Quote Originally Posted by QuestionKing View Post
    When I log in, new unread topics are in bold (as expected), but opening any unread post will set all unread posts to the normal type (as if each one had been visited). It did not work this way before. Now every time I return to the listing after reading one new post, I have to try to figure out what is new and what is old.
    Are you sure? I can't reproduce the behavior you're describing. Everything works exactly as it usually does. Perhaps the problem is client-side (cookie-related)?

  6. #6
    and the hat of sweating
    Join Date
    Aug 2007
    Location
    Toronto, ON
    Posts
    3,545
    Quote Originally Posted by QuestionKing View Post
    When I log in, new unread topics are in bold (as expected), but opening any unread post will set all unread posts to the normal type (as if each one had been visited). It did not work this way before. Now every time I return to the listing after reading one new post, I have to try to figure out what is new and what is old.
    Are you sure you're still logged in?
    I noticed something like this with Firefox on my Linux system at work. I left a tab open to this site 24/7 and after a while all posts looked like they were already read. Then I noticed I wasn't logged in anymore. When I logged in, everything was fine.
    "I am probably the laziest programmer on the planet, a fact with which anyone who has ever seen my code will agree." - esbo, 11/15/2008

    "the internet is a scary place to be thats why i dont use it much." - billet, 03/17/2010

  7. #7
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    I confess I don't exactly understand what happened. I didn't know filling up tmp could lead to data corruption. What exactly are you using it for? I'd expect temporary tables only besides normal mysql temp files?

    Had a few similar problems before but all I got was the expected crash and a log message. Thankfully never got corrupted data. But I don't understand how it could get corrupted unless you put something else in there.

    In any case good job fixing it. And thanks. Restoring the database was a noble attempt.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  8. #8
    In my head happyclown's Avatar
    Join Date
    Dec 2008
    Location
    In my head
    Posts
    391
    I was banned on 3 Jan, for defending my belief that gets, scanf and goto are perfectly safe.

    Now that we're back to 31 Dec, I am unbanned!

    Woohoo!!
    OS: Linux Mint 13(Maya) LTS 64 bit.

  9. #9
    In my head happyclown's Avatar
    Join Date
    Dec 2008
    Location
    In my head
    Posts
    391
    Just kidding!
    OS: Linux Mint 13(Maya) LTS 64 bit.

  10. #10
    Registered User QuestionKing's Avatar
    Join Date
    Jan 2009
    Posts
    68
    @ arpsmack & cpjust - yes, I am sure and yes I am still logged in
    Nothing client side has changed. Cookies are enabled, remember me is checked, I am automatically logged in when the page opens. : / Your guess is better than mine. I just tried again after checking everything, there were 6 new posts, and after viewing the first of only 6 in bold, they are all back to normal type again. I will log out and log in again and check it out (after waiting for new posts) but I don't expect anything to change.

    "Welcome, QuestionKing. You last visited: Today at 03:33 PM"
    :/ I have not manually logged in for so long, I am going to have to look up the p/w.
    Using IE v 8 . 0 . 6001 . 18702
    As far as reproducing the problem, it happens every time without fail from here. I will also try to check from another pc.
    Asking a question you already know the answer to might teach you something you did not know...

  11. #11
    Registered User QuestionKing's Avatar
    Join Date
    Jan 2009
    Posts
    68
    Ok, interesting...
    As soon as I posted the above message, I returned to the C++ forum and the remaining 5 "new-to-me" posts were back in bold. Furthermore everything seems to function normally now. I did forget to mention a detail, when returning to the forum I use the link at the top of the page to return to the forum listing, rather than prev/next post or the browser's back button. For example C Board > Community Boards > General Discussions on this page.
    Asking a question you already know the answer to might teach you something you did not know...

  12. #12
    Registered User VirtualAce's Avatar
    Join Date
    Aug 2001
    Posts
    9,607
    I'm getting deja vu answering some of the posts. I think I've already answered them at one time but due to the db issue they were lost. Oh well at least it gets the posts and threads moving again.

    Thanks for fixing the db. I think the decision to move forward instead of wasting 30 days to save 5 days of posts is a good one indeed.

  13. #13
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    Look on the good side: maybe this time your answers will be improved without the need for editing
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  14. #14
    Registered User VirtualAce's Avatar
    Join Date
    Aug 2001
    Posts
    9,607
    Nice. Nope they still need edited. I can't type to save my life.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Pls repair my basketball program
    By death_messiah12 in forum C++ Programming
    Replies: 10
    Last Post: 12-11-2006, 05:15 AM
  2. What happened?
    By KrAzY CrAb in forum A Brief History of Cprogramming.com
    Replies: 4
    Last Post: 02-25-2003, 07:10 AM
  3. WHat happened to the master...
    By vasanth in forum A Brief History of Cprogramming.com
    Replies: 4
    Last Post: 02-04-2003, 03:53 AM
  4. What happened to the GD?
    By kermi3 in forum A Brief History of Cprogramming.com
    Replies: 8
    Last Post: 08-15-2002, 06:01 PM
  5. What happened to my thread??
    By VirtualAce in forum A Brief History of Cprogramming.com
    Replies: 4
    Last Post: 04-24-2002, 04:27 PM