Printable View
It looks like we had a database failure similar to the last one. This time, I immediately decided to recover from backup, but it still takes a while to restore the backups. We've lost a few days of posts, as I had to restore from last week's backup.
I'm not yet certain what caused the actual corruption of the DB that required recovery.
I guess it would be a good idea to make more frequent backups for the next few weeks or so...
Just out of curiosity, what database do you use?
MySQL was what was mentioned previously.
So MySQL has some unresolved issues then? Wondering if I should switch DB's now. :p
Any ideas about possible culprits?
I'd look into a hardware failure before looking into the database.
I don't see how mySQL could corrupt a database file in this manner. Right the first time it happened I was surprised to learn it was because of lack of space on the tmp partition. And I still am. I don't know how that could possibly corrupt a database file, unless the actual file or the indexes were stored there.
And now it happens again. Either you "found" a yet unknown bug, or the tmp partition is being misused and you should know better, or the hardware is at fault... memory?
edit: or vBulletin has some weird bug.
I say it was Yarin. It was the only way he could get that list of IP's taken down, otherwise the folks from Amsterdam will kill his entire family then chop his fingers off.Quote:
Originally Posted by webmaster
No - it was vBulletin and they threatened to sack all the adverts and smiley's, dammit...couldn't have that, now could we? =}Quote:
Originally Posted by MK27
Thanks for inspiring my next avatar.Quote:
Originally Posted by Sebastiani
Oh, right.Quote:
Originally Posted by MK27
Code:_ ___
/ \ \
| =\} |
\___\_/
This is killing my board mojo. Normally I remember where certain threads are and what has been said and now with two failures I'm lost as to where each thread is.
Thanks for fixing it quickly. Why has this suddenly become an issue after all these years? Does it have to do with the board updates or possibly a bug in the board software?
While I agree hardware may be a factor, hardware is often only the means by which software design problems become evident.Quote:
Originally Posted by Mario F.
There are a lot of I/O operations involved in managing user connections, data received or sent over user connections, and data that gets stored in the database. If code doing those various I/O operations does not detect failure of an I/O operation, then data corruption can invisibly occur along the way.Quote:
Originally Posted by Mario F.
If some of those I/O operations involve writing data (temporarily) to /tmp or reading from there, then one way of causing a I/O failure is an attempt to write more data to /tmp than allowed. If some other process - the SQL server itself is one of several candidates - assumes data in /tmp is valid, and just copies it "verbatim" into the database, then the database itself may be corrupted.
It's not difficult to envisage scenarios in which the data placed in /tmp at any point in time depends on the amount of data in the database, the number of connected users over the last 24 hours, or other things. All things that are associated with the forum site being more active.
It is also not difficult to envisage scenarios in which some file written to /tmp is not cleaned up when no longer needed, so /tmp gradually fills up over time.
It all depends on the quality of implementation of various bits of software (the operating system, device drivers, the php code that talks to the SQL server, the php interpreter, the SQL server ......).
But, shouldn't an entire transaction fail if a write fails? But then again, transactions aren't really used a lot...Quote:
Originally Posted by grumpy
But still, write() should be checked for failure - as it can fail - and should it fail, shouldn't a database do anything it can to prevent the database from going corrupt? And I can't imagine there's nothing it can do.
I'm not saying there's nothing that can be done. I certainly agree write()s should be checked for failure. But all it needs is an oversight so an error condition is not detected, and Voila!Quote:
Originally Posted by EVOEx
There's nothing magical about database servers (or any other software that is a component of a forum suite) that make them less prone to design or coding deficiencies than any other software of reasonable size (however you measure "size"). The probability of one or more defects lurking in code, just waiting to be triggered by the right set of input conditions, does tend to increase with code size. The probability also decreases as technical rigour in the development process increases, but it's very difficult to anticipate all possible error conditions or to find the causes of errors that occur quite rarely in practice (eg after a few months of up time on a site like this). And exhaustive testing is prohibitively expensive too.
Even developers of safety critical systems - where people can die if something goes wrong - can't realistically aim for a zero defect rate. Expecting that in a general purpose database is a tall order .....
Yet another reason I've never been a fan of vBulletin; but hey, maybe that's because as a kid I couldn't afford a license and was forced to use alternatives like Ikonboard -> IPB. Database error/bugs with vBulletin aren't as common as say ASP products (database error hell), but with the PHP alternatives it's a wonder people keep using vB. Perhaps try upgrading?
I agree entirely grumpy. But my comment about the use of the /tmp partition didn't take into account the possibility of the sql server using this partition for anything else other than what's established by the engine.Quote:
Originally Posted by grumpy
That is, as far as I know, when the SQL Server engine is using this partition for its own internal operations (which include among other things the creation of temporary tables as required by an execution plan), I can't see any way a database could be corrupt, even in the face of exhaustion of storage space. The engine includes all code necessary for proper checking of write operations and it's been years since I've personally last hear of any problem in this area(*).
However, it is true that if the user of the sql engine (read, Vbulletin on this case) is also using the /tmp partition for storage of its own temporary tables and what else it feels should go there, then indeed there's a risk for corruption if that user doesn't do all the necessary write/read checks.
But then this begs the question: What on earth made Vbulletin programmers think they should use /tmp for storage of temporary tables? Or, if this is a setting on Vbulletin, why is that it is being set to /tmp?
It shouldn't. User-made temporary tables shouldn't go into /tmp.
...
(*) currently, the behavior is to terminate the server and write a log entry. All operations are rolled back and the database(s) are left intact.