Thread: BIG Sorting Problem!

Originally Posted by Adak

[...]

A database.

Well, ah - that would be NO. I intend to work everything in RAM as much as possible, so that's mmap function sounds like a winner. Thanks, I'll read up on it.

Right now I'm writing up some multi-key comparison code in Quicksort.

Thanks for your reply.

Then mmap() won't help, unless you want to use a raw binary file in your computer as a big cache of the database.

mmap() would be the way to go if the data were in a huge binary file instead of a database.

Since you have a database, the ideal would be to read/write large blocks of data and work on them. If there is some transparent way to do this buffering, use it.

Since you are using quicksort, you might want to have two buffers (or sliding windows): one for the lower extreme and one for the upper extreme.

One stupid question: Can't your database sort the data for you?

Originally Posted by comocomocomo

Then mmap() won't help, unless you want to use a raw binary file in your computer as a big cache of the database.

mmap() would be the way to go if the data were in a huge binary file instead of a database.

Since you have a database, the ideal would be to read/write large blocks of data and work on them. If there is some transparent way to do this buffering, use it.

Since you are using quicksort, you might want to have two buffers (or sliding windows): one for the lower extreme and one for the upper extreme.

One stupid question: Can't your database sort the data for you?

The "database" is what was inferred - but it's output FROM a database - which isn't the same thing. The actual raw data format, has not been shared with me yet. (and I doubt it will be, since I'm a very unofficial helper type).

As I understand it, the project won't be going forward without showing the ability to sort the data, and obtain useful info from the work. Whether this is a commercial enterprise or simply an educational project to see if it's feasible, is up in the air.