Massive storage contest at Techgage

Some of you know I'm a staff member and editor at Techgage. I don't ever try to promote that website here in any way, not even through my signature. And I never speak of it. However once and a while something happens there that I feel may be of interest to my fellow Cboarders.

This is the second time I do this here. Anyone concerned with this post please PM me.

Techgage is currently holding a contest in partnership with Kingston for rather excellent prizes (SSDs, RAM, Wi-Drives, USB Pens, SDHCs, etc). In order to participate no signing up to Techgage or Kingston websites is required.

Kingston & Techgage Present: Free Kit for Friendship Contest – Techgage

Best of luck.

Boooooo :( :( :(

Quote:

---

This contest runs until December 10, 8:00 AM EST, and is open to those living in the United States and Canada. There are three prize packs to be won!

---

I know :(

That stuff annoys me beyond belief. It's country laws and sometimes also impositions by the companies offering the contest. Bah!

For a second there I thought you were telling us about a programming contest related to massive storage problems.

I thought there might finally be something interesting to think about ;)

Animal if you look at post #2, you will see that the contest is open only to those that line in US and Canada... :/
(your location says Helsinki :p )

I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level. Seems this is nothing but a clever advertisement for Kingston products.

Quote:

---

Originally Posted by nonoob

I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level.

---

I'm not sure what made you think there was a programming challenge in there. In any case, it's a safe bet, if you wish to search for programming topics, to not include this General Discussions subforum (subtitled "Non-programming related topics") in your search.

Quote:

---

Originally Posted by nonoob

Seems this is nothing but a clever advertisement for Kingston products.

---

I can help you clear any doubts. It is a clever advertisement for Kingston products. Why else would that company hold a contest?

Quote:

---

Originally Posted by Nominal Animal

For a second there I thought you were telling us about a programming contest related to massive storage problems.

I thought there might finally be something interesting to think about ;)

---

No worries. I can give you something interesting to think about.

Many of us would like a relational filesystem. In the likes of DBFS or the now defunct WinFS. This is true, for instance, of anyone wishing to develop a DMS. However Windows isn't happy about anyone wanting to replace NTFS. How to develop a relational-like filesystem and abstract it away from the NTFS store with a minimum impact on IO performance?

Quote:

---

Originally Posted by Mario F.

Many of us would like a relational filesystem.

---

I don't use Windows, but Dokan seems like it might be a good starting point there.

On Linux, you could achieve that using FUSE, by writing a userspace filesystem driver. (See the list of database filesystem projects at SourceForge, for example.)

The Oracle DBFS seems to be also based on FUSE on Linux, but I can't tell whether it is for binary blobs (SecureFiles) only, or whether it provides a real interface to the database.

Unfortunately, I'm not that good with relational databases, as I'm more of a structured data person. I usually work with structured data produced in a simulation of some kind; typically there's lots of data to handle, sometimes distributed over a number of systems. I really don't know enough about the real world use cases, and efficiency requirements, to map something like a subset of SQL to a filesystem interface. (Although Michael Raskin's article on query filesystems (PDF) is certainly interesting.)

If you ask me, I'd look into replacing SQL with a simple C API instead, using well-defined structures (instead of strings) to describe the queries and responses -- i.e. split into components, each component separately specified --, and make sure it is easy to use correctly for typical use cases. Bindings for other languages can be written afterwards, as the C basis should yield maximum portability.

In particular, if names and values were specified using a C pointer and a length, then they would be truly opaque to the library/database server, in the sense that no escaping would ever be necessary, and there should be no possibility of buffer overrun. Helper functions would be used to construct queries before dispatch. Pool-based memory management (similar to what Apache does to keep track of all memory related to a specific query, or other task) would help programmers avoid usual dynamic memory management pitfalls, and so on. The interface could be asynchronous, using sockets perhaps, so that a lazy programmer would not need to learn any new complex stuff, but proficient programmers could interleave work and database access easily, using techniques familiar from non-blocking or async I/O.

I think that would be as easy to use for programmers as a relational filesystem would be. The main difference is that SQL generation and parsing is not only slow (it is a relatively complicated syntax to parse or to generate), but it is also too easy to do wrong, causing all sorts of security issues. Vulnerability to Bobby Tables and similar seem to be especially common in web applications.

I'll take a look at this Dokan library. Thank you. I'm more interested in the driver code (assuming I can make anything of it). It seems too centered around the traditional directory/file structure, for which I have no use. But if I can understand the driver code, I'll gain the necessary knowledge to try and make something of my own.

My concern with a user space file system is that this will not take advantage of DMA. I seem to recall reading somewhere that as of Windows 7, support for user mode DMA is still not complete.

Thank you once again.