Thread: Massive storage contest at Techgage

  1. #1
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446

    Massive storage contest at Techgage

    Some of you know I'm a staff member and editor at Techgage. I don't ever try to promote that website here in any way, not even through my signature. And I never speak of it. However once and a while something happens there that I feel may be of interest to my fellow Cboarders.

    This is the second time I do this here. Anyone concerned with this post please PM me.

    Techgage is currently holding a contest in partnership with Kingston for rather excellent prizes (SSDs, RAM, Wi-Drives, USB Pens, SDHCs, etc). In order to participate no signing up to Techgage or Kingston websites is required.

    Kingston & Techgage Present: Free Kit for Friendship Contest – Techgage

    Best of luck.
    Last edited by Mario F.; 11-19-2012 at 12:58 PM.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    Boooooo
    This contest runs until December 10, 8:00 AM EST, and is open to those living in the United States and Canada. There are three prize packs to be won!
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    I know

    That stuff annoys me beyond belief. It's country laws and sometimes also impositions by the companies offering the contest. Bah!
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  4. #4
    Ticked and off
    Join Date
    Oct 2011
    Location
    La-la land
    Posts
    1,728
    For a second there I thought you were telling us about a programming contest related to massive storage problems.

    I thought there might finally be something interesting to think about

  5. #5
    SAMARAS std10093's Avatar
    Join Date
    Jan 2011
    Location
    Nice, France
    Posts
    2,694
    Animal if you look at post #2, you will see that the contest is open only to those that line in US and Canada... :/
    (your location says Helsinki )

  6. #6
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,834
    I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level. Seems this is nothing but a clever advertisement for Kingston products.

  7. #7
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    Quote Originally Posted by nonoob View Post
    I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level.
    I'm not sure what made you think there was a programming challenge in there. In any case, it's a safe bet, if you wish to search for programming topics, to not include this General Discussions subforum (subtitled "Non-programming related topics") in your search.

    Quote Originally Posted by nonoob View Post
    Seems this is nothing but a clever advertisement for Kingston products.
    I can help you clear any doubts. It is a clever advertisement for Kingston products. Why else would that company hold a contest?
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  8. #8
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    Quote Originally Posted by Nominal Animal View Post
    For a second there I thought you were telling us about a programming contest related to massive storage problems.

    I thought there might finally be something interesting to think about
    No worries. I can give you something interesting to think about.

    Many of us would like a relational filesystem. In the likes of DBFS or the now defunct WinFS. This is true, for instance, of anyone wishing to develop a DMS. However Windows isn't happy about anyone wanting to replace NTFS. How to develop a relational-like filesystem and abstract it away from the NTFS store with a minimum impact on IO performance?
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  9. #9
    Ticked and off
    Join Date
    Oct 2011
    Location
    La-la land
    Posts
    1,728
    Quote Originally Posted by Mario F. View Post
    Many of us would like a relational filesystem.
    I don't use Windows, but Dokan seems like it might be a good starting point there.

    On Linux, you could achieve that using FUSE, by writing a userspace filesystem driver. (See the list of database filesystem projects at SourceForge, for example.)

    The Oracle DBFS seems to be also based on FUSE on Linux, but I can't tell whether it is for binary blobs (SecureFiles) only, or whether it provides a real interface to the database.

    Unfortunately, I'm not that good with relational databases, as I'm more of a structured data person. I usually work with structured data produced in a simulation of some kind; typically there's lots of data to handle, sometimes distributed over a number of systems. I really don't know enough about the real world use cases, and efficiency requirements, to map something like a subset of SQL to a filesystem interface. (Although Michael Raskin's article on query filesystems (PDF) is certainly interesting.)

    If you ask me, I'd look into replacing SQL with a simple C API instead, using well-defined structures (instead of strings) to describe the queries and responses -- i.e. split into components, each component separately specified --, and make sure it is easy to use correctly for typical use cases. Bindings for other languages can be written afterwards, as the C basis should yield maximum portability.

    In particular, if names and values were specified using a C pointer and a length, then they would be truly opaque to the library/database server, in the sense that no escaping would ever be necessary, and there should be no possibility of buffer overrun. Helper functions would be used to construct queries before dispatch. Pool-based memory management (similar to what Apache does to keep track of all memory related to a specific query, or other task) would help programmers avoid usual dynamic memory management pitfalls, and so on. The interface could be asynchronous, using sockets perhaps, so that a lazy programmer would not need to learn any new complex stuff, but proficient programmers could interleave work and database access easily, using techniques familiar from non-blocking or async I/O.

    I think that would be as easy to use for programmers as a relational filesystem would be. The main difference is that SQL generation and parsing is not only slow (it is a relatively complicated syntax to parse or to generate), but it is also too easy to do wrong, causing all sorts of security issues. Vulnerability to Bobby Tables and similar seem to be especially common in web applications.

  10. #10
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Ireland
    Posts
    8,446
    I'll take a look at this Dokan library. Thank you. I'm more interested in the driver code (assuming I can make anything of it). It seems too centered around the traditional directory/file structure, for which I have no use. But if I can understand the driver code, I'll gain the necessary knowledge to try and make something of my own.

    My concern with a user space file system is that this will not take advantage of DMA. I seem to recall reading somewhere that as of Windows 7, support for user mode DMA is still not complete.

    Thank you once again.
    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Massive Unicode Grumble....
    By CommonTater in forum General Discussions
    Replies: 25
    Last Post: 05-20-2011, 01:55 PM
  2. Massive Steam sale
    By VirtualAce in forum General Discussions
    Replies: 16
    Last Post: 07-05-2010, 01:45 PM
  3. a massive data structure
    By AngKar in forum C Programming
    Replies: 1
    Last Post: 01-18-2006, 07:31 PM
  4. getting massive compile errors, need help
    By jlmac2001 in forum C++ Programming
    Replies: 3
    Last Post: 03-20-2003, 04:05 PM
  5. Scope and Massive arrays
    By Unregistered in forum C++ Programming
    Replies: 1
    Last Post: 04-05-2002, 04:28 PM