Like Tree3Likes
  • 1 Post By Salem
  • 1 Post By Nominal Animal
  • 1 Post By Nominal Animal

Massive storage contest at Techgage

This is a discussion on Massive storage contest at Techgage within the General Discussions forums, part of the Community Boards category; Some of you know I'm a staff member and editor at Techgage. I don't ever try to promote that website ...

  1. #1
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,383

    Massive storage contest at Techgage

    Some of you know I'm a staff member and editor at Techgage. I don't ever try to promote that website here in any way, not even through my signature. And I never speak of it. However once and a while something happens there that I feel may be of interest to my fellow Cboarders.

    This is the second time I do this here. Anyone concerned with this post please PM me.

    Techgage is currently holding a contest in partnership with Kingston for rather excellent prizes (SSDs, RAM, Wi-Drives, USB Pens, SDHCs, etc). In order to participate no signing up to Techgage or Kingston websites is required.

    Kingston & Techgage Present: Free Kit for Friendship Contest – Techgage

    Best of luck.
    Last edited by Mario F.; 11-19-2012 at 11:58 AM.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  2. #2
    and the hat of wrongness Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    32,338
    Boooooo
    This contest runs until December 10, 8:00 AM EST, and is open to those living in the United States and Canada. There are three prize packs to be won!
    Mario F. likes this.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.
    I support http://www.ukip.org/ as the first necessary step to a free Europe.

  3. #3
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,383
    I know

    That stuff annoys me beyond belief. It's country laws and sometimes also impositions by the companies offering the contest. Bah!
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  4. #4
    Left
    Join Date
    Oct 2011
    Posts
    816
    For a second there I thought you were telling us about a programming contest related to massive storage problems.

    I thought there might finally be something interesting to think about
    laserlight likes this.

  5. #5
    SAMARAS std10093's Avatar
    Join Date
    Jan 2011
    Location
    Glyfada,Athens
    Posts
    2,650
    Animal if you look at post #2, you will see that the contest is open only to those that line in US and Canada... :/
    (your location says Helsinki )

  6. #6
    Registered User
    Join Date
    Sep 2008
    Location
    Toronto, Canada
    Posts
    1,831
    I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level. Seems this is nothing but a clever advertisement for Kingston products.

  7. #7
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,383
    Quote Originally Posted by nonoob View Post
    I wish the actual programming problem/question was easy to find. That way I can know whether I should even bother to register if the challenge is above my skill level.
    I'm not sure what made you think there was a programming challenge in there. In any case, it's a safe bet, if you wish to search for programming topics, to not include this General Discussions subforum (subtitled "Non-programming related topics") in your search.

    Quote Originally Posted by nonoob View Post
    Seems this is nothing but a clever advertisement for Kingston products.
    I can help you clear any doubts. It is a clever advertisement for Kingston products. Why else would that company hold a contest?
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  8. #8
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,383
    Quote Originally Posted by Nominal Animal View Post
    For a second there I thought you were telling us about a programming contest related to massive storage problems.

    I thought there might finally be something interesting to think about
    No worries. I can give you something interesting to think about.

    Many of us would like a relational filesystem. In the likes of DBFS or the now defunct WinFS. This is true, for instance, of anyone wishing to develop a DMS. However Windows isn't happy about anyone wanting to replace NTFS. How to develop a relational-like filesystem and abstract it away from the NTFS store with a minimum impact on IO performance?
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

  9. #9
    Left
    Join Date
    Oct 2011
    Posts
    816
    Quote Originally Posted by Mario F. View Post
    Many of us would like a relational filesystem.
    I don't use Windows, but Dokan seems like it might be a good starting point there.

    On Linux, you could achieve that using FUSE, by writing a userspace filesystem driver. (See the list of database filesystem projects at SourceForge, for example.)

    The Oracle DBFS seems to be also based on FUSE on Linux, but I can't tell whether it is for binary blobs (SecureFiles) only, or whether it provides a real interface to the database.

    Unfortunately, I'm not that good with relational databases, as I'm more of a structured data person. I usually work with structured data produced in a simulation of some kind; typically there's lots of data to handle, sometimes distributed over a number of systems. I really don't know enough about the real world use cases, and efficiency requirements, to map something like a subset of SQL to a filesystem interface. (Although Michael Raskin's article on query filesystems (PDF) is certainly interesting.)

    If you ask me, I'd look into replacing SQL with a simple C API instead, using well-defined structures (instead of strings) to describe the queries and responses -- i.e. split into components, each component separately specified --, and make sure it is easy to use correctly for typical use cases. Bindings for other languages can be written afterwards, as the C basis should yield maximum portability.

    In particular, if names and values were specified using a C pointer and a length, then they would be truly opaque to the library/database server, in the sense that no escaping would ever be necessary, and there should be no possibility of buffer overrun. Helper functions would be used to construct queries before dispatch. Pool-based memory management (similar to what Apache does to keep track of all memory related to a specific query, or other task) would help programmers avoid usual dynamic memory management pitfalls, and so on. The interface could be asynchronous, using sockets perhaps, so that a lazy programmer would not need to learn any new complex stuff, but proficient programmers could interleave work and database access easily, using techniques familiar from non-blocking or async I/O.

    I think that would be as easy to use for programmers as a relational filesystem would be. The main difference is that SQL generation and parsing is not only slow (it is a relatively complicated syntax to parse or to generate), but it is also too easy to do wrong, causing all sorts of security issues. Vulnerability to Bobby Tables and similar seem to be especially common in web applications.
    Mario F. likes this.

  10. #10
    (?<!re)tired Mario F.'s Avatar
    Join Date
    May 2006
    Location
    Portugal
    Posts
    7,383
    I'll take a look at this Dokan library. Thank you. I'm more interested in the driver code (assuming I can make anything of it). It seems too centered around the traditional directory/file structure, for which I have no use. But if I can understand the driver code, I'll gain the necessary knowledge to try and make something of my own.

    My concern with a user space file system is that this will not take advantage of DMA. I seem to recall reading somewhere that as of Windows 7, support for user mode DMA is still not complete.

    Thank you once again.
    The programmer’s wife tells him: “Run to the store and pick up a loaf of bread. If they have eggs, get a dozen.”
    The programmer comes home with 12 loaves of bread.


    Originally Posted by brewbuck:
    Reimplementing a large system in another language to get a 25% performance boost is nonsense. It would be cheaper to just get a computer which is 25% faster.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Massive Unicode Grumble....
    By CommonTater in forum General Discussions
    Replies: 25
    Last Post: 05-20-2011, 01:55 PM
  2. Massive Steam sale
    By VirtualAce in forum General Discussions
    Replies: 16
    Last Post: 07-05-2010, 01:45 PM
  3. a massive data structure
    By AngKar in forum C Programming
    Replies: 1
    Last Post: 01-18-2006, 06:31 PM
  4. getting massive compile errors, need help
    By jlmac2001 in forum C++ Programming
    Replies: 3
    Last Post: 03-20-2003, 03:05 PM
  5. Scope and Massive arrays
    By Unregistered in forum C++ Programming
    Replies: 1
    Last Post: 04-05-2002, 03:28 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21