Using a file like Virtual Memory

This is a discussion on Using a file like Virtual Memory within the C Programming forums, part of the General Programming Boards category; Hello, I put on an earlier post about using linked lists for data, but I've seen the error of my ...

  1. #1
    Registered User
    Join Date
    Jun 2008
    Location
    Northern Va
    Posts
    18

    Unhappy Using a file like Virtual Memory

    Hello,
    I put on an earlier post about using linked lists for data, but I've seen the error of my ways (and the limited RAM which I tried to employ). What I would like to do now is read and write to a file (hopefully one) to store frequency data. What I'd like my program to do is load a 3d array of data and write that data to a file when a datum is outside the specifications. Then, when another datum is accessed/generated within a certain range, if that is already present in the file to load a chunk of data to the array, write the file with the chunk currently in there, and update the new chunk. Here's some pseudocode to illustrate what I'm talking about:

    Sample datum, in format (A,B,Theta,Frequency): 0 200 180 1

    Code:
    make a 3d array of doubles
    while(there are still more sample points)\
    {
    -Generate specific sample point
    -if(sample point is within the array's boundaries)
       add 1 to the appropriate spot
       continue
    -else
      if( the current array has already been stored)
         update the data in the file
      else
         insert the array data into the file
      flush the array and change it's boundary conditions
      load data in the region of the new point to the array (if it exists)
      Store the Sample point
    }
    When there are no more sample points, read the file and divide each frequency number by the total number of points.
    I don't know too many file commands. Before, I had been using simple things like fopen() fclose() fscanf() and fprintf().
    I want to make sure that I don't overwrite useful data while at the same time making sure that my file doesn't have repeats or anything so that I don't have unnecessarily large files to read/write.

    I was hoping to organize the file somewhat like this
    #A
    $B1
    A\tB1\tTheta\tfrequency
    A\tB1\tTheta\tfrequency
    ...
    $B2
    A\tB2\tTheta\tfrequency
    ...
    #A2

    Any Suggestions on what commands I should use, etc. Would be greatly appreciated. Thanks

  2. #2
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,424
    If you want to manipulate file contents as if it were in memory, I would suggest you try to store as much data in memory as possible and write it to file simply to save the data.
    The reason is that while there are functions that can map files into virtual memory, often they have drawbacks and those APIs are also non-portable.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  3. #3
    Registered User
    Join Date
    Jun 2008
    Location
    Northern Va
    Posts
    18
    I'd love to do that, but unfortunately, my test sample size is designed to generate data from 4000 points. Ideally, if my code works, I'll move up to 40,015 or perhaps 41,000 if I need that much. What functions should I use to find different spots in the file?

  4. #4
    Frequently Quite Prolix dwks's Avatar
    Join Date
    Apr 2005
    Location
    Canada
    Posts
    8,046
    You can use ftell() and fseek() (or fgetpos() and fsetpos() in C99) to jump around in a file. If you open a file in, say, "r+" or "w+" mode, you can even overwrite records in the file.

    By treating every 100 bytes or whatever as an element, you can sort of have a large array stored on the disk. You can access any element easily and (relatively) quickly, and you can overwrite elements. But you can't insert new elements into the structure, just as with normal arrays. (Though I think you can add new data to the end of the file.)

    This only works, of course, if each element in the file is the same size.
    dwk

    Seek and ye shall find. quaere et invenies.

    "Simplicity does not precede complexity, but follows it." -- Alan Perlis
    "Testing can only prove the presence of bugs, not their absence." -- Edsger Dijkstra
    "The only real mistake is the one from which we learn nothing." -- John Powell


    Other boards: DaniWeb, TPS
    Unofficial Wiki FAQ: cpwiki.sf.net

    My website: http://dwks.theprogrammingsite.com/
    Projects: codeform, xuni, atlantis, nort, etc.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Newbie homework help
    By fossage in forum C Programming
    Replies: 3
    Last Post: 04-30-2009, 04:27 PM
  2. help with text input
    By Alphawaves in forum C Programming
    Replies: 8
    Last Post: 04-08-2007, 04:54 PM
  3. Encryption program
    By zeiffelz in forum C Programming
    Replies: 1
    Last Post: 06-15-2005, 03:39 AM
  4. Possible circular definition with singleton objects
    By techrolla in forum C++ Programming
    Replies: 3
    Last Post: 12-26-2004, 09:46 AM
  5. C++ XML Class
    By edwardtisdale in forum C++ Programming
    Replies: 0
    Last Post: 12-10-2001, 10:14 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21