Thread: dynamically loading data?

  1. #1
    Registered User
    Join Date
    Jun 2008
    Posts
    106

    dynamically loading data?

    i have a function that loads datasets, which is quite complex, but you can use it as such:

    Code:
    loadset setname ("filename.h5");
    How can I make a wrapper function that I can use to dynamically load these sets?
    For example, It is up to the user to determine how many sets must be loaded simultaneously. So I would need to dynamically comeup with 'setname', such as, if the user enters "5", then this:
    Code:
    for (int i = 0; i < chunk; i++){
    loadset string("setname" + i) ("filename1.h5");
    where string("setname" + i) is the variable's name. obviously this code won't work.

  2. #2
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    perhaps putting it in a struct or class, and then dynamically creating a class or structure, how would I do such a thing?

  3. #3
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    Quote Originally Posted by elninio View Post
    Code:
    loadset setname ("filename.h5");
    Is this a declaration or code?

  4. #4
    Registered User
    Join Date
    Jan 2005
    Posts
    7,366
    Are you trying to create the filename, as in filename.h1, filename.h2, filename.h3? If so, use a stringstream to create the filename.

    Otherwise, how are the sets broken up? All in the same file?

  5. #5
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    loadset is a class. setname is an object of loadset. it loads a dataset from the file "filename.h5" into memory. i'm trying to load filenames dynamically into memory depending on how many a user specifies. each object of loadset should correspond to a unique file.

    So, setname1 corresponds to file1.h5 , setname2 corresponds to file2.h5 etc..

  6. #6
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    I'm thinking of using a recursive linked list. But the problem will be specifying loading a different filename for each node

  7. #7
    Registered User
    Join Date
    Jan 2005
    Posts
    7,366
    Ok, use a stringstream to create the filename.

    Use an array (or probably a vector) to store each loadset. So instead of setname1, it would be loadset_list[0].

    Depending on how expensive it is to copy your loadset object, you might consider using a ptr_vector from boost or a vector of shared_ptrs (or a carefully managed vector of raw pointers).

  8. #8
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    Quote Originally Posted by Daved View Post
    Ok, use a stringstream to create the filename.

    Use an array (or probably a vector) to store each loadset. So instead of setname1, it would be loadset_list[0].

    Depending on how expensive it is to copy your loadset object, you might consider using a ptr_vector from boost or a vector of shared_ptrs (or a carefully managed vector of raw pointers).

    I didn't know that was possible - I have no idea how to implement this, I don't even have a guess!

    these files are datasets which contain subdirectories and matrices, do you mean to store a pointer to each loadset? I cringe just writing this:

    int array[10];

    array[1]fail

    ok i give up.

    I have no idea how it is possible to store an entire dataset in an array, and I have no idea how to declare a pointer to it (doubting if it is even possible). Sorry :/

  9. #9
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    Quote Originally Posted by Daved View Post
    Ok, use a stringstream to create the filename.

    Use an array (or probably a vector) to store each loadset. So instead of setname1, it would be loadset_list[0].

    Depending on how expensive it is to copy your loadset object, you might consider using a ptr_vector from boost or a vector of shared_ptrs (or a carefully managed vector of raw pointers).
    loadset has to be declared dynamically, thus it needs to have an indentifyer that is also declared dynamically.

  10. #10
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    Nope, it doesn't work:

    Code:
    'array' has a previous declaration as 'int array [50]'
    for
    Code:
    int array [50];
    dataset_loader array[0]("pic3.h5");
    dataset_loader array[0]("pic2.h5");
    Last edited by elninio; 07-27-2008 at 09:58 PM.

  11. #11
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    can someone show me how i could make a datastructure that would dynamically load these files

    struct node(
    dataset_loader set("file1.h5");
    node *next;
    )

    I could make a loop that generates through this, but how would I make it so that "file1.h5" would be changed to file2 file3 file4 etc.. for each iteration of the loop?

  12. #12
    Registered User
    Join Date
    Jan 2005
    Posts
    7,366
    Don't use your own linked list, use a vector or array. You have the right idea, except your array should be an array of dataset_loader objects, not an array of int.

    Is there a way to create a dataset_loader object without specifying a filename? If you create an array, you cannot specify the filename in the constructor because you won't know what the filenames are until the program is running.

    You should really use vector for this. Are you familiar with vectors? Now would be a good time to do a little research on them because they will help you do exactly what you want.

  13. #13
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    Is there a way to create a dataset_loader object without specifying a filename?

    no. there isn't you have to create objects of that class by specifying a filename. the reason for that is because you would the run the following function

    dataset_buf filename = load_data_set("/root/folder/folder2");

    Basically it returns a reference to a dataset buffer. The you can start accessing its members such as filename(x,y,z);

  14. #14
    Registered User
    Join Date
    Jan 2005
    Posts
    7,366
    You keep changing the names you use and it is confusing me. What do the following mean:

    loadset
    dataset_loader
    dataset_buf
    load_data_set

    ?


    Can you copy the loadset?
    Code:
    loadset setname ("file1.h5");
    loadset copy = setname;

  15. #15
    Registered User
    Join Date
    Jun 2008
    Posts
    106
    sorry,

    these are my commands:

    dataset_loader someset("filename");
    dataset_buf somebuf = someset.load_data_set("/folder/folder2");
    somebuf(x,y,z);

    dataset_loader and dataset_buf are functions in the dataset_loader class

    Can you copy the loadset?
    no you can't do this. datasets are .h5 files, these are complex hierarchies made of folders and matrices.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. simultaneously waiting for data on FIFO and UDP using select call
    By yogesh3073 in forum Networking/Device Communication
    Replies: 2
    Last Post: 01-05-2007, 09:53 AM
  2. Where's the EPIPE signal?
    By marc.andrysco in forum Networking/Device Communication
    Replies: 0
    Last Post: 12-23-2006, 08:04 PM
  3. Dynamic data members?
    By confusalot in forum C++ Programming
    Replies: 4
    Last Post: 02-27-2005, 11:15 AM
  4. C Programming Question
    By TK in forum A Brief History of Cprogramming.com
    Replies: 13
    Last Post: 07-04-2002, 07:11 PM
  5. Loading a struct with data from a file
    By TankCDR in forum C Programming
    Replies: 1
    Last Post: 10-28-2001, 08:58 AM