how to store file of big size

This is a discussion on how to store file of big size within the C Programming forums, part of the General Programming Boards category; file1: Code: 12E3F 1E34F 34EF5 ...... there are 20000000000000 Rows in file1. Can I store the file into 20000000000000*5 array ...

  1. #1
    Registered User
    Join Date
    Aug 2009
    Posts
    168

    how to store file of big size

    file1:
    Code:
    12E3F
    1E34F
    34EF5
    ......
    there are 20000000000000 Rows in file1.
    Can I store the file into 20000000000000*5 array ?
    if not , how to implement it ?

  2. #2
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    By 'store', I assume you mean 'load into memory all at the same time', which leads me to ask: Why?

    To answer your question: No, you can't make a true array that size. You'll run out of stack space. Your best shot is to dynamically allocate it with something like malloc. But again, you're probably out of luck unless you're on a 64 bit OS. 100000000000000 is too big to fit in a 32bit data type.


    Quzah.
    Hope is the first step on the road to disappointment.

  3. #3
    Registered User
    Join Date
    Aug 2009
    Posts
    168
    Quote Originally Posted by quzah View Post
    By 'store', I assume you mean 'load into memory all at the same time', which leads me to ask: Why?

    To answer your question: No, you can't make a true array that size. You'll run out of stack space. Your best shot is to dynamically allocate it with something like malloc. But again, you're probably out of luck unless you're on a 64 bit OS. 100000000000000 is too big to fit in a 32bit data type.


    Quzah.
    Code:
    #include "stdlib.h"
    #include "stdio.h"
    #include "string.h"
    
    int main(int argc,char *argv[])
    {
        char **p = (char **) malloc ( 50*sizeof(char));
        FILE *fr = fopen("test.data","r");
        char a;
        int i = 0;
        int m = 0;
        int n = 0;
        while ( fscanf(fr, "%c", &a ) == 1 )
        {
                //char line[10];
                //fgets(line,10,fr);
                
                
                if ( a == '\n' )
                {
                     m = 0;
                     n++;
                } 
                else
                {
                    *(*(p+n)+m) = a;
                    m++;
                }
                
                
                
        }
                
                
        return 0;
    }
    this is my test program . why error ?

  4. #4
    C++まいる!Cをこわせ! Elysia's Avatar
    Join Date
    Oct 2007
    Posts
    22,614
    I have a feeling we've been over this before... take a look at your other threads again.
    And no, you can't store a file that is 93 TB in memory. Not that I would ever see such a big file.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  5. #5
    ATH0 quzah's Avatar
    Join Date
    Oct 2001
    Posts
    14,826
    Why are you using a pointer to a pointer? If you want to pretend it's a 2D array, then you need two steps:
    Code:
    char **p;
        p = malloc( sizeof( char * ) * numberofrows );
        for( x = 0; x < numberofrows; x++ )
            p[ x ] = malloc( numberofcolumnsperrow );
    You are using fscanf to read a single character. That's not its best suit. Why not just use fgetc if you only want one character?

    But to answer your question: it's crashing because you're not allocating memory right. See point one again.

    Edit: And to go along with what Elysia said, stop making new threads for the same exact topic. You have like four now.


    Quzah.
    Hope is the first step on the road to disappointment.

  6. #6
    Registered User
    Join Date
    Aug 2009
    Posts
    168
    Quote Originally Posted by quzah View Post
    Why are you using a pointer to a pointer? If you want to pretend it's a 2D array, then you need two steps:
    Code:
    char **p;
        p = malloc( sizeof( char * ) * numberofrows );
        for( x = 0; x < numberofrows; x++ )
            p[ x ] = malloc( numberofcolumnsperrow );
    You are using fscanf to read a single character. That's not its best suit. Why not just use fgetc if you only want one character?

    But to answer your question: it's crashing because you're not allocating memory right. See point one again.

    Edit: And to go along with what Elysia said, stop making new threads for the same exact topic. You have like four now.


    Quzah.
    Your answer is appreciated, Thanks

  7. #7
    Registered User
    Join Date
    Aug 2009
    Posts
    168
    Quote Originally Posted by quzah View Post
    Why are you using a pointer to a pointer? If you want to pretend it's a 2D array, then you need two steps:
    Code:
    char **p;
        p = malloc( sizeof( char * ) * numberofrows );
        for( x = 0; x < numberofrows; x++ )
            p[ x ] = malloc( numberofcolumnsperrow );
    You are using fscanf to read a single character. That's not its best suit. Why not just use fgetc if you only want one character?

    But to answer your question: it's crashing because you're not allocating memory right. See point one again.

    Edit: And to go along with what Elysia said, stop making new threads for the same exact topic. You have like four now.


    Quzah.
    your suggestion is appreciated

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. To find the memory leaks without using any tools
    By asadullah in forum C Programming
    Replies: 2
    Last Post: 05-12-2008, 07:54 AM
  2. gcc link external library
    By spank in forum C Programming
    Replies: 6
    Last Post: 08-08-2007, 03:44 PM
  3. C++ std routines
    By siavoshkc in forum C++ Programming
    Replies: 33
    Last Post: 07-28-2006, 12:13 AM
  4. Unknown Memory Leak in Init() Function
    By CodeHacker in forum Windows Programming
    Replies: 3
    Last Post: 07-09-2004, 09:54 AM
  5. simulate Grep command in Unix using C
    By laxmi in forum C Programming
    Replies: 6
    Last Post: 05-10-2002, 04:10 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21