Max Size of an Array?

This is a discussion on Max Size of an Array? within the C Programming forums, part of the General Programming Boards category; Hello! Im new to this site. Programming has become one of my side hobbies, even though I'm not very good ...

  1. #1
    Registered User
    Join Date
    Jan 2008
    Posts
    3

    Max Size of an Array?

    Hello! Im new to this site. Programming has become one of my side hobbies, even though I'm not very good at it Anyway, I have created a program that converts a text file that contains a HEX dump into a binary file. The text file looks like so...:

    0xDDDDDDDD
    0xDDDDDDDD
    0xDDDDDDDD
    0xDDDDDDDD
    0xDDDDDDDD
    ... and so on

    D = predetermined hex data

    Heres the beginning of my code...


    #include <stdio.h>

    int main()
    {
    FILE *binfile;
    FILE *textfile;

    long int hexdata[499999];
    long int x;
    int *data;

    ... it also opens a text file called dump.txt, and uses fwrite to write each line into a binary file. The problem I am having is that if I increase the size of the array to say 7 digits (99999999) to accomidate for larger text files, the program crashes upon start. It doesnt even attempt to execute. Accordind to this C programming book, a long integer can be at max 10 digits (2,147,483,647) So why cant I declare my array to a similar size? Thanks for any help.

  2. #2
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    See this other thread. Also, the size of the array has nothing to do with the maximum data value representable.

    Also, I really really really hope you're not trying to read, process, and write 500000 bytes at a time. (What do you need so many values for?)

    Edited to add: Of course, the other thread talks about std::vector, which is one of those C++ things that isn't in C. If you really needed an array that large, you would go the malloc/free route, but I seriously doubt you need an array that large.
    Last edited by tabstop; 01-05-2008 at 10:59 PM. Reason: That's not the preview button :(

  3. #3
    Registered User
    Join Date
    Oct 2005
    Posts
    23
    How much system memory do you have? That could be the issue. Because from what I know about C (which admittedly is not a ton, but it is enough to get me by), when it allocates an array, it creates a new variable (sorta) for each slot in the array, so if you don't have enough system memory to handle all the space taken up by an extremely large array, the program would segfault and go kaput. It may just be a hardware limitation.

    Then again, I might be completely wrong. The above is just a guess, not a hard fact.

    Shane

  4. #4
    Registered User
    Join Date
    Jan 2008
    Posts
    3
    Well, actually the text file will represent a 32 MB memory dump. Therefore, the text file is actually close to 100 MB, even though it represents 32 MB. Oh, can vectors be used within C and not just C++?

    EDIT: To Answer 2nd reply: I have 1.5 GB of memory. I checked my process list and my app is only consuming 3 MB of memory while its running. i have 970 MB + of memory Free.

    EDIT 2: I was just thinking of trying to use one variable to store one line of text, write that to disk in binary, clear the variable, and write the next line from the text file into it.... However, the character and string reading functions tend to require an array as an argument... *Scratches Head*
    Last edited by Derek8588; 01-05-2008 at 11:03 PM.

  5. #5
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    Quote Originally Posted by Derek8588 View Post
    Well, actually the text file will represent a 32 MB memory dump. Therefore, the text file is actually close to 100 MB, even though it represents 32 MB. Oh, can vectors be used within C and not just C++?

    EDIT: To Answer 2nd reply: I have 1.5 GB of memory. I checked my process list and my app is only consuming 3 MB of memory while its running. i have 970 MB + of memory Free.
    Sorry I missed the vector thing at first, see the edit.

    Anyway, the size (be it 32MB, 100MB, or whatever) is not relevant, because you're not going to process the whole file at a time; if you ever even try to put the whole bloody file in memory, you're just being silly. It looks like you have one (four-byte) hex number per line? Read in the number (you should be able to use the %x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex. No reason to deal with more than one line at a time.

  6. #6
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    Quote Originally Posted by Derek8588 View Post
    EDIT 2: I was just thinking of trying to use one variable to store one line of text, write that to disk in binary, clear the variable, and write the next line from the text file into it.... However, the character and string reading functions tend to require an array as an argument... *Scratches Head*
    A string is just a bunch of characters all in a row, so is represented by char [] (or char *). However, if your text really is just a lot of hex numbers, you can read them in as such.

  7. #7
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Posts
    23,014
    Quote Originally Posted by Derek8588 View Post
    Oh, can vectors be used within C and not just C++?
    Well, I suppose that's yes and no. Vector is a class inside the STL for C++ that emulates the behavior for an array.
    In C, you can't use classes, so you'd have to do it through functions. Then it would be more like a dynamic array and not a/the vector.
    There isn't one handily available either, so you'll have to write one yourself or crawl the web for some code.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  8. #8
    Registered User
    Join Date
    Jan 2008
    Posts
    3
    Quote Originally Posted by tabstop View Post
    Sorry I missed the vector thing at first, see the edit.

    Anyway, the size (be it 32MB, 100MB, or whatever) is not relevant, because you're not going to process the whole file at a time; if you ever even try to put the whole bloody file in memory, you're just being silly. It looks like you have one (four-byte) hex number per line? Read in the number (you should be able to use the &#37;x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex. No reason to deal with more than one line at a time.
    Yes. It is one four-byte hex number per line. Im using fscanf to read the data in as %X. fgets() and fread() gave me problems so I found it easier for me to use that. fscanf reads each line into an array, then afterward, the array is written back to file in binary. I will try doing what you said... "Read in the number (you should be able to use the %x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex." I will attempt this a little later today. I have errands to tend to as of now. Thanks for your help. I will post results later this evening.

    EDIT: It worked like a charm. Reads value into variable, writes it to bin file, reads new value into same var.... Thanks for all your help!
    Last edited by Derek8588; 01-06-2008 at 05:13 PM.

  9. #9
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    You can't make an array that large without it being dynamically allocated because it is allocated on the stack, which has a limited (small) capacity.

  10. #10
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Just do the sensible thing and make all your variables global unless there is a specific need
    for it to be local.

    Then again you could follow the advice of the 'experts' and have endless troubles :O)

    It's your choice , it's a free world!! Choose carefully!!

  11. #11
    and the Hat of Guessing tabstop's Avatar
    Join Date
    Nov 2007
    Posts
    14,185
    Quote Originally Posted by esbo View Post
    Just do the sensible thing and make all your variables global unless there is a specific need
    for it to be local.

    Then again you could follow the advice of the 'experts' and have endless troubles :O)

    It's your choice , it's a free world!! Choose carefully!!
    Except the program has been "working like a charm" for 29 hours now.

  12. #12
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by tabstop View Post
    Except the program has been "working like a charm" for 29 hours now.

    Has it?
    What was the problem then? I can't seem to find that in the thread.
    What was the solution?

  13. #13
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by robwhit View Post
    You can't make an array that large without it being dynamically allocated because it is allocated on the stack, which has a limited (small) capacity.
    That does not make sense to me.
    Variables allocated on the stack are dynamic by definition.
    Did you mean static when you said dynamic?

  14. #14
    Registered User
    Join Date
    Oct 2001
    Posts
    2,129
    search the board for malloc.

  15. #15
    Fountain of knowledge.
    Join Date
    May 2006
    Posts
    794
    Quote Originally Posted by robwhit View Post
    search the board for malloc.

    I don't need to as malloc is not the only form of dynamic allocation and it is not the
    form of dynamic allocation being discussed here. Variables local to a function are also
    dynamic and that is what is beinig discussed here.
    So when you say dynamically allocated variables you need to be clear as to what you mean.

Page 1 of 2 12 LastLast
Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Max array size question
    By serge in forum C++ Programming
    Replies: 21
    Last Post: 06-16-2009, 05:13 AM
  2. Replies: 16
    Last Post: 05-29-2009, 08:25 PM
  3. How do you use variable to initialize array size?
    By yougene in forum C Programming
    Replies: 11
    Last Post: 09-04-2007, 03:50 PM
  4. Class Template Trouble
    By pliang in forum C++ Programming
    Replies: 4
    Last Post: 04-21-2005, 05:15 AM
  5. templates, unresolved external error
    By silk.odyssey in forum C++ Programming
    Replies: 9
    Last Post: 06-09-2004, 05:39 PM

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21