Max Size of an Array?

Hello! Im new to this site. Programming has become one of my side hobbies, even though I'm not very good at it :( Anyway, I have created a program that converts a text file that contains a HEX dump into a binary file. The text file looks like so...:

0xDDDDDDDD
0xDDDDDDDD
0xDDDDDDDD
0xDDDDDDDD
0xDDDDDDDD
... and so on

D = predetermined hex data

Heres the beginning of my code...


#include <stdio.h>

int main()
{
FILE *binfile;
FILE *textfile;

long int hexdata[499999];
long int x;
int *data;

... it also opens a text file called dump.txt, and uses fwrite to write each line into a binary file. The problem I am having is that if I increase the size of the array to say 7 digits (99999999) to accomidate for larger text files, the program crashes upon start. It doesnt even attempt to execute. Accordind to this C programming book, a long integer can be at max 10 digits (2,147,483,647) So why cant I declare my array to a similar size? Thanks for any help.

See this other thread. Also, the size of the array has nothing to do with the maximum data value representable.

Also, I really really really hope you're not trying to read, process, and write 500000 bytes at a time. (What do you need so many values for?)

Edited to add: Of course, the other thread talks about std::vector, which is one of those C++ things that isn't in C. If you really needed an array that large, you would go the malloc/free route, but I seriously doubt you need an array that large.

How much system memory do you have? That could be the issue. Because from what I know about C (which admittedly is not a ton, but it is enough to get me by), when it allocates an array, it creates a new variable (sorta) for each slot in the array, so if you don't have enough system memory to handle all the space taken up by an extremely large array, the program would segfault and go kaput. It may just be a hardware limitation.

Then again, I might be completely wrong. The above is just a guess, not a hard fact.

Shane

Well, actually the text file will represent a 32 MB memory dump. Therefore, the text file is actually close to 100 MB, even though it represents 32 MB. Oh, can vectors be used within C and not just C++?

EDIT: To Answer 2nd reply: I have 1.5 GB of memory. I checked my process list and my app is only consuming 3 MB of memory while its running. i have 970 MB + of memory Free.

EDIT 2: I was just thinking of trying to use one variable to store one line of text, write that to disk in binary, clear the variable, and write the next line from the text file into it.... However, the character and string reading functions tend to require an array as an argument... *Scratches Head*

Quote:

---

Originally Posted by Derek8588

Well, actually the text file will represent a 32 MB memory dump. Therefore, the text file is actually close to 100 MB, even though it represents 32 MB. Oh, can vectors be used within C and not just C++?

EDIT: To Answer 2nd reply: I have 1.5 GB of memory. I checked my process list and my app is only consuming 3 MB of memory while its running. i have 970 MB + of memory Free.

---

Sorry I missed the vector thing at first, see the edit.

Anyway, the size (be it 32MB, 100MB, or whatever) is not relevant, because you're not going to process the whole file at a time; if you ever even try to put the whole bloody file in memory, you're just being silly. It looks like you have one (four-byte) hex number per line? Read in the number (you should be able to use the %x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex. No reason to deal with more than one line at a time.

Quote:

---

Originally Posted by Derek8588

EDIT 2: I was just thinking of trying to use one variable to store one line of text, write that to disk in binary, clear the variable, and write the next line from the text file into it.... However, the character and string reading functions tend to require an array as an argument... *Scratches Head*

---

A string is just a bunch of characters all in a row, so is represented by char [] (or char *). However, if your text really is just a lot of hex numbers, you can read them in as such.

Quote:

---

Originally Posted by Derek8588

Oh, can vectors be used within C and not just C++?

---

Well, I suppose that's yes and no. Vector is a class inside the STL for C++ that emulates the behavior for an array.
In C, you can't use classes, so you'd have to do it through functions. Then it would be more like a dynamic array and not a/the vector.
There isn't one handily available either, so you'll have to write one yourself or crawl the web for some code.

Quote:

---

Originally Posted by tabstop

Sorry I missed the vector thing at first, see the edit.

Anyway, the size (be it 32MB, 100MB, or whatever) is not relevant, because you're not going to process the whole file at a time; if you ever even try to put the whole bloody file in memory, you're just being silly. It looks like you have one (four-byte) hex number per line? Read in the number (you should be able to use the &#37;x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex. No reason to deal with more than one line at a time.

---

Yes. It is one four-byte hex number per line. Im using fscanf to read the data in as %X. fgets() and fread() gave me problems so I found it easier for me to use that. fscanf reads each line into an array, then afterward, the array is written back to file in binary. I will try doing what you said... "Read in the number (you should be able to use the %x format specifier; make sure the target is a four-byte unsigned), process, write it out. Lather, rinse, repeat until you're out of hex." I will attempt this a little later today. I have errands to tend to as of now. Thanks for your help. I will post results later this evening.

EDIT: It worked like a charm. Reads value into variable, writes it to bin file, reads new value into same var.... Thanks for all your help!

You can't make an array that large without it being dynamically allocated because it is allocated on the stack, which has a limited (small) capacity.

Just do the sensible thing and make all your variables global unless there is a specific need
for it to be local.

Then again you could follow the advice of the 'experts' and have endless troubles :O)

It's your choice , it's a free world!! Choose carefully!!

Quote:

---

Originally Posted by esbo

Just do the sensible thing and make all your variables global unless there is a specific need
for it to be local.

Then again you could follow the advice of the 'experts' and have endless troubles :O)

It's your choice , it's a free world!! Choose carefully!!

---

Except the program has been "working like a charm" for 29 hours now.

Quote:

---

Originally Posted by tabstop

Except the program has been "working like a charm" for 29 hours now.

---

Has it?
What was the problem then? I can't seem to find that in the thread.
What was the solution?

Quote:

---

Originally Posted by robwhit

You can't make an array that large without it being dynamically allocated because it is allocated on the stack, which has a limited (small) capacity.

---

That does not make sense to me.
Variables allocated on the stack are dynamic by definition.
Did you mean static when you said dynamic?

search the board for malloc.

Quote:

---

Originally Posted by robwhit

search the board for malloc.

---

I don't need to as malloc is not the only form of dynamic allocation and it is not the
form of dynamic allocation being discussed here. Variables local to a function are also
dynamic and that is what is beinig discussed here.
So when you say dynamically allocated variables you need to be clear as to what you mean.

>I don't need to as malloc is not the only form of dynamic allocation and it is not the
form of dynamic allocation being discussed here.
I was discussing it.

>Variables local to a function are also dynamic and that is what is beinig discussed here.
I said dynamically allocated, not dynamic. Nobody says dynamic variables.

>So when you say dynamically allocated variables you need to be clear as to what you mean.
I did, you just don't understand what they mean.

http://en.wikipedia.org/wiki/Dynamic_memory_allocation
http://en.wikipedia.org/wiki/Malloc

[QUOTE=robwhit;704327]>

>Variables local to a function are also dynamic and that is what is beinig discussed here.
I said dynamically allocated, not dynamic. Nobody says dynamic variables.

/QUOTE]

5,160,000 web pages disagree with you.

http://www.google.com/search?q=dynam...B_enGB210GB210

Hence I won't bother commentating on your other errors.

Quote:

---

Originally Posted by esbo

Hence I won't bother commentating on your other errors.

---

Haha! You're funny.

Any dynamic variables in C?

Quote:

---

Originally Posted by esbo

Has it?
What was the problem then? I can't seem to find that in the thread.
What was the solution?

---

Look at the post from 1:29 pm (and its edit).

Quote:

---

Originally Posted by esbo

Just do the sensible thing and make all your variables global unless there is a specific need
for it to be local.

Then again you could follow the advice of the 'experts' and have endless troubles :O)

It's your choice , it's a free world!! Choose carefully!!

---

Don't listen to this moron. This advice is stupid and considered extremely bad practice.
Not to mention this esbo can't tell the difference between dynamically allocated memory and static and doesn't know what the stack is nor the heap, since esbo only uses global variables. Truly a moron.

It's the other way around. Use local variables unless there's a specific need for global variables.
Remember that some compilers won't even optimize global variables, making it even worse.