How much memory may I use?

I want to implement a mail download function for an own POP3 client (please no shoutings "use a library" as that's not a convenient solution for me) for linux. The mails should be downloaded and then directly reprocessed by another program.

My problem:

Usually mails have quite different sizes. Some consist of just 3 or 4 kb, some take up even 20 megs or more. The program I write, however, should be capable of running as a demon on older machines (e.g. 166mhz pentium I with 64 mb memory) as well.

My questions:

How should I store the emails fromt he moment they're downloaded till the moment they are processed and not needed anymore?
Should I use the memory (i.e. the heap) or would it be better to use temporary files?
If I use temporary files but will need to use string functions on the content of the mail, is mapped memory (mmap()) the solution I want to access the file?
And how much memory would be suitable to allocate, for example for smaller mails?

My solutions so far:

I thought of setting a split value at like 128kbyte - every mail staying below this limit I will process directly using the memory, the other mails I'll first save into temporary files and will then use mmap() to process them part by part.

I'd be glad if someone more experienced on memory and swap modes issues than me could help!

Sloede

I'd use the file system for everything until you've got it working.

Then work out whether the performance is adequate with that solution. Remember that even if you're receiving mail 24/7 (which is extremely unlikely), that you'll have plenty of time to process whatever mail you receive.

> is mapped memory (mmap()) the solution I want to access the file?

Code:

---

char buff[BUFSIZ];
FILE *fp = fopen( "mail.txt", "r" );
while ( fgets( buff, BUFSIZ, fp ) != NULL ) {
    if ( strstr( buff, "magic" ) != NULL ) {
        /* do stuff */
    }
}

---

Now what's so difficult about that?

If at the end you're still stuck, then you can consider some of your other ideas in a measured way (you can time be before and after), and decide whether such a change is really worth the coding effort.

Are you trying to implement a filter?
ls -l | cat | lpr
Here, cat is a filter (which does nothing) but it does pass its input to its output

An email filter could look something like this...
pop3daemon | myfilter | myemailreader
Because if you are, then there could be no need to store anything at all, and you just process all the data on the fly as it were.

First, thanks for answering my request.

The reason why I thought to use the memory or the mmap functions was that they seem to be easier and more efficient to use. But as you suggested, I'll find out about this by trying :)

I think a filter wouldn't describe it correctly. I want to store the mail data into a sql database, but before this happens the important information should be extracted from the message (like headers and different body parts). I don't know if this can be done on-the-fly so I think it'd be better to store them temporarily. That would also give me the possibility to continue a broken download.

Well, I'll try to do it with temporary files and if this doesn't work I'll search for other solutions.

I just needed some assurance that I'm not completely out of my mind with my ideas.

Sloede