Thread: How much memory may I use?

  1. #1
    Registered User
    Join Date
    Jan 2004
    Posts
    13

    How much memory may I use?

    I want to implement a mail download function for an own POP3 client (please no shoutings "use a library" as that's not a convenient solution for me) for linux. The mails should be downloaded and then directly reprocessed by another program.

    My problem:

    Usually mails have quite different sizes. Some consist of just 3 or 4 kb, some take up even 20 megs or more. The program I write, however, should be capable of running as a demon on older machines (e.g. 166mhz pentium I with 64 mb memory) as well.

    My questions:

    How should I store the emails fromt he moment they're downloaded till the moment they are processed and not needed anymore?
    Should I use the memory (i.e. the heap) or would it be better to use temporary files?
    If I use temporary files but will need to use string functions on the content of the mail, is mapped memory (mmap()) the solution I want to access the file?
    And how much memory would be suitable to allocate, for example for smaller mails?

    My solutions so far:

    I thought of setting a split value at like 128kbyte - every mail staying below this limit I will process directly using the memory, the other mails I'll first save into temporary files and will then use mmap() to process them part by part.

    I'd be glad if someone more experienced on memory and swap modes issues than me could help!

    Sloede

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    I'd use the file system for everything until you've got it working.

    Then work out whether the performance is adequate with that solution. Remember that even if you're receiving mail 24/7 (which is extremely unlikely), that you'll have plenty of time to process whatever mail you receive.

    > is mapped memory (mmap()) the solution I want to access the file?
    Code:
    char buff[BUFSIZ];
    FILE *fp = fopen( "mail.txt", "r" );
    while ( fgets( buff, BUFSIZ, fp ) != NULL ) {
        if ( strstr( buff, "magic" ) != NULL ) {
            /* do stuff */
        }
    }
    Now what's so difficult about that?

    If at the end you're still stuck, then you can consider some of your other ideas in a measured way (you can time be before and after), and decide whether such a change is really worth the coding effort.

    Are you trying to implement a filter?
    ls -l | cat | lpr
    Here, cat is a filter (which does nothing) but it does pass its input to its output

    An email filter could look something like this...
    pop3daemon | myfilter | myemailreader
    Because if you are, then there could be no need to store anything at all, and you just process all the data on the fly as it were.
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    Registered User
    Join Date
    Jan 2004
    Posts
    13
    First, thanks for answering my request.

    The reason why I thought to use the memory or the mmap functions was that they seem to be easier and more efficient to use. But as you suggested, I'll find out about this by trying

    I think a filter wouldn't describe it correctly. I want to store the mail data into a sql database, but before this happens the important information should be extracted from the message (like headers and different body parts). I don't know if this can be done on-the-fly so I think it'd be better to store them temporarily. That would also give me the possibility to continue a broken download.

    Well, I'll try to do it with temporary files and if this doesn't work I'll search for other solutions.

    I just needed some assurance that I'm not completely out of my mind with my ideas.

    Sloede

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 4
    Last Post: 01-13-2008, 02:14 AM
  2. Question regarding Memory Leak
    By clegs in forum C++ Programming
    Replies: 29
    Last Post: 12-07-2007, 01:57 AM
  3. Memory problem with Borland C 3.1
    By AZ1699 in forum C Programming
    Replies: 16
    Last Post: 11-16-2007, 11:22 AM
  4. Shared Memory - shmget questions
    By hendler in forum C Programming
    Replies: 1
    Last Post: 11-29-2005, 02:15 AM
  5. What's the best memory (RAM) type?
    By Unregistered in forum A Brief History of Cprogramming.com
    Replies: 17
    Last Post: 12-15-2001, 12:37 AM