Reading a File - FASTEST WAY POSSIBLE??

[Mastadex]

Hi, Im still a bit new to programming in C++ and what imn trying to achieve is to read a file into memory as fast as possible. Here is what I do:

...


	char * sMessage;
	register long lSize;
	FILE * fp;

	if (argc)
	{
		fp = fopen(argv[1], "rb");

		// THIS FINDS THE SIZE OF THE FILE
		fseek(fp , 0 , SEEK_END);
		lSize = ftell (fp);
		rewind (fp);

		printf("SIZE: %d\n", lSize);

		// ALLOCATES NEW MEMORY FOR THE ARRAY 
		sMessage = new char[lSize];

                // THIS IS THE PART I WANT TO SPEED UP
                // I READ THE CONTENTS FROM A BINARY FILE
                // ONE BYTE AT A TIME
		for (i = 0; i < lSize; i++)
		{
			sMessage[i] = getc(fp);
		}


		printf("READ: %d\n", lSize);


}

return 0;

Im trying to speed up the reading process so it can read ~200Mb files in onle a couple of seconds. Does anyone have any suggestions????

[Thantos]

Since you know the size of the file you should be able to do this:

fread(sMessage, lSize, 1, fp);

I don't know of a good c++ method sorry. You should also check the return value of fread for errors.

[Mastadex]

I tried using fread and when it comes to files that are 200Mb big, its a bit sluggish, therefore I just use a loop and read it one byte at a time.

Sine the buffer on most machines is quite small, around 512Kb, a 200Mb file is not going to fit in there, so the machine has to read from the file many times before my program can play around with it.

[Thantos]

Well if you are on a *nix machine you might try read() (can't remember if its avilable on windows or not). And if you can only read 512k at a time just make several calls to fread(). Making 4 or 5 calls to fread() will be faster then making 2 million calls to fgetc(). Also if you need to load an entire 200mb file into memory you really need to rethink your solution.

[Mastadex]

Well what do you guys suggest, all my program does right now is create a CRC32 for large files. If you have any other methods of approaching this then by all means...

The actual algorithm takes little time to process on my machine, the file reading is alot more time consuming.

[-=SoKrA=-]

Try this and see what you get:

//assumming we're using namespace std
ifsteam file("file.txt", ios::in);

//go to the end of the file
file.seekg(0, ios::end);
//and get the cursor position
//which gives us the size
int length = file.tellg():

//we use buf as our buffer. We add
//1 to the length for the null-terminator
char* buf = new char[length + 1];
//go to the beginning of the file
file.tellg(0, ios::beg);

//now read everything in one go
file.read(buf, length);
buf[length +1] = '\0'; //don't forget this
                      //or we might run into some memory
                     //which is a definite no-no

//and close the file
file.close();

I recently used it in a project, although it compiles I'm not sure how efficient it is, because I've not been able to test it.

[Salem]

> Im trying to speed up the reading process so it can read ~200Mb files in onle a couple of seconds.
Unless you've got the latest kit, I doubt you'll get anywhere near that.

> Sine the buffer on most machines is quite small, around 512Kb
Says who?
And which buffer?
You can't know in general how many buffers there are, let alone how big they are.

What are you intending to do with the data once you're read it?
If its something simple like

    for ( i = 0 ; i < len ; i++ ) {
        sum += buff[i];
    }

Then its a waste of time and resource to allocate memory for the whole file.

A couple of ideas for you

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

#define BSIZE   BUFSIZ*100
void calcs ( unsigned char *buff, size_t len ) {
    int sum = 0;
    size_t  i;
    for ( i = 0 ; i < len ; i++ ) {
        sum += buff[i];
    }
    printf( "Sum=%d\n", sum );
}

void exp1 ( char *filename ) {
    FILE *fp;
    fp = fopen( filename, "rb" );
    if ( fp ) {
        long len;
        unsigned char *mem;
        fseek( fp, 0, SEEK_END );
        len = ftell(fp);
        fseek( fp, 0, SEEK_SET );
        printf( "Len=%ld\n", len );
        mem = malloc( len );
        if ( mem ) {
            int n;
            unsigned char *p = mem;
            while ( (n=fread(p,1,BSIZE,fp)) > 0 ) {
                p += n;
            }
            calcs( mem, len );
            free(mem);
        }
        fclose(fp);
    }
}

void exp2 ( char *filename ) {
    int fd;
    fd = open( filename, O_RDONLY );
    if ( fd != -1 ) {
        off_t len;
        unsigned char *mem;
        len = lseek( fd, 0, SEEK_END );
        mem = mmap( 0, len, PROT_READ, MAP_SHARED, fd, 0 );     /* map the whole file */
        if ( mem != NULL ) {
            calcs( mem, len );
            munmap( mem, len );
        } else {
            perror("oops");
        }
    }
}

int main ( int argc, char *argv[] ) {
    printf( "%d\n", clock() );
    exp1( argv[1] );
    printf( "%d\n", clock() );
    exp2( argv[1] );
    printf( "%d\n", clock() );
    return 0;
}

For me at least, the mmap() solution is twice as quick as fread()ing large blocks.

EDIT
So you are just touching each byte once. It's a waste of time reading the whole file into a huge buffer. Use a small buffer about BUFSIZ in size and keep freading into that and performing next block of your CRC

[Mastadex]

I will try your Idea, the calculation im doing the the standard CRC32 algorithm.

for (i = 0; i < lSize; i++){
	iCRC = ((iCRC >> 8) & 0xFFFFFFFF) ^ table[(iCRC ^ sMessage[i]) & 0xFF];
}

This part is extreemely fast on my machine. Ill try the mapping feature instead.

[Sebastiani]

>
> Reading a File - FASTEST WAY POSSIBLE??
>

good old C (memory requirement: one byte) -

unsigned int copyfile(const char * to, const char * from)
{
 int byte;
 unsigned int size = 0;
 FILE * in = fopen(from, "rb");
 if(!in) {
  return 0;
  }
 FILE * out = fopen(to, "wb");
 if(out) {
  chsize(fileno(out), 0);
  while(EOF != (byte = fgetc(in))) {
    fputc(byte, out);
    ++size;  
   }
  fclose(out);
  }  
 fclose(in);  
 return size;
}

[swoopy]

I think Mastadex said: FASTEST WAY POSSIBLE??

[Salem]

> This part is extreemely fast on my machine. Ill try the mapping feature instead.
As it should be - but irrelevant to the overall time it takes to read a file.

Randomly picked hard disk spec
Would take over a second to read your file in even the most perfect environment crafted to show off your work in the best possible light. A hard disk can never keep a processor even remotely occupied for any sustained length of time.

Throw an unoptimised, off the self, multitasking operating system and file system on top of it (can you say fragmented files) and you start to see the problems.

For example, a 10ms seek time equates to about for ( i = 0 ; i < 20E6 ; i++ ) for your 2GHZ processor. That's like 10% of the work you need to perform and you haven't even opened the file yet.

Here's another 10% of your work gone - the OS decides to schedule some other activity for 10ms, whilst it waits for the disk to catch up.

[Mastadex]

Unfortunately what im coing is only windows specific, so it wont be ported to Lunix/Unix. Is there a windows equivelent of the mmap function??

[Salem]

Yes

[Mastadex]

And what would it be??

[Thantos]

All answers will be show to you once you CLICK THE LINK PROVIDED!