Thread: Communication with txt files (continuing)

There are generally two choices when dealing with reading and processing file data. The first one is to read a file by pieces. It's advantage is low memory consumption.

Code:

bool LoadListbox(char* lpFileName,HWND hWndReceiver){
    char *lpData=NULL,*lpLine=NULL;
    bool bRtn=true;
    DWORD dwRead=0,dwCount=0;
    HANDLE hFile=CreateFile(lpFileName,GENERIC_READ,7,0,OPEN_EXISTING,0,0);
    if(hFile==INVALID_HANDLE_VALUE){ return false; }
    lpData=new char[512];
    lpLine=new char[512];
    if(!ReadFile(hFile,lpData,512,&dwRead,0)){ bRtn=false; }
    while(dwRead&&bRtn){
        for(int i=0;i<dwRead;i++){
            if(lpData[i]=='\n'){
                lpLine[dwCount]='\0';
                SendDlgItemMessage(hWndReceiver,IDC_LISTEN,LB_ADDSTRING,0,(LPARAM)lpLine);
                dwCount=0;
            }
            else if(lpData[i]!='\r'&&dwCount<511){
                lpLine[dwCount]=lpData[i];
                dwCount++;
            }
        }
        if(!ReadFile(hFile,lpData,512,&dwRead,0)){ bRtn=false; }
    }
    if(dwCount){
        lpLine[dwCount]='\0';
        SendDlgItemMessage(hWndReceiver,IDC_LISTEN,LB_ADDSTRING,0,(LPARAM)lpLine);
    }
    delete[] lpLine;
    delete[] lpData;
    CloseHandle(hFile);
    return bRtn;
}

And your way:

Code:

bool LoadListboxBig(char* lpFileName,HWND hWndReceiver){
    DWORD dwRead=0,dwCount=0,dwSize=0;
    char *lpData=NULL,*lpLine=NULL;
    HANDLE hFile=CreateFile(lpFileName,GENERIC_READ,7,0,OPEN_EXISTING,0,0);
    if(hFile==INVALID_HANDLE_VALUE){ return false; }
    dwSize=GetFileSize(hFile,0);
    if(dwSize){
        lpData=(char*)HeapAlloc(GetProcessHeap(),0,dwSize);
        if(!lpData){
            CloseHandle(hFile);
            return false;
        }
        if(!ReadFile(hFile,lpData,dwSize,&dwRead,0)){
            CloseHandle(hFile);
            HeapFree(GetProcessHeap(),0,lpData);
            return false;
        }
        lpLine=new char[512];
        for(int i=0;i<dwRead;i++){
            if(lpData[i]=='\n'){
                lpLine[dwCount]='\0';
                SendDlgItemMessage(hWndReceiver,IDC_LISTEN,LB_ADDSTRING,0,(LPARAM)lpLine);
                dwCount=0;
            }
            else if(lpData[i]!='\r'&&dwCount<511){
                lpLine[dwCount]=lpData[i];
                dwCount++;
            }
        }
        if(dwCount){
            lpLine[dwCount]='\0';
            SendDlgItemMessage(hWndReceiver,IDC_LISTEN,LB_ADDSTRING,0,(LPARAM)lpLine);
        }
        delete[] lpLine;
    }
    HeapFree(GetProcessHeap(),0,lpData);
    CloseHandle(hFile);
    return true;
}

I haven't tested them so don't mind any little mistakes - you get the idea.

Well, if I've got a choice between Win32 API and CRT I would always use Win32 API functions unless advised otherwise by Microsoft documentation. CRT functions adventually call the Win32 API functions but to that point they manage to bloat the executable and performance. They are only meant for portability, not for high-performance Windows-based applications.

I prefer to read the whole file in to a dynamically alloc'ed buffer with one read. Then close the file and process the data.

This has the advantage of speed (as each read is very slow) but then I deal with files with 10,000s of records.

Also the file is open for the shortest time possible, reducing shared access confilicts.

Low memory consumption is not usually a factor in systems with Gb to spare.

EDIT:
I see that is what you had before.

You just need to look for the end of line characters in the file buffer once loaded. ie a way of determining the start and end of a line.
Then send that portion of the buffer to the list (after adding a terminator '\0')

Well, if I've got a choice between Win32 API and CRT I would always use Win32 API functions unless advised otherwise by Microsoft documentation. CRT functions adventually call the Win32 API functions but to that point they manage to bloat the executable and performance.

I do just the opposite. If I've got a choice, I go with fopen() and friends. You say they "bloat" the executable, and performance. On my system, a minimal example executable with fopen() uses <17KB, before any optimizations or stripping. Not enough bloat, if any, to worry about. (Other libraries add half a MB or more... *cough*wxWidgets*cough*)
As for performance, that's a harder one, so I wrote a program to read a file, and time it: (I don't actually know how much faster Win32 would be, so, let's find out.)

Code:

CRT_Read(): 730036224 bytes read.
CRT  : 981156 milliseconds.
Win32_Read(): 730036224 bytes read.
Win32: 962075 milliseconds.

Indeed, Win32 is faster. (I should hope so.) However, it only faired 2&#37; better here. This test was conducted on an AMD (450 MHz) Win98 box. Advances in processor power should narrow the margin between fopen() and the Win32 equivalents even more. That said, I find the loss in portability from not using fopen() to outweigh any speed or size benefits. (The test program itself was 6KB, and used both methods.)
Now, if I was working on a project that did insane amounts of file I/O and needed every last performance enhancement, I might think about it.
Furthermore, I'd expect whatever storage media you're reading off of to be more of a bottleneck than differences CRT/Win32.
(Edit: The moderators on the Microsoft forums seem to think fopen()...)

The test I conducted was on the above box, on a ~700MB file. (The Knoppix CD ISO, actually.) Hopefully a large file will reduce/eliminate any caching of the file. (If Win9x does so.) I might repeat a later test on XP. Need a big file though...
The program I used to test with is attached, if it interests anyone.

I guess 97&#37; percent goes to hard drive reading, 1% goes to Win32 API function code itself and 2% to the CRT.
In the contrary, you need to test on little files.

Test trying to open a file that doesn't exist 100 000 times.

Test trying to open a file that doesn't exist 100 000 times.

Alright. Win98 faired pathetically. I actually only looped it 1 000 times. (Win98 / 450 MHz)

Code:

Trying to open file "c:\nonexists" 1000 times.
CRT  : 11041 milliseconds.
Win32: 7015 milliseconds.

I repeated the test on WinXP, (1.8 GHz):

Code:

Trying to open file "c:\nonexists" 100000 times.
CRT  : 9672 milliseconds.
Win32: 9953 milliseconds.

They're much closer together on XP. (Not to mention oodles faster.)
Lastly, I tried Linux. (500 MHz):

Code:

Trying to open file "nonexists" 100000 times.
CRT  : 836 milliseconds.

None of the tested operating systems seem to do actual disk I/O for the entire duration. (Which is good.)
The CRT outperformed the Win32 API once on Win9x, but no other test did. It outperformed the Win32 API every time on XP. (I ran more than one test, the results above are just samples.)

EDIT: The above test is probably skewed. The three directories that it is looking for "nonexists" in had varying numbers of files, which may have helped/hindered some systems.