Thread: vector::insert is so slow

Originally Posted by rcgldr

For Windows, it doesn't matter if the operating system is 32 bit or 64 bit. For FAT32, maximum file size is 4GB - 1. For NTFS, maximum file size depends on the NTFS parameters that a version of Windows supports.

I can only get the size of a file of 2GB with ifstream and around 3.5GB with CreateFile(). They will open it but give back a false size.
I used a long long to be sure. Win7 64bit. 32bit program.

I also checked my max value is 9223372036854775807.

Code:

cout << numeric_limits<streamoff>::max() << endl;

Originally Posted by Ducky

I can only get the size of a file of 2GB with ifstream and around 3.5GB with CreateFile().

I tried this old program with a 6GB .avi file captured by fraps, and it returned the proper file size:6,555,552,672 == 0x186bdcba0 bytes. This was run on Windows XP Pro 32 bit. It takes a bit less than 100 seconds to read the entire file (August, 2011 system, Intel 2600K cpu, 4GB ram, three 2 TB hard drives).

Code:

#include <windows.h>                    /* Win32 API */
#include <stdio.h>

#define BFRSZ 0x40000000ul              /* buffer size == 1GB */

static HANDLE   hSrc;                   /* source file params*/
static DWORD    dwSrcCnt;
static LARGE_INTEGER liSrcSize;
static LARGE_INTEGER liSrcCnt;

static DWORD    dwErr;                  /* getlasterror params */
static BYTE     abErrMsg[128];

static LPVOID   lpBfr;                  /* allocated bufer */

int main (int argc, BYTE *argv[])
{
    hSrc = INVALID_HANDLE_VALUE;        /* init handle */

    hSrc = CreateFile(argv[1],          /* open source file */
        GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, 0);

    if(hSrc == INVALID_HANDLE_VALUE){
        hSrc = 0;
        goto exit0;
    }

    if(0 == GetFileSizeEx(hSrc, &liSrcSize)){
        dwErr = GetLastError();
        FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM,
            NULL, dwErr, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
            (LPTSTR) abErrMsg, sizeof(abErrMsg), NULL );
        printf("\n GetFileSizeEx() failed, code %d:%s", dwErr, abErrMsg);
    }
    printf("%8lx%08lx file size\n", liSrcSize.HighPart, liSrcSize.LowPart);

    lpBfr = malloc(BFRSZ);              /* allocate buffer */
    if(lpBfr == NULL)
        goto exit0;

    while(1){                           /* read file */
        ReadFile(hSrc, lpBfr, BFRSZ, &dwSrcCnt, NULL);
        if(!dwSrcCnt)
            break;
        liSrcCnt.QuadPart += dwSrcCnt;
    }
    printf("%8lx%08lx bytes read\n", liSrcCnt.HighPart, liSrcCnt.LowPart);
  
exit0:
    if(hSrc)                            /* close file */
        CloseHandle(hSrc);
    if(lpBfr)                           /* release mem */
        free(lpBfr);

    return(0);
}

@rcgldr
Thanks. It's working for me too with a file more than 8GB.
The problem was that I was using GetFileSize and not GetFileSizeEx.
Looks like I will just have to use CreateFile() instead of ifstream if I want to copy larger files.

Originally Posted by Ducky

Looks like I will just have to use CreateFile() instead of ifstream if I want to copy larger files.

How about you compile a 64-bit executable instead?

I could definitely do that. Most of the home computers are 64-bit nowadays.

How about you get on with the times and use a modern compiler?
For that matter, how about not using goto and stop relying on explicit close, and using constructors?

Code:

#include <ios>
#include <iostream>
#include <fstream>

int main()
{
	std::ifstream  is("J:/Software/Windows 8.iso", std::ios::binary);
	if (is.fail())
	{
		std::cout << "can't open file" << std::endl;
		return 1;
	}
	is.seekg(0, std::ios::end);
	std::cout << sizeof(std::streamoff) << std::endl;
	std::cout << is.tellg() << std::endl;
	is.seekg(0);
	return 0;
}

Output:
8
3581853696

Visual Studio 2012
In fact, it doesn't matter whether it's a 32-bit or 64-bit executable. The output is the same.

(Yes, this is a legal copy of Windows 8 from dreamspark premium, and no, I am still using Windows 7.)

Originally Posted by Elysia

How about you get on with the times and use a modern compiler?

It's $500 (my wife would object). For this I could use VS 2012 C++ express though.

Originally Posted by Elysia

For that matter, how about not using goto and stop relying on explicit close, and using constructors?

I started with an old generic program that uses a common (single) exit point. Normally the code after exit0 contains stuff to close or free up everything that was allocated, like files, memory, other threads, mutexes, sempahores, ...

Originally Posted by phantomotap

You are running an Itanium?

I'm running Windows XP Professional 64 bit edition (X86-64), not Window XP 64 bit edition (Itanium). Wiki article:

Windows XP Professional x64 Edition - Wikipedia, the free encyclopedia

I also have Windows 7 64 bit, but rarely use it, as I'm looking for a file oriented backup and restore app before I transition over to it. (For XP, you can multi-boot into another OS and do a file / folder copy to backup or restore a partition, but for Windows Vista and later, reparse points and sercurity settings on folders is an issue).

Originally Posted by rcgldr

It's $500 (my wife would object). For this I could use VS 2012 C++ express though.

Not proposing you'd shell out $500 for an IDE o_O
You could use Express as you pointed out, or use gcc or clang for compilers. They are both better than msvc in terms of C/C++ support and are generally, as far as I've found, less buggy and more standards conformant.

I started with an old generic program that uses a common (single) exit point. Normally the code after exit0 contains stuff to close or free up everything that was allocated, like files, memory, other threads, mutexes, sempahores, ...

That's what the RAII idom is for.