Thread: vector::insert is so slow

  1. #31
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    Quote Originally Posted by rcgldr View Post
    For Windows, it doesn't matter if the operating system is 32 bit or 64 bit. For FAT32, maximum file size is 4GB - 1. For NTFS, maximum file size depends on the NTFS parameters that a version of Windows supports.
    I can only get the size of a file of 2GB with ifstream and around 3.5GB with CreateFile(). They will open it but give back a false size.
    I used a long long to be sure. Win7 64bit. 32bit program.

    I also checked my max value is 9223372036854775807.

    Code:
    cout << numeric_limits<streamoff>::max() << endl;
    Windows 10 with Visual Studio 2019 and MingW.

  2. #32
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by Ducky View Post
    I can only get the size of a file of 2GB with ifstream and around 3.5GB with CreateFile().
    I tried this old program with a 6GB .avi file captured by fraps, and it returned the proper file size:6,555,552,672 == 0x186bdcba0 bytes. This was run on Windows XP Pro 32 bit. It takes a bit less than 100 seconds to read the entire file (August, 2011 system, Intel 2600K cpu, 4GB ram, three 2 TB hard drives).

    Code:
    #include <windows.h>                    /* Win32 API */
    #include <stdio.h>
    
    #define BFRSZ 0x40000000ul              /* buffer size == 1GB */
    
    static HANDLE   hSrc;                   /* source file params*/
    static DWORD    dwSrcCnt;
    static LARGE_INTEGER liSrcSize;
    static LARGE_INTEGER liSrcCnt;
    
    static DWORD    dwErr;                  /* getlasterror params */
    static BYTE     abErrMsg[128];
    
    static LPVOID   lpBfr;                  /* allocated bufer */
    
    int main (int argc, BYTE *argv[])
    {
        hSrc = INVALID_HANDLE_VALUE;        /* init handle */
    
        hSrc = CreateFile(argv[1],          /* open source file */
            GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, 0);
    
        if(hSrc == INVALID_HANDLE_VALUE){
            hSrc = 0;
            goto exit0;
        }
    
        if(0 == GetFileSizeEx(hSrc, &liSrcSize)){
            dwErr = GetLastError();
            FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM,
                NULL, dwErr, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT),
                (LPTSTR) abErrMsg, sizeof(abErrMsg), NULL );
            printf("\n GetFileSizeEx() failed, code %d:%s", dwErr, abErrMsg);
        }
        printf("%8lx%08lx file size\n", liSrcSize.HighPart, liSrcSize.LowPart);
    
        lpBfr = malloc(BFRSZ);              /* allocate buffer */
        if(lpBfr == NULL)
            goto exit0;
    
        while(1){                           /* read file */
            ReadFile(hSrc, lpBfr, BFRSZ, &dwSrcCnt, NULL);
            if(!dwSrcCnt)
                break;
            liSrcCnt.QuadPart += dwSrcCnt;
        }
        printf("%8lx%08lx bytes read\n", liSrcCnt.HighPart, liSrcCnt.LowPart);
      
    exit0:
        if(hSrc)                            /* close file */
            CloseHandle(hSrc);
        if(lpBfr)                           /* release mem */
            free(lpBfr);
    
        return(0);
    }
    Last edited by rcgldr; 06-17-2013 at 07:53 AM.

  3. #33
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    @rcgldr
    Thanks. It's working for me too with a file more than 8GB.
    The problem was that I was using GetFileSize and not GetFileSizeEx.
    Looks like I will just have to use CreateFile() instead of ifstream if I want to copy larger files.
    Windows 10 with Visual Studio 2019 and MingW.

  4. #34
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by Ducky View Post
    Looks like I will just have to use CreateFile() instead of ifstream if I want to copy larger files.
    How about you compile a 64-bit executable instead?
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  5. #35
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    I could definitely do that. Most of the home computers are 64-bit nowadays.
    Windows 10 with Visual Studio 2019 and MingW.

  6. #36
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by Ducky View Post
    Looks like I will just have to use CreateFile() instead of ifstream if I want to copy larger files.
    Quote Originally Posted by Elysia View Post
    How about you compile a 64-bit executable instead?
    Quote Originally Posted by Ducky View Post
    I could definitely do that. Most of the home computers are 64-bit nowadays.
    Be sure to make sure this works with Visual Studio 2010. The code below fails with Visual Studio 2005 in 64 bit mode, running with Windows XP Pro 64 bit, if the file size is >= 2GB. In 32 bit mode, sizeof(streamoff) == 4 and in 64 bit mode, sizeof(streamoff) == 8, but somewhere in the 64 bit mode, a 32 bit signed integer is getting involved, even though the input and outputs are 64 bits. If the file is >= 2GB, then "so" ends up as -1 (0xffffffffffffffff).

    Code:
    #include <ios>
    #include <iostream>
    #include <fstream>
    
    using namespace std;
    
    int main(int argc, char **argv)
    {
        ifstream  is;
        streamoff so = 0;
        is.open("test.bin", ios::binary );
        if(is.fail()){
            cout << "can't open file" << endl;
            goto exit0;
        }
        is.seekg(0, ios::end);
        so = is.tellg();
        is.seekg(0);
        is.close();
    
        cout << sizeof(so) << endl;
        cout << so << endl;
    
    exit0:
        return(0);
    }

  7. #37
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    How about you get on with the times and use a modern compiler?
    For that matter, how about not using goto and stop relying on explicit close, and using constructors?

    Code:
    #include <ios>
    #include <iostream>
    #include <fstream>
    
    int main()
    {
    	std::ifstream  is("J:/Software/Windows 8.iso", std::ios::binary);
    	if (is.fail())
    	{
    		std::cout << "can't open file" << std::endl;
    		return 1;
    	}
    	is.seekg(0, std::ios::end);
    	std::cout << sizeof(std::streamoff) << std::endl;
    	std::cout << is.tellg() << std::endl;
    	is.seekg(0);
    	return 0;
    }
    Output:
    8
    3581853696

    Visual Studio 2012
    In fact, it doesn't matter whether it's a 32-bit or 64-bit executable. The output is the same.

    (Yes, this is a legal copy of Windows 8 from dreamspark premium, and no, I am still using Windows 7.)
    Last edited by Elysia; 06-17-2013 at 05:18 PM.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  8. #38
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Windows XP Pro 64 bit
    O_o

    You are running an Itanium?

    That's just weird, but kind of awesome. I haven't seen one in a long time.

    Soma

  9. #39
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by Elysia View Post
    How about you get on with the times and use a modern compiler?
    It's $500 (my wife would object). For this I could use VS 2012 C++ express though.

    Quote Originally Posted by Elysia View Post
    For that matter, how about not using goto and stop relying on explicit close, and using constructors?
    I started with an old generic program that uses a common (single) exit point. Normally the code after exit0 contains stuff to close or free up everything that was allocated, like files, memory, other threads, mutexes, sempahores, ...

    Quote Originally Posted by phantomotap View Post
    You are running an Itanium?
    I'm running Windows XP Professional 64 bit edition (X86-64), not Window XP 64 bit edition (Itanium). Wiki article:

    Windows XP Professional x64 Edition - Wikipedia, the free encyclopedia

    I also have Windows 7 64 bit, but rarely use it, as I'm looking for a file oriented backup and restore app before I transition over to it. (For XP, you can multi-boot into another OS and do a file / folder copy to backup or restore a partition, but for Windows Vista and later, reparse points and sercurity settings on folders is an issue).
    Last edited by rcgldr; 06-17-2013 at 06:43 PM.

  10. #40
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by rcgldr View Post
    It's $500 (my wife would object). For this I could use VS 2012 C++ express though.
    Not proposing you'd shell out $500 for an IDE o_O
    You could use Express as you pointed out, or use gcc or clang for compilers. They are both better than msvc in terms of C/C++ support and are generally, as far as I've found, less buggy and more standards conformant.

    I started with an old generic program that uses a common (single) exit point. Normally the code after exit0 contains stuff to close or free up everything that was allocated, like files, memory, other threads, mutexes, sempahores, ...
    That's what the RAII idom is for.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  11. #41
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Normally the code after exit0 contains stuff to close or free up everything that was allocated, like files, memory, other threads, mutexes, sempahores, ...
    O_o

    If you are going to program in C++, you need to learn to use C++ constructs, canonical standards, and best practices.

    You can use `goto' in C for cleanup if you must, but `goto' can't do anything for you in C++ in the face of exceptions. (Before you post some idiotic link confusing the exception mechanism, you should actually read about it from experts; the use of `goto' for cleanup will not protect you in the face of exceptions.) As Elysia says, you need to use "RAII" for that in C++.

    I'm running Windows XP Professional 64 bit edition (X86-64), not Window XP 64 bit edition (Itanium).
    My confusion was born of your common mistake: "Windows XP Professional 64bit" != "Windows XP Professional x64"

    Soma

  12. #42
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by phantomotap View Post
    My confusion was born of your common mistake: "Windows XP Professional 64bit" != "Windows XP Professional x64"
    You can blame Microsoft for the confusion. Win XP Pro 64 == Windows XP Professional 64 bit != Windows XP 64 bit. The difference is that there's no "Pro" or "Professional" in the name for the Itanium version.

  13. #43
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    The difference is that there's no "Pro" or "Professional" in the name for the Itanium version.
    O_o

    There is also no "bit" in the x64 version.

    *shrug*

    I suppose the marketing guys were just nuts.

    Soma

  14. #44
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by phantomotap View Post
    There is also no "bit" in the x64 version.
    I get the names confused myself. I use it on a multi-boot system to backup Win XP Pro (32 bit), and to test 64 bit programs. There were some games that optionally supported XP Pro X64, but I never tried them out. One nice thing about XP Pro X64 on a multi-boot system is if a partition gets messed up with mangled directories or security info, it can reformat (quick format) that partition when other versions of Windows will refuse to do so. Eventually I'll have to transition to Windows 7 since most new hardware and ever growing percentage of software apps require Windows 7 or Windows 8.

  15. #45
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by phantomotap View Post
    goto can't do anything for you in C++ in the face of exceptions.
    True, but I still see examples of goto for cleanup such as this one from MSDN which is C++ specific:

    Creating a Security Descriptor for a New Object in C++ (Windows)

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 04-27-2011, 05:31 AM
  2. Insert new element into a vector
    By appointment in forum C++ Programming
    Replies: 7
    Last Post: 08-16-2009, 01:54 AM
  3. vector<...>::iterators and insert - I'm stumped.
    By Dino in forum C++ Programming
    Replies: 6
    Last Post: 12-25-2007, 06:11 AM
  4. vector insert functions
    By kes103 in forum C++ Programming
    Replies: 4
    Last Post: 03-24-2003, 03:12 PM
  5. Replies: 1
    Last Post: 09-17-2001, 05:46 AM