Thread: vector::insert is so slow

  1. #16
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    I see that there is some kind of error checking possible but it doesn't seem to be as detailed as Win32 error codes and even if they were what would be the benefit to learn them seeing that these functions call the same Win32 functions themselves.
    Windows 10 with Visual Studio 2019 and MingW.

  2. #17
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,223
    Quote Originally Posted by Ducky
    even if they were what would be the benefit to learn them seeing that these functions call the same Win32 functions themselves.
    I have not checked the standard library implementation to verify, but I am fairly certain that they do not call the Win32 functions on the computer I use at work
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  3. #18
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by Ducky View Post
    I see that there is some kind of error checking possible but it doesn't seem to be as detailed as Win32 error codes and even if they were what would be the benefit to learn them seeing that these functions call the same Win32 functions themselves.
    We've been over this before. It's called abstraction. Win32 is low level APIs. Iostreams are much higher level, and they are purely C++ to boot, which give them a much better integration with the language.
    For example, say you open a file. Say an exception is thrown. With iostreams, the files are closed automatically (and any written data is flushed). Win32? Not so much. Data corruption possible!
    Stick with the same language as much as possible! Ignore the Win32 api. Learn how to use C++ properly and well instead of trying to understand these dangerous low-level C apis.
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  4. #19
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    Thank you Elysia, I didn't know about that.
    Windows 10 with Visual Studio 2019 and MingW.

  5. #20
    Guest Sebastiani's Avatar
    Join Date
    Aug 2001
    Location
    Waterloo, Texas
    Posts
    5,708
    Another common bottleneck is the strlen function being invoked repeatedly on the same data in a loop. Recalculate variables *only* when necessary, and always be mindful of the various ramifications of calling *any* function whatsoever, whether it be a question of speed, memory footprint, global dependencies, or what have you...
    Code:
    #include <cmath>
    #include <complex>
    bool euler_flip(bool value)
    {
        return std::pow
        (
            std::complex<float>(std::exp(1.0)), 
            std::complex<float>(0, 1) 
            * std::complex<float>(std::atan(1.0)
            *(1 << (value + 2)))
        ).real() < 0;
    }

  6. #21
    Registered User
    Join Date
    May 2010
    Posts
    4,581
    What is "slow"?

    What type of variable is filesize and what exactly is it's value?

    What is the actual size of your vector?
    @jimblumberg No problem with the size as its less than 2GB and it works with smaller files.
    Are you sure?

    What type of variable is filesize? This is an important question. Looking at the code you posted in post #8 you're now using a variable named iFileSize which is a long long (by the way since you also now appear to be using the crappy Hungarian notation it should be LLi not i). This could cause problems, because a vector can only std::vector.max_size() elements. The max_size() is quite often smaller than a long long. Since you're trying to use a long long to size a vector you could run into problems.

    So what is the max() for your long long on your machine, and the vector.max_size()? On my machine they are:

    Code:
    4294967295  // Maximum number of vector elements.
    9223372036854775807 // Maximum value of a long long.
    Remember on most systems a file can be much larger than what can be held in any type of container.

    Jim

  7. #22
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by laserlight View Post
    I have not checked the standard library implementation to verify, but I am fairly certain that they do not call the Win32 functions on the computer I use at work.
    Somewhere in the process, the standard library file I/O functions will have to deal with FAT or NTFS file system. I don't know if this is done via Win32 functions or some other method common to both.

    Quote Originally Posted by jimblumberg View Post
    What type of variable is filesize?
    Assuming NTFS file system, filesize is returned as 64 bits (I don't know what the actual max size is, but I suspect it's 2 terabytes), but assuming the file size is less than 4GB, then truncating the long long to an unsigned int won't be an issue, and if filesize is less than 2GB, then truncating long long to signed int won't be an issue.
    Last edited by rcgldr; 06-15-2013 at 06:22 PM.

  8. #23
    C++まいる!Cをこわせ!
    Join Date
    Oct 2007
    Location
    Inside my computer
    Posts
    24,654
    Quote Originally Posted by jimblumberg View Post
    So what is the max() for your long long on your machine, and the vector.max_size()? On my machine they are:

    Code:
    4294967295  // Maximum number of vector elements.
    9223372036854775807 // Maximum value of a long long.
    Remember on most systems a file can be much larger than what can be held in any type of container.

    Jim
    On my machine, they are:

    18446744073709551615 // Maximum vector elements
    9223372036854775807 // Maximum long long value
    Quote Originally Posted by Adak View Post
    io.h certainly IS included in some modern compilers. It is no longer part of the standard for C, but it is nevertheless, included in the very latest Pelles C versions.
    Quote Originally Posted by Salem View Post
    You mean it's included as a crutch to help ancient programmers limp along without them having to relearn too much.

    Outside of your DOS world, your header file is meaningless.

  9. #24
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Somewhere in the process, the standard library file I/O functions will have to deal with FAT or NTFS file system. I don't know if this is done via Win32 functions or some other method common to both.
    O_o

    Did you even hear the joke fly over?

    Soma

  10. #25
    Registered User
    Join Date
    May 2010
    Posts
    4,581
    Assuming NTFS file system, filesize is returned as 64 bits (I don't know what the actual max size is, but I suspect it's 2 terabytes), but assuming the file size is less than 4GB, then truncating the long long to an unsigned int won't be an issue, and if filesize is less than 2GB, then truncating long long to signed int won't be an issue.
    You're assuming entirely too much!

    Sure the OP states that today the files sizes are less than 4GB, but what happens tomorrow? My point is you don't want to assume anything, use the correct type of variables. In this case use a vector<char>::size_type for the size, and for the file size use the std::ios::streamsize when dealing with streams. These typedefs take the guessing out of the equation. Then always check that the streamsize is less than the vector::max_size() before trying to resize your vector based on the file size.

    Jim

  11. #26
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    @jimblumberg My values are like yours. Didnt know they were machine dependent. I used a long long cause later I wanted to open files larger than 4GB.
    Thanks for pointing this out. I guess it just better to read them in chunks then.
    But I would still need a long long to know the size of a file larger than 4GB, wouldn't I.

    @Sebastiani Thanks for the clarification, I will change those strlen()s.
    Last edited by Ducky; 06-15-2013 at 11:51 PM.
    Windows 10 with Visual Studio 2019 and MingW.

  12. #27
    Registered User
    Join Date
    May 2010
    Posts
    4,581
    But I would still need a long long to know the size of a file larger than 4GB, wouldn't I.
    No, you will need a std::ios::streamsize type. A streamsize is not necessarily a long long, it is a type that can hold the file position information for any sized file that can be handled on the particular operating system being used.
    Didnt know they were machine dependent.
    The size of the largest file, and the maximum number of items a container can hold are compiler specific values, based on the underlying operating system values. Always use the correct type variables.

    The purpose of these typedefs is to allow the program to reliably work on any operating system that has a standard compliant compiler. You should always check that your index values are within the acceptable limits of your compiler/operating system. By using these typedefs and insuring the index values are acceptable, you'll make using files and containers (including arrays) safer no matter what operating system or compiler you're using. The compiler writers know the limits of the operating system better than most of us so use this knowledge.


    Jim

  13. #28
    Registered User
    Join Date
    Dec 2007
    Posts
    920
    Thanks. I will use streamsize then. On a 32bit system we cannot open a file greater than 2GB either way as I understood.
    We can open it but the size will be undefined.
    Windows 10 with Visual Studio 2019 and MingW.

  14. #29
    Registered User
    Join Date
    May 2010
    Posts
    4,581
    On a 32bit system we cannot open a file greater than 2GB either way as I understood.
    We can open it but the size will be undefined.
    If your operating system doesn't support files larger than 2GB then there will be no files larger than 2GB to open, the operating system limits the maximum file size.

    If your running a 64 bit operating system your file sizes may be larger than what is available on a 32 bit operating system. And if you're compiling a program on a 64 bit operating system with a 32 bit compiler the actual file sizes could possibly be larger than your program could support.

    Jim

  15. #30
    Registered User
    Join Date
    Apr 2013
    Posts
    1,658
    Quote Originally Posted by jimblumberg View Post
    If your operating system doesn't support files larger than 2GB then there will be no files larger than 2GB to open, the operating system limits the maximum file size.

    If your running a 64 bit operating system your file sizes may be larger than what is available on a 32 bit operating system. And if you're compiling a program on a 64 bit operating system with a 32 bit compiler the actual file sizes could possibly be larger than your program could support.
    For Windows, it doesn't matter if the operating system is 32 bit or 64 bit. For FAT32, maximum file size is 4GB - 1. For NTFS, maximum file size depends on the NTFS parameters that a version of Windows supports. Wiki article:

    NTFS - Wikipedia, the free encyclopedia

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 04-27-2011, 05:31 AM
  2. Insert new element into a vector
    By appointment in forum C++ Programming
    Replies: 7
    Last Post: 08-16-2009, 01:54 AM
  3. vector<...>::iterators and insert - I'm stumped.
    By Dino in forum C++ Programming
    Replies: 6
    Last Post: 12-25-2007, 06:11 AM
  4. vector insert functions
    By kes103 in forum C++ Programming
    Replies: 4
    Last Post: 03-24-2003, 03:12 PM
  5. Replies: 1
    Last Post: 09-17-2001, 05:46 AM