Thread: vector::insert is so slow

I opened up a large file of 700MB and read it in a vector<char> and then I want to insert in front of the file its size.
I turns out to be not only very slow but also max out the CPU.
Is it normal or am I doing something wrong?
It works well on smaller files though.

Code:

    char szDigits[32] = "700111222";
    vector<char> v(filesize+260, '0'); // contains file 

    for(int i = 0; i< 9; i++)  // it gets very slow here
    {
        v.insert(v.begin()+i, szDigits[i]);
        cout << "i: " << i << endl;
    }

Originally Posted by Ducky

I opened up a large file of 700MB and read it in a vector<char> and then I want to insert in front of the file its size.

Reserve enough bytes in teh beginning of the vector to write size later. start reading file from the preset offset in the vector. Then just overwrite the reserved bytes with actual size.

@antred Can I dereference a deque<char> as a vector<char> &v.at(0) or I have to switch back and forth between them?

@vart I'm using ReadFile() to read it to the vector and ReadFile() can only read to the beginning of the file.
It also crashes if I only reserve the size and not initialize it.
But I will try it like ReadFile(&v.at(offset)).

@jimblumberg No problem with the size as its less than 2GB and it works with smaller files.

Originally Posted by Ducky

I'm using ReadFile() to read it to the vector and ReadFile() can only read to the beginning of the file.

I do not see how it is relevant here. And I'm sure ReadFile is able to read at any offset in the file... You just do not need this feature.

Originally Posted by Ducky

It also crashes if I only reserve the size and not initialize it.

So you doing it in the wrong way. Show us the code.

What I mean - you create a vector of say 15 bytes

read starting from the offset 5 10 bytes from file
and after you finish reading - write the number of bytes read into the first 5 bytes of the vector.

Thanks, I worked it out with resize().
- so I push back the header info in an empty vector
- I resize it to the right size
- then I read the file content just after the header info

It's very fast now. Do you think I still need reserve()?
It seem to work without it.

Code:

int main()
{
    char * Buf2 =  "C:\\a.txt";
    char Buf3[260];
    HANDLE hFile=CreateFile(Buf2,
                            GENERIC_READ,
                            FILE_SHARE_WRITE,
                            NULL,
                            OPEN_ALWAYS,0,NULL);

    cout << "File created... " << endl;

    LARGE_INTEGER lpFileSize;
    if(!GetFileSizeEx(hFile,&lpFileSize))
    {
        return 1;
    }

    long long iFileSize=lpFileSize.QuadPart, r;

    int iWholeSize = 1 + strlen(Buf2) + 1 + iFileSize;
    int iDigits = GetNumberOfDigits(iWholeSize);
    iWholeSize += iDigits;
    char szDigits[32];
    itoa (iWholeSize, szDigits,10);
    sprintf(Buf3, "%s*%s*", szDigits, Buf2);

    vector<char> v;
    /// ////////////////////////////
    /// Insert header
    for(int i = 0; i< strlen(Buf3); i++)
    {
        v.push_back(Buf3[i]);
    }
    v.resize(iFileSize+260);
    /// ////////////////////////////
    /// Read file
    DWORD dwRead = 0;
    if(!ReadFile(hFile, &v.at(strlen(Buf3)),
                 iFileSize,
                 &dwRead, NULL))

You don't need reserve if you are using resize, but let me ask you a couple of things:
- Why are you using Win32 API instead of iostreams?
- Why are you using char arrays and char pointers instead of std::string?
- Why are you using sprintf instead of stringstreams?
- Why are you manually calculating offsets? Why aren't you using a struct to store all the information you need? I see a lot of magic numbers everywhere too.

1. There is no error checking possible with iostream.
2. Very good question. I've still got it stuck in my mind that Win32 functions won't take anything but pointers. But now I that I know that they can take a vector<char> I will change them all. Thanks for the reminder.
3. Ditto
4. A struct. Yes that would make it much cleaner but it needed only to calculate once so there isn't much calculation involved.

Thanks for the help!
Cheers.

Originally Posted by Ducky

1. There is no error checking possible with iostream.

Why not?

There are no return values with error codes like with Win32 functions.

fstream:pen return value: none
fstream:pen - C++ Reference

Originally Posted by Ducky

There are no return values with error codes like with Win32 functions.

fstream::open return value: none
fstream::open - C++ Reference

There is a specific part on the site you linked to that describes what happens when there's an error. I suggest you take a closer look.