Thread: Fstream.read unicode file

1. If I open a unicode file with notepad, I get: "Package Requires Reboot = no".

But If I read it with fstream, I get: "P a c k a g e R e q u i r e s R e b o o t = n o".

I found the reason and a solution for it in this article but I'm wondering if that is the right solution?

Code:

wstring ReadUTF16(const string & filename)
{
    ifstream file(filename.c_str(), std::ios::binary);
    stringstream ss;
    ss << file.rdbuf() << '\0';
    return wstring((wchar_t *)ss.str().c_str());
}

http://cfc.kizzx2.com/index.php/read...-in-windows-c/

Wouldn't this solution be faster because the '<<' operator copies byte by byte?

Code:

wstring ReadUTF16(char * filename)
{
    ifstream ifs(filename, std::ios::binary);
    ifs.seekg(0, ifs.end);
    std::streampos length = ifs.tellg();
    ifs.seekg(0, ifs.beg);
  
    std::vector<char> cVec(length);
    ifs.read(&cVec[0],length);
    return wstring((wchar_t *)&cVec[0]);
}

2. And if I want to write a function to read both ANSI and Unicode should I check for the two first byte to know the encoding first?

So I determine first the encoding of the file and then I open it up with fstream or wfstream accordingly?

And then I use wcstombs(str, wstr, sizeof(str)); to translate it to ANSI?

Originally Posted by Ducky

1. If I open a unicode file with notepad, I get: "Package Requires Reboot = no".

If the file is in UTF-16, then each character takes up at least two bytes, where for every character in the ASCII set is the same in the unicode set except the second byte is 0. So that explains why.

>>char * filename
Don't use char* for string literals! Use std::string.

Originally Posted by Ducky

2. And if I want to write a function to read both ANSI and Unicode should I check for the two first byte to know the encoding first?

Originally Posted by Ducky

So I determine first the encoding of the file and then I open it up with fstream or wfstream accordingly?

All files don't have encoding information. Files written in linux are very often UTF-8 without any byte marker indicating that it's unicode. Indeed, this is actually the recommended way, so determining the encoding is not always easy.

Originally Posted by Ducky

And then I use wcstombs(str, wstr, sizeof(str)); to translate it to ANSI?

Never use ANSI. Use UTF-8 internally and store it in a std::string. I've got code to convert between ANSI <--> Unicode, as well as UTF8 <--> UTF16. I can share it if you're interested.

Originally Posted by Elysia

Never use ANSI. Use UTF-8 internally and store it in a std::string. I've got code to convert between ANSI <--> Unicode, as well as UTF8 <--> UTF16. I can share it if you're interested.

Yes please, I would be interested because with the CP_UTF8 parameter I get some weird characters and with CP_ACP it wont get translated at all, it stays Unicode.

Code:

        wchar_t * wbuf = new wchar_t [length];
        ifs.read(wbuf,length);

        char * buf = new char[length];
        int ret=WideCharToMultiByte(
                CP_ACP,
                0,
                wbuf,// wide-character string to be converted
                -1,  //this parameter can be set set to -1 if the string is null-terminated.
                buf, //buffer that receives the converted string.
                length, // output buf size
                NULL, NULL);

        string s(buf);
        cout << " \n" << buf << '\n';

Originally Posted by MOS-6581

Determining the encoding of a file is not easy. There are functions in Windows that can allow you to detect the encoding/code page through statistical analysis of the text file but in general you shouldn't try to detect that at all. Most programs simply go with whatever encoding/code page that is the default for the system if the encoding is unknown and then allow the user to manually reload the file with the correct encoding if needed.

I want to use std::find. So if the file is in Unicode it wont work if I compare 'hello' with 'h e l l o'.

Originally Posted by MOS-6581

Keep in mind that there's not a one-to-one mapping between UTF-8 and UTF-16. A character encoded as UTF-16 can take up more or less bytes when converted to UTF-8. To obtain the correct size of the buffer where the conversion is stored then you should first call WideCharToMultiByte like this:

Code:

int buffer_size_needed = WideCharToMultiByte(CP_UTF8, WC_ERR_INVALID_CHARS, string_to_convert, -1, NULL, 0, NULL, NULL);

Thanks a lot. I tried it that way but I still only get garbage chars.

Code:

     int buffer_size_needed = WideCharToMultiByte(CP_UTF8,
                                                     WC_ERR_INVALID_CHARS,
                                                     wbuf, //string_to_convert,
                                                     -1,
                                                     NULL,
                                                     0,
                                                     NULL,
                                                     NULL);

     int ret = WideCharToMultiByte(CP_UTF8,
                                    0,
                                    wbuf,// wide-character string to be converted
                                    -1,  //this parameter can be set set to -1 if the string is null-terminated.
                                    buf, //buffer that receives the converted string.
                                    buffer_size_needed, // output buf size
                                    NULL, NULL);
                    

     for(int i=0; i<length; i++) cout << (char)buf[i];

Thanks.

This is the .txt file I open:

Package Requires Reboot = no
New Uninstall Key = inf2
Progress Thread Wait = Success
[Finish]
[ResponseResult]
ResultCode = 0

It is read into wbuf I verified it. I also tried to attach it but the site says its an invalid file.

Code:

#include <Windows.h>
#include <locale>
#include <vector>
#include <string>
#include <sstream>
#include <fstream>
#include <iostream>     // std::cout
using namespace std;


int main()
{
    string szFilename = "IntelChipset.log";
    std::wifstream ifs(szFilename.c_str(), std::ifstream::binary);
    if(ifs)
    {
        // get length of file:
        ifs.seekg(0, ifs.end);
        std::streampos length = ifs.tellg();
        ifs.seekg(0, ifs.beg);

        wchar_t * wbuf = new wchar_t [length];

        cout << " Reading " << length << " characters... \n";
        // read data as a block:
        ifs.read(wbuf,length);

        wstring wstr;
        wstr.copy(wbuf,length,0);
        wcout << " wstr " << wstr << '\n';
        for(int i=0; i<length; i++) cout << (char)wstr[i];

        std::string::size_type pos = 0;
        if((pos = wstr.find(L"R e b o o t", pos)) != std::string::npos)
        {
            cout << "Found at "<< pos << '\n';
        }
        else
        {
           cout << " Not found " << '\n';
        }

        char * buf = new char[length];

        UINT CodePage = CP_UTF8;

        int buffer_size_needed = WideCharToMultiByte(
                                   CodePage,
                                   WC_ERR_INVALID_CHARS,
                                   wbuf, //string_to_convert,
                                   -1,
                                   NULL,
                                   0,
                                   NULL,
                                   NULL);

        cout << " buffer_size_needed " << buffer_size_needed << '\n';

        //int ret = wcstombs(buf, wbuf, length);
        int ret = WideCharToMultiByte(
                    CodePage,
                    0,
                    wbuf,
                    -1, 
                    buf,
                    buffer_size_needed,
                    NULL,
                    NULL);

        cout << " ret " << ret << '\n';

        string s(buf);
        
        cout << " \n string " << s << '\n';
    }
    else
    {
        cout << " Error opening " << szFilename << "\n";
        return 1;
    }
    ifs.close();

    return 0;
}

Here's my version: How to set up libconv in visual c++

gg

I did some testing myself and found some strange results that I can't explain. I first created a simple file in Notepad with the text "Hello" and then saved it as "Unicode". Windows reports that the file size is 12 bytes which is what I would expect since it's five characters + BOM.

Looking at the integer values of what I read with wifstream::read then I get this:
255 (BOM)
254 (BOM)
87
0
111
0

How the heck did I end up reading that? If I open the text file in Notepad again then I can still see "Hello" and the file size is still reported as 12 bytes.

EDIT: It seems that wifstream::read actually reads the file one byte at a time and then stores each byte in a wchar_t. In order words it's useless for reading UTF-16 on Windows.

Yes, wifstream will simply perform the wchar_t -> char conversion under the hood unless you replace the codecvt: Unicode text file

Another solution: https://msdn.microsoft.com/en-us/library/ee292208.aspx

And a MS CRT solution is to use the "ccs=ENCODING" extension: https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx

gg