Thread: Unicode Support for std::getline()

Originally Posted by Codeplug

>> but beware - even though it expects UTF16LE, it actually expects a fixed-length string, so any character that takes more than two bytes to store won't work!
I don't believe that is true. Windows supports UTF16 in all it's glory.

Perhaps my information is outdated. I was under the impression that it did not support it.
A quick test shows that it does work on W10, though.

Code:

#include <Windows.h>

int main()
{
	std::wstring w = L"��"; // Yes, this takes two UTF16 words (Looks like the board can't handle the character; get one from here).
	MessageBoxW(nullptr, w.c_str(), L"", 0);
}

So yeah, sorry about that. It seems that my information may not be reliable.

Originally Posted by Elysia

The easiest way to do Unicode is to use UTF8. UTF8 just "works" with your typical strings, algorithms and whatnot that expects all characters to be one byte. Save your files in UTF8, read your files in UTF8 (using the narrow versions of the API) and you should have 90% of the issues down.

I would be very cautious saying it just "works". As long as you deal with in-memory strings only it might be fine. However, if you try to use Unicode with any other standard C++ stuff, you are out of luck.

Streams: as of now there is completely no standard way to open a file stream using a Unicode path. The only standard constructors are const char* and std::string, which ofc. expect MBCS on Windows. There is no way (no trivial way?) to read UTF-16 stream either. Detecting BOM? Please...

Locales: the locale support in C++ is also broken. Functions like isalpha() or toupper() work on single chars, which means that they ignore the whole idea of variable-length encodings.

Regexes: standard regular expressions use locales under the hood, so they cannot be any better.

The C++ support of Unicode is terrible. I find the overall design of code converters and locales overengineered, yet broken at their fundamental level. Last time I tried to use them to perform a simple Unicode-specific operation, I simply gave up and written my own solution from scratch.

Originally Posted by kmdv

I would be very cautious saying it just "works". As long as you deal with in-memory strings only it might be fine. However, if you try to use Unicode with any other standard C++ stuff, you are out of luck.

Eh, I mean, I'm oversimplifying. It "mostly works" might be a better term.

Streams: as of now there is completely no standard way to open a file stream using a Unicode path. The only standard constructors are const char* and std::string, which ofc. expect MBCS on Windows. There is no way (no trivial way?) to read UTF-16 stream either. Detecting BOM? Please...

To deal with unicode paths, you have to use wide streams. Use a UTF locale to deal with wide -> narrow conversion (see Boost.Locale).
Reading a UTF16 file is trivial if you keep yourself to english filenames. Just use a narrow stream and read into a string, then convert it however you want.

The C++ support of Unicode is terrible. I find the overall design of code converters and locales overengineered, yet broken at their fundamental level. Last time I tried to use them to perform a simple Unicode-specific operation, I simply gave up and written my own solution from scratch.

I couldn't agree more, unfortunately. The easiest way to do Unicode string processing is to either use a library (best) or convert to UTF16/32 using wchar and use wide character library functions.

Originally Posted by Elysia

To deal with unicode paths, you have to use wide streams.

I'm not sure if we understand each other. I meant that there is no standard way to open a stream given a Unicode path. MSVC comes to the rescue and provides additional overrides of the open() function: https://msdn.microsoft.com/en-us/library/70bb7saf.aspx which take const wchar_t*. These overloads, however, do not exist in standard basic_ifstream/basic_ofstream templates which is a shame (it does not matter whether ifstream/ofstream or wifstream/wstreams are used, as these string types are not templated).

I did not actually know there is no const wchar_t* overload (I figured there was one for std::wstream, but alas). I know Microsoft provides a const wchar_t* overload for narrow streams.
Yeah, that makes it a little harder.

Originally Posted by Abyssion

So when we tell Visual Studio to use a "Multi-Byte Character Set", which encoding does that refer to?

Depends on your locale, so avoid it. Use Unicode only.