get wide character and multibyte character value

Hello everyone,


I need to know the wide character (unicode) and multibyte (UTF-8) values of a character string of czech. I personally know nothing about czech. Is the following approach correct?

1. I use L on the character string and watch memory to get the wide character representation of the character string in little endian form;

2. I change the computer region/language to czech, and use function WideCharToMultiByte, and use CP_ACP as input code page and use the L character string as input to get the output multibyte character string output from parameter lpMultiByteStr.

Is (1) and (2) correct? Any more efficient and smart ways?


thanks in advance,
George

1) You use the term Unicode wrong.
2) You misunderstand what Unicode is all about.
3) You misunderstand the effect of CP_ACP.

So your approach is completely wrong.

Hi CornedBee,


What is your solution? :-)

Quote:

---

Originally Posted by CornedBee

1) You use the term Unicode wrong.
2) You misunderstand what Unicode is all about.
3) You misunderstand the effect of CP_ACP.

So your approach is completely wrong.

---

regards,
George

You can read the documentation of WideCharToMultiByte again.

You can read more about Unicode.

Hi CornedBee,


Could you provide some hints please? I have used WideCharToMultiByte before for a couple of times, and it converts wide character to multibyte character well as designed and as I stated. On Windows 32-bit platforms, it convers UTF-16 to UTF-8.

Which point I am wrong? :-)

Quote:

---

Originally Posted by CornedBee

You can read the documentation of WideCharToMultiByte again.

You can read more about Unicode.

---

regards,
George

1) UTF-16 is not Unicode. Unicode is a combination of a character set (the Unicode character set, equivalent to the universal character set (UCS) of ISO-10046) and algorithms for comparing, sorting and casing that take into account the peculiarities of scripts and language from all over the world. UTF-16 and UTF-8 are encodings of the codepoints of this character sets into a byte stream. (More correctly, UTF-16 is an encoding to 16-bit units, and UTF-16LE and UTF-16BE define further serialization to byte streams.) Win32 programmers and the API docs are frighteningly careless in mixing up the two terms. It's as if UTF-16 was Unicode to Win32 programmers, which is stupid.

2) Given that Unicode attempts to cover all languages at the same time, doing anything Czech-specific in the conversion of UTF-16 to UTF-8 (which is just an encoding conversion) is an absurd idea. The conversion means going through the UTF-16 string, decoding the code points (1 16-bit unit for most characters, 2 for surrogate pairs), and encoding them as UTF-8 (between 1 and 4 bytes) into the target buffer. There is absolutely nothing language-specific about this.

3) CP_ACP uses the currently defined Ansi Code Page, as understood by the Win32 API. This code page is typically set to 437 (US) or 850 (Western European). By setting the language to Czech, you probably set the ACP to 851 (Eastern European). In any case, you won't get UTF-8.
More to the point, it's absolutely disallowed to set the ACP to UTF-8, even though it's apparently possible. However, many API functions are not able to handle multi-byte characters longer than 2 bytes, so they can't work with UTF-8. See Michael Kaplan's multitude of comments on the subject.
So to convert to UTF-8, you must pass CP_UTF8. The MSDN page could have told you that, but apparently you are banned from MSDN, because otherwise you surely would have followed my advice and known that already.

Thanks CornedBee,


I got more insight. I want to confirm with you that the wide character on Windows is UTF-16, right? So, I could use L on the character string to get its UTF-16 (wide character) binary value?

(I am in debug mode and monitor the L character string buffer). Is it a correct approach to get the UTF-16 binary value of the character?

Quote:

---

Originally Posted by CornedBee

1) UTF-16 is not Unicode. Unicode is a combination of a character set (the Unicode character set, equivalent to the universal character set (UCS) of ISO-10046) and algorithms for comparing, sorting and casing that take into account the peculiarities of scripts and language from all over the world. UTF-16 and UTF-8 are encodings of the codepoints of this character sets into a byte stream. (More correctly, UTF-16 is an encoding to 16-bit units, and UTF-16LE and UTF-16BE define further serialization to byte streams.) Win32 programmers and the API docs are frighteningly careless in mixing up the two terms. It's as if UTF-16 was Unicode to Win32 programmers, which is stupid.

2) Given that Unicode attempts to cover all languages at the same time, doing anything Czech-specific in the conversion of UTF-16 to UTF-8 (which is just an encoding conversion) is an absurd idea. The conversion means going through the UTF-16 string, decoding the code points (1 16-bit unit for most characters, 2 for surrogate pairs), and encoding them as UTF-8 (between 1 and 4 bytes) into the target buffer. There is absolutely nothing language-specific about this.

3) CP_ACP uses the currently defined Ansi Code Page, as understood by the Win32 API. This code page is typically set to 437 (US) or 850 (Western European). By setting the language to Czech, you probably set the ACP to 851 (Eastern European). In any case, you won't get UTF-8.
More to the point, it's absolutely disallowed to set the ACP to UTF-8, even though it's apparently possible. However, many API functions are not able to handle multi-byte characters longer than 2 bytes, so they can't work with UTF-8. See Michael Kaplan's multitude of comments on the subject.
So to convert to UTF-8, you must pass CP_UTF8. The MSDN page could have told you that, but apparently you are banned from MSDN, because otherwise you surely would have followed my advice and known that already.

---

regards,
George

Quote:

---

Originally Posted by George2

I got more insight. I want to confirm with you that the wide character on Windows is UTF-16, right?

---

Correct. It should also be noted that this is a violation of the C++ standard, albeit a minor one.

Quote:

---

So, I could use L on the character string to get its UTF-16 (wide character) binary value?

---

You use L on the character string to make it a wide literal. If you subsequently look at the memory, you'll find the UTF-16 representation, yes. (UTF-16LE, to be specific.)

Thanks for your confirmation CornedBee!


1.

Quote:

---

Originally Posted by CornedBee

Correct. It should also be noted that this is a violation of the C++ standard, albeit a minor one.

---

What the violation do you mean?

2.

Quote:

---

Originally Posted by CornedBee

You use L on the character string to make it a wide literal. If you subsequently look at the memory, you'll find the UTF-16 representation, yes. (UTF-16LE, to be specific.)

---

When I use wide character to multibyte character conversion, should I use UTF-8 code page or some special Czech code page?

I am not sure for everyong Czech character encoded in wide character, it could be encoded into UTF-8?


regards,
George

Quote:

---

Originally Posted by George2

What the violation do you mean?

---

The standard requires that every character of the extended execution character set can be represented in a single wide character. Windows claims that its eecs is Unicode, but the UTF-16 of wchar_t is a variable-width encoding.

Quote:

---

When I use wide character to multibyte character conversion, should I use UTF-8 code page or some special Czech code page?

---

That depends on what you need the multibyte string for.

Quote:

---

I am not sure for everyong Czech character encoded in wide character, it could be encoded into UTF-8?

---

UTF-8 and UTF-16 are both complete. There is no Unicode character that cannot be represented in them.

Thanks CornedBee,

Quote:

---

Originally Posted by CornedBee

The standard requires that every character of the extended execution character set can be represented in a single wide character. Windows claims that its eecs is Unicode, but the UTF-16 of wchar_t is a variable-width encoding.

---

Correct me if I am wrong. I think on Windows, wide character or UTF-16 is implemented in a way which one character is always 2 bytes -- sizeof (unsigned short). So it should be fixed-width encoding.

Why you say it is "variable-width encoding"?


regards,
George

CornedBee just mentioned that wchar_t is variable width, just like the typical char, the 1 byte multi-string.
Details are unknown to me, but unless wchar_t is 4 bytes (the number of bytes needed to represent all characters in fixed-width), it must be variable-width, or have limited support for all the characters. And that does somehow not look like unicode to me.

Thanks Elysia,


You mean wchar_t is always 2 bytes and the 2-byte range is not enough to represent all the values in unicode character and 4-byte range is required?

Quote:

---

Originally Posted by Elysia

CornedBee just mentioned that wchar_t is variable width, just like the typical char, the 1 byte multi-string.
Details are unknown to me, but unless wchar_t is 4 bytes (the number of bytes needed to represent all characters in fixed-width), it must be variable-width, or have limited support for all the characters. And that does somehow not look like unicode to me.

---

regards,
George

Yes, unicode represents far more than 65536 characters, so 2 bytes are not enough.

Thanks Elysia,


I got your point now. If I remember correctly, Windows only implement a part of unicode called UCS-2.

Quote:

---

Originally Posted by Elysia

Yes, unicode represents far more than 65536 characters, so 2 bytes are not enough.

---

regards,
George

I'm pretty sure Windows claims to support the full Unicode range.

And yeah, UTF-16 is a variable-width encoding. The characters from Unicode's basic multilingual plane are 2 bytes large, all others are represented with surrogate pairs, two 16-bit units, and are therefore 4 bytes large.

Hi CornedBee,


Code is more powerful than descriptions. Could you let us know which character on Windows will be represented as 4 bytes please, under wide character format -- i.e. L with string literal. :-)

Sorry for my challenging question.

Quote:

---

Originally Posted by CornedBee

I'm pretty sure Windows claims to support the full Unicode range.

And yeah, UTF-16 is a variable-width encoding. The characters from Unicode's basic multilingual plane are 2 bytes large, all others are represented with surrogate pairs, two 16-bit units, and are therefore 4 bytes large.

---

regards,
George

You're wrong. In this case, code is not more powerful that descriptions.

Here's a list of tables:
http://en.wikipedia.org/wiki/Mapping...ode_characters
Scroll all the way down.


Here's some:
http://www.decodeunicode.org/en/cjk_...hs_extension_b

A colleague of mine has the Unicode 5.0 standard document - it's a book about 3"/7.5 cm thick - I haven't looked inside it. So I expect that any "few pages web-site" or "a few lines of code" will not describe this problem properly.

--
Mats

Sorry CornedBee,


My question is confusing. What I mean is not UTF-16, but Windows' implementation/representation of UTF-16 -- using wchar_t and it is type defined to unsigned short -- onoy 2 bytes.

My question should be, Windows uses only 2 bytes to represent UTF-16 and can not represent all of the UTF-16, right?

BTW: what you provide is the general theory of UTF-16, but what I am interested is how Windows represent and limitation of Windows. How could an unsigned short represent 4 bytes to cover all characters? :-)

Quote:

---

Originally Posted by CornedBee

You're wrong. In this case, code is not more powerful that descriptions.

Here's a list of tables:
http://en.wikipedia.org/wiki/Mapping...ode_characters
Scroll all the way down.


Here's some:
http://www.decodeunicode.org/en/cjk_...hs_extension_b

---

regards,
George

You are confused. UTF-16 is a variable width encoding. That means it can be longer if necessary, just like UTF-8. A 16-bit value cannot represent all unicode characters (which is around 100.000 iirc), so it's variable width. Usually, the first byte will tell how long the character value is (is it 1, 2 or 3 bytes?). So the application processing the unicode string will interpret the first or second byte (I have no idea how UTF-16 works), and then read the applicable amount of characters that is needed to represent the entire unicode value.
Does that make any sense?

Thanks Elysia,


It makes senses about general UTF-16 theory.

But on Windows, as we know, wide character (UTF-16LE form) is represented by wchar_t, unsigned short, whose length is 2-byte.

Do you think on Windows 2-byte is enough to represent all the UTF-16 encoded character?

Quote:

---

Originally Posted by Elysia

You are confused. UTF-16 is a variable width encoding. That means it can be longer if necessary, just like UTF-8. A 16-bit value cannot represent all unicode characters (which is around 100.000 iirc), so it's variable width. Usually, the first byte will tell how long the character value is (is it 1, 2 or 3 bytes?). So the application processing the unicode string will interpret the first or second byte (I have no idea how UTF-16 works), and then read the applicable amount of characters that is needed to represent the entire unicode value.
Does that make any sense?

---

regards,
George

*bangs head against desk*

Look, just because a single unit is 2 bytes, doesn't mean that every character is 2 bytes. Surrogate pairs are two units (4 bytes) that together form a character.
That's why it's called a variable-width encoding.

Why would 2 bytes magically be able to hold more on Windows? What kind of absurd thinking is that?

Thanks CornedBee,


I think you mean using two unsigned short (4 bytes) to represent a character in UTF-16LE encoding, right?

As far as I know, Windows does not. However, I am not an expert, and it is appreciated if you could point out some sample characters which on Windows requires 4 bytes other than 2 bytes in the form of wide (UTF-16LE) character.

Quote:

---

Originally Posted by CornedBee

*bangs head against desk*

Look, just because a single unit is 2 bytes, doesn't mean that every character is 2 bytes. Surrogate pairs are two units (4 bytes) that together form a character.
That's why it's called a variable-width encoding.

Why would 2 bytes magically be able to hold more on Windows? What kind of absurd thinking is that?

---

regards,
George

You're not getting the point. It's variable width. That means a character can be 1, 2, or even 4 bytes. It all depends on what type of character it's trying to represent!
If all characters were 4 bytes, then it would be fixed width.

Hi Elysia,


I have not made myself understood.

Your point is, UTF-16 encoded character could be represented either by 2-bytes or by 4-bytes format (it is so called variable length). The "16" means the encoding will be represented by 16-bit basis -- 16 bits, 32-bits, etc.

My point is, on Windows UTF-16 encoding seems to have limitations (Windows only uses 2-byte subset of UTF-16LE), and Windows is not able to represent UTF-16 character encoding in 4 bytes, so only a subset of UTF-16 encoding is supported. Do you agree?

I am not sure whether this time I have made myself understood? :-)

Quote:

---

Originally Posted by Elysia

You're not getting the point. It's variable width. That means a character can be 1, 2, or even 4 bytes. It all depends on what type of character it's trying to represent!
If all characters were 4 bytes, then it would be fixed width.

---

regards,
George

Windows has basic support for surrogate pairs since Windows 2000.

Thanks CornedBee,


It should be my memory error. :-)

Quote:

---

Originally Posted by CornedBee

Windows has basic support for surrogate pairs since Windows 2000.

---

regards,
George