Printable View
I figured this might as well be a fresh thread. Does anyone know if I can expect tolower() to perform locale adjusted conversions? I don't have any non-ascii alphabets handy to test this...Ronix has pointed out:
but Codeplug elaborated:Quote:
Originally Posted by Ronix
Almost implies to me that tolower() itself is locale dependant, which would be great since I can't use strcasestr. The GNU manual has no comment and neither did a couple of quick googles.Quote:
Originally Posted by Codeplug
The C standard says:
The blurb for towlower is almost identical.Quote:
If the argument is a character for which isupper is true and there are one or more corresponding characters, as specified by the current locale, for which islower is true, the tolower function returns one of the corresponding characters (always the same one for any given locale); otherwise, the argument is returned unchanged.
Phew -- except how could this apply to multi-byte utf8 chars? Which I imagine is all those little accented letters.Quote:
Originally Posted by tabstop
That's why I mentioned towlower -- wouldn't you be using wide characters for utf8?Quote:
Originally Posted by MK27
I believe the parameter for tolower / toupper must be in the range [0 .. UCHAR_MAX), which would make it useless for multi-byte characters.Quote:
Originally Posted by MK27
The standard says nothing about the encoding of wide characters. However, on most Unix platforms, wchar_t represents UTF-32 code points (a property you can check by veryfing the existance of the __STDC_ISO_10646__ macro), whereas on windows, it usually represents UCS-2 or UTF-16. So one could use towlower / towupper and then convert back to UTF-8 according to the host platform.Quote:
Originally Posted by tabstop
Oh I'm not using them at all. I thought a wide character was one that actually occupied more screen space. That's fine, since the character count will be the same and I imagine "wide character" alphabets (like ideograms) do not really use upper and lower case. Altho that begs the question: what is towlower for?Quote:
Originally Posted by tabstop
But the romance languages, etc contain a lot of "modified" ascii characters (an e with an accent, etc) which I presume, since they are not part of ASCII, must be UTF-8, and can be capitalized (an E with an accent).
They wouldn't be UTF-8 if they were wide chars, they's be UTF-16 or UTF-32...Quote:
Originally Posted by tabstop
It wouldn't be worth having a function for it if it was not locale-specific.Quote:
Originally Posted by MK27
At any rate, I think you're giving yourself an unnecessary headache with this UTF-8 thing. UTF-8 is intended primarily as a transfer encoding. For data processing, it is much easier to deal with unescaped UTF-16.
IMHO, the first thing you should do with the UTF-8 data is translate it to UTF-16, then life will get a lot easier.
(There is more than one UTF-16 encoding, and some of them do have escapes, but you are far less likely to encounter them)
Fair enough then (apparently character handling needs to be my next project). For whatever reason, I was thinking the shift status (or whatever it's called) would be used for this sort of thing. Maybe not.Quote:
Originally Posted by cpjust
I hope so now. I like my own version of strcasestr. However, I don't see how tolower() could possibly work on more than one byte, so I will have to come back to this issue. If strcasestr itself is locale safe, then perhaps it does not use/no longer uses tolower().Quote:
Originally Posted by brewbuck
Technically, it's postscript which has been filtered through col -b to produce plain text. That means that the single quote normally used in conjugations is replaced with the more proper apostrophe, using I am pretty sure: unicode, utf8.Quote:
At any rate, I think you're giving yourself an unnecessary headache with this UTF-8 thing. UTF-8 is intended primarily as a transfer encoding.
I have been consulting the "UTF-8 and Unicode FAQ for Unix/Linux". which implies to me that unicode is a singular standard. It also say that it is in the process of becoming the defacto standard on unix/linux machines, and that some standard commands like ls and applications like vi and emacs had to be re-written because of this.
Ronix is right. I was wrong when I said that strcasestr() would only work if the current locale (LC_CTYPE) is UTF8.Quote:
Originally Posted by MK27
The manual says:
> Like strcasecmp, it is locale dependent how uppercase and lowercase characters are related.
Looking at the source for strcasestr(), it processes "characters" one byte at a time - making strcasestr() useless with UTF8. However, toupper/tolower are affected by the LC_CTYPE of the current locale. Which means tolower and toupper only really work for single-byte code pages.
If the locale's LC_CTYPE is UTF8, you would need to do a multi-byte to wide conversion. Then you can use the corresponding wide string functions like wcsstr() instead of strstr() - or towupper() instead of toupper().
There doesn't seem to be a wide version of strcasestr().
If your data is not locale dependent (you know it'll always be UTF8), then you use the iconv library to convert to wide strings (UTF32 on *nix).
gg
I have perhaps misunderstood what an E with an accent could be. So those letters would be what -- a single byte outside of the ascii range?Quote:
However, toupper/tolower are affected by the LC_CTYPE of the current locale. Which means tolower and toupper only really work for single-byte code pages.
Do they have upper case letters in Arabic?
You're getting into hairy territory. An E with an acute accent, for instance, has more than one codepoint in Unicode. These characters are distinct, even though they look exactly the same. The definition of a character is more than just what it looks like.Quote:
Originally Posted by MK27
There was the other guy today in the forum -- "exception handling" or something like that -- who had exactly that, an E with an accent. In UTF-8, according to the links in that thread, it's two bytes: C3 88.Quote:
Originally Posted by MK27
Thanks. I honestly had not noticed that. I now think my project is haunted tho -- yesterday I realized that I was using an array to hold the find coordinates, and that I'd set the array size to 666 arbitrarily, which limited the number of finds. But how's this for an illusory problem: I sat there for twenty minutes after enlarging the array trying to figure out where else that limit was applied, because I keep getting the same result.Quote:
Originally Posted by tabstop
Then I realized that the source code (which I was using as input) really did contain "gtk" (case insensitive) 666 times.
>> An E with an acute accent, for instance, has more than one codepoint in Unicode
That's mixing terms a bit. The term "code point" in used to describe a single unit of storage for a particular Unicode encoding. So for UTF8 each byte is code point, for UTF16 each word is a code point etc... It can take multiple code points to represent a single Unicode character.
>> So those letters would be what -- a single byte outside of the ascii range?
There are various "code pages" that map an integral value to a character glyph. A single-byte code page would contain a maximum of 256 mappings. For any code page (that I know of) the characters between 0 and 0x7F (127) are always the same. These are you basic "ascii" characters.
To illustrate, here's a great site that indexes many character glyphs to the values they have under various code pages. Here is "Latin Capital Letter E With Acute" - http://www.tachyonsoft.com/uc0000.htm#U00C9
As you see, it can the following single-byte values, depending on code page: 0x90, 0xC9, 0x4A, 0x71, or 0xE0.
Under Unicode (UTF32 or UCS-4) it's U+000000C9.
Encoded with UTF8 it's the two bytes 0xC3 0x89.
Even though UTF8 can use multiple bytes to encode a single character, don't confuse it with a multi-byte code page. UTF8 is just an 8bit encoding of Unicode characters. (So you can kinda think of Unicode as a really big, 32bit "code page".)
Here are examples of multi-byte code pages in Windows: http://www.microsoft.com/globaldev/reference/WinCP.mspx
They call is "DBCS" for obvious reasons.
gg
hmmm....