cast unsigned char* to (the default) signed char*

Hello folks, long time no see.

I have a quicky...

I need some advice with the way I am dealing with SQLite data extraction functions. The culprit is one - of two, I don't use the other - function that returns an UTF-8 text representation of a single column of the current result row. The prototype is:

Code:

---


const unsigned char* sqlite3_column_text(sqlite3_stmt*, int);

---

My implementation defines "char" as being of type signed char by default. So, in order to put the result of ths function into a std::string I first need to cast it to the appropriate type:

Code:

---

char* result = reinterpret_cast<char*>( sqlite3_column_text(/*...*/) );

---

I'm only partially worried with the resulting data loss since the database in question only uses characters in the acceptable range (english characters only). However two question arise:

1. Can I do better than a reinterpret_cast? Again data loss is no concern.

2. What if data loss becomes a concern? How can I handle this? The problem is that the character 'ó', for instance, is well within the capabilities of a signed char. However the cast alters that and renders latin characters unusable.

> The problem is that the character 'ó', for instance, is well within the capabilities of a signed char.
> However the cast alters that and renders latin characters unusable.
How would the cast affect it?
AFAIK, all the unsigned types have the same number of bits as their signed counterparts. The only difference is one of interpretation. If you then go back to unsigned, you should be back to where you started.

> function that returns an UTF-8 text representation of a single column of the current result row.
http://en.wikipedia.org/wiki/Utf-8
It seems to me that 'ó' would be encoded in two octets. Are you sure that you're not mistaking this encoding for data corruption?

It is possible. I can't check it right now. But will get back to it later today.
My understanding was that any characters above ASCII 128 were not being converted correctly, but I didn't think to look in the database.

Thanks Salem.

UTF-8 and other encodings have little to nothing to do with the signedness of the underlying char type.

also note that you remove const modifier (that you shouldn't for just converting unsigned to signed char)

BTW what about using basic_string with correct CharType template specifier to avoid casting at all?

Another option may be to use a compiler switch to treat char as unsigned by default.

This problem kind of dissapeared on its own and I don't know why. Since things don't just solve by themselves, my guess is that the data on the database was corrupt and was fixed by someone in the meantime since I didn't touch my code.

Thanks everyone for the replies. Meanwhile basic_string seems a reasonable approach no doubt as opossed to casting. thanks.

Quote:

---

Originally Posted by vart

BTW what about using basic_string with correct CharType template specifier to avoid casting at all?

---

This problem resurfaced. Sorry for bringing back the thread. Not too old, I reckon.

I can't seem to figure out how to follow this advise from Vart. I should I go about doing it?

You could try it.

Code:

---

typedef std::basic_string<unsigned char> ucstring;

---

I don't think it will help. The WinAPI wants plain chars, cin and cout are based on plain chars. At some point, you have to convert, and the different strings only delay the moment.

I still think it's an encoding issue and has nothing to do with signedness.

Yup. It doesn't work. Just as you pointed out...

Ok... hmm... let me collect my thoughts... Please bear with me. I have trouble understanding this world of encodings, different text formats and whatmore.

The database text columns hold text in UTF-8 format. The function responsible for returning text from a text field is sqlite3_column_text which returns a null-terminated const unsigned char*.

I expect text in these columns to always contain plain english characters. So one could say, data loss is not a concern. Under these conditions reinterpret_cast is a solution. However I it is screaming undefined behavior, is it not? If, as I think it can happen, some UTF-8 character falls out of the signed char range, reinterpret_cast will not do.

How can I handle this conversion safely?

I gave this some thought and came out with a possible solution involving the excellent boost::numeric cast. However this means iterating through the unsigned char null-terminated array. Probably too expensive of a solution.

> If, as I think it can happen, some UTF-8 character falls out of the signed char range,
Or indeed, all of them.
http://en.wikipedia.org/wiki/UTF-8

Any octet sequence which is encoding an ordinal character >=128 has the top-bit set in all the characters encoding the character.

Nor can you just take a sequence of encoding octets and simply "assign" them to some wide character. There is a degree of bit-fiddling to extract the data bits from the encoding bits.

Quote:

---

Originally Posted by Mario F.

My implementation defines "char" as being of type signed char by default. So, in order to put the result of ths function into a std::string I first need to cast it to the appropriate type:

---

Are you sure? You can create std::strings which hold unsigned chars instead of signed ones.

Code:

---

std::basic_string<unsigned char> utf8_string;

---

In order to do this, you'll have to create a specialization of std::char_traits<unsigned char>, if your environment doesn't already provide one.

It does not. And that's where I would need some help. Vart suggested it also. I'm not sure how to do it.

Quote:

---

Originally Posted by Mario F.

It does not. And that's where I would need some help. Vart suggested it also. I'm not sure how to do it.

---

Does the following link help:

http://www.sgi.com/tech/stl/character_traits.html

You need to define a specialization:

Code:

---

namespace std
{
    template <>
    class char_traits<unsigned char>
    {
        ...
    };
}

---

Look at the sections on that page called "Associated Types" and "Valid Expressions." They define the things you have to provide. The types "char" and "unsigned char" are so similar to each other that you might be able to get away with just copying the char_traits<char> out of the standard library and using it. Hell, even deriving from it might work:

Code:

---

namespace std
{
    template <>
    class char_traits<unsigned char> : public std::char_traits<char>
    {
    public:
        typedef unsigned char char_type;
    };
}

---

Quote:

---

Originally Posted by brewbuck

Does the following link help

---

Absolutely. Thanks. I think I can finally take it from here. I'll try to build my own just for fun and in the process better understand this concept of traits. However, if after enough testing deriving from char_traits<char> does not bring any problems, that will be the final solution.

Most excellent. Thanks again.

Deriving from char_traits<char> doesn't sound like a good idea.

Honestly, in my opinion, C++'s whole character handling is broken beyond repair. That's not much of a consolation to you, of course.

Quote:

---

Originally Posted by CornedBee

Deriving from char_traits<char> doesn't sound like a good idea.

---

I can't think of a reason why not.

Because the members inherited have different signatures than those needed for char_traits<unsigned char>.

Quote:

---

Originally Posted by CornedBee

Because the members inherited have different signatures than those needed for char_traits<unsigned char>.

---

Hrm. Yeah. That sure is stupid. Methods in a "traits" class?

Static methods, yes. That's what the traits class is for.

OK, so by modern terminology it's a policy class.

What exactly are you trying to DO with the string?

If you're using Win32, I recommend you use the conversion function (MultiByteToWideChar) to get a UTF-16 string; the WinAPI can use UTF-16 (it's the native character set on anything Win2K and beyond).

Oh, and you absolutely can't just cast a UTF-8 string and expect it to properly display any characters outside of the ASCII values (0-127).

Every character not in that range occupies a minimum of two bytes, and can be as many as 4 bytes. Any byte value between 0x80 and 0xFF (that is, any byte with the MSB set) is part of a two, three, or four byte character.

Your best bet is to write an actual Unicode app; the Win32 API supports Unicode fully. Use UTF-16 (aka WCHAR, LPWCSTR, etc.) internally within your program, as that's the native character set of Windows and the character set it will accept all strings in, and then convert to and from UTF-8 when accessing the database.

Quote:

---

2. What if data loss becomes a concern? How can I handle this? The problem is that the character 'ó', for instance, is well within the capabilities of a signed char.

---

Maybe. Maybe not. It certainly wouldn't work on my machine; it would not only incorrectly display that character, it would "eat" the following character assuming it to be the second byte of a multibyte Japanese character.

It all depends on what character set the end user has set to default on their computer. Are they using Latin-1? Windows-1251? Windows-1252? Shift-JIS? Big5? EUC-KR? Unless you're specifically overriding the default character encoding, how Windows treats a char* string depends entirely on the language options the end user has set in their Control Panel. Some of those won't even HAVE the letter 'ó', and others will have it but encode it with a different value. Don't make assumptions that the user of your program has the same default code page you do.

That's the beauty of Unicode, and why you should not "work around" it, but rather embrace it. Unicode is universal, it will work on my machine (default code page: Shift-JIS) just as easily as it works on yours.

Probably that's my best bet. As you say, I need to embrace Unicode. And I'm sure that's the best advise one can give.
I'm not prepared or willing to do it for now, though. I have other more emerging concerns; like learning C++ :). And at my age, I no longer find it easy to multitask. I prefer the one thing at a time approach.

Anyways, giving that there's no one-solution-fits-all to this issue. I decided to go for the next best thing; That is, lower my requirements to the bare essentials. I promise I'll sooner or later start to become interested in Unicode under C++. But until then...

A recap:

- SQLite text extracting funtions (there's actually two. But i'm only interested on the UTF-8 one) return a null-terminated const unsigned char*.
- I need to translate this return value into a std::string.
- My system defines char as signed char and doesn't implement basic_string<unsigned char>.
- The code wants to be portable across systems.
- The database is expected to contain only characters in the 0-128 ascii range. Conversion seems painless on this case. But...
- I need to deal with those cases where exceptionally it may not.
- The resulting std::string is expected to be used in every context where basic_string<char> is normally used; streams, for instance.
- Programming for Unicode is not an option at this moment.

Solution 1.
reinterpret_cast<const char*>()

Solution 2.
typedef std::basic_char<unsigned char> utf8_string

Solution 3.
a class implementing:
for each char in utf8 string
. boost::numeric_cast<char>()

Solution 1 is my current solution. The one I wish to not use. Solution 2 is appealing but seems to be more trouble than its worth and not possible to implement with streams. Or is it? Solution 3 is sexy but slow (in some contexts outside the C++ this is actually a good thing).

What would you do?

Implement a converter that does some minimal UTF-8 parsing: it should simply replace all UTF-8 characters by question marks.