fread + non-english characters

A question.
Consider the code:

Code:

	FILE* f = fopen("settings.xml", "r");
	char buf[10240];
	fread(buf, 10240, 1, f);
	fclose(f);

This would be used to read in an entire file. The problem is that when there are international characters, they become all messed up in the reading process.
Also, this is not my code. The tinyxml C++ library reads files this way and obviously it's causing me problems!

So I was wondering what the best way to fix this would be?
Preferably, I don't want to rewrite the tinyxml code, so workarounds are welcome.
As a last resort, I may rewrite parts of it, but then the question would be what the appropriate way to read files with international characters would be?
The file should be in UTF-8 (no leading bytes).

I know nothing about different char sets, but using c shouldn't you at least be reading as binary? rb, not r?

How do they get messed up? UTF-8 has no endianness issues, so fread should work just fine.

I think it's more likely that they are messed up in the output process since you do have all the data, just not how the display mechanism is expecting it.

One way of doing it, is while characters are being read in, you examine each byte to see if it falls within range of certain values. This tells you if the following byte is also part of the character or a new character itself. Then you can store each character itself in a wchar array so you have a more direct representation of the characters and not just the bytes in the file.

I suppose I could try this to find out if it works, but it's more fun to make you do it: you've got all the bytes in your buffer. Could you put that in a stringstream and then read out of that stream in the usual way?

The data is somehow changed. I know this is the case, as when it reads "Våglära" and I compare it to "Våglära", I don't get the same match. Ie, they're not equal.
Further research suggests:
- The data is "corrupt" inside the internal FILE structure.
- fgets does not read data correctly.
- Using unsigned char doesn't help.

If you are getting corrupt data by reading in, I think Oldman47 is correct when suggesting you use read binary, otherwise it's doing some wonky business assuming ASCII text.

Edit: Otherwise, pass the proper encoding to fopen, but that will require use of wchar stuff to store things.

Just out of curiosity, have you tried setting the `TiXmlEncoding' parameter to `TIXML_ENCODING_UTF8'?

How are you comparing those strings?

Soma

On Windows you'll get CRLF->LF conversions (when not opened in binary mode). And terminal-EOF characters are interpreted as EOF (that's Ctrl-D on *nix and Ctrl-Z on Windows).

Other than that, the data shouldn't be "changed" or interpreted.

>> and I compare it to "Våglära", I don't get the same match
It's not hard to get this wrong. Especially when you put non-basic character literals directly in your source code - at which point you're in implementation defined territory.

gg

Originally Posted by syzygy

If you are getting corrupt data by reading in, I think Oldman47 is correct when suggesting you use read binary, otherwise it's doing some wonky business assuming ASCII text.

Edit: Otherwise, pass the proper encoding to fopen, but that will require use of wchar stuff to store things.

Just reading binary into char doesn't help. Just tried that. Will try more later.

Originally Posted by phantomotap

Just out of curiosity, have you tried setting the `TiXmlEncoding' parameter to `TIXML_ENCODING_UTF8'?

Soma

I don't know where that parameter would do.
I have tried debugging the TinyXML code, however, and it does detect and set UTF-8 encoding properly.

>> and I compare it to "Våglära", I don't get the same match
It's not hard to get this wrong. Especially when you put non-basic character literals directly in your source code - at which point you're in implementation defined territory.

Yes, I'm aware. I would use Unicode (the rest of the project is), but unfortunately, by some misfortune, TinyXML doesn't support wchar_t.
I'm not an expert on the whole codepage field, though, so that's why I'm asking.

I would use Unicode (the rest of the project is), but unfortunately, by some misfortune, TinyXML doesn't support wchar_t.

^_^

Wow. You are squarely in Microsoft's pocket.

What you are missing (or what you failed to answer): the compiler may store the string in a different UTF8 binary representation than what "TinyXML" produces. The string may be seven bytes from one source and nine bytes (I think) from another. Again, how are you comparing these strings? What, exactly, do you mean by "not equal"?

Soma

Apparently, the comparison is done via strcmp (again, what TinyXML uses). So basically I read some data from the XML-file and compare it to a directory name (that should be encoded in Unicode).
I believe I have found the issue. It's probably ANSI/UTF-8 mismatch. If I save the file in ANSI encoding, it looks "right". If I save it in UTF-8 encoding, it looks "wrong".
This might be because VS is interpreting char as ANSI. But I did notice the same thing when I examined it in a hex editor.
My main theory so far is that TinyXML reads raw UTF-8 data, yet strings (char*) are encoded in ANSI. I have to verify this later.

I have been able to make it work when saving the file as ANSI (yet stating UTF-8 encoding in the xml header).
Apparently, 'å' and L'å' yields the same "magic number", so this works (since I am internally using wchar_t for my strings and hence has to convert to char when comparing to or from data from the xml file).
However, I feel like I am out on deep water here.

This might be because VS is interpreting char as ANSI.

Is it a nice pocket? ^_^v

Apparently, the comparison is done via strcmp.

O_o

If "TinyXML" uses `strcmp' it is probably safe to assume that the library does not touch the binary representation.

I'd advise you to always encode a character using the hexadecimal escape sequences for any value that isn't in the seven bit `ASCII' range.

You need to check that the string stored in the executable has the same binary representation that you have in the "XML" file.

Soma

I've just confirmed that converting L"Våglära" to UTF-8 creates an equal string that was stored inside the file.
I suppose I can use std::string to hold the UTF-8, and it should work. I just cannot expect to get any sensible output. That's my next test.

I've just confirmed that converting L"Våglära" to UTF-8 creates an equal string that was stored inside the file.

O_o

May we assume you are using the `MultiByteToWideChar' API?

You should be careful with that function; it is a nightmare.

I just cannot expect to get any sensible output. That's my next test.

I can't help but think that you are going about this the wrong way.

True `UTF8' strings have no embedded nulls. You should get what you set.

Soma