Hi,
I made a program that inputs a text file and analyses it (line by line with fgets in a loop, main string functions used are strlen, strtok, strchar, strcpy, strcmp). When the input file is encoded in Unicode, it crashes at some point while reading the file. Interestengly, it doesn't crash if the input file is in UTF-8 encoding.
How can I check if the file encoding is ANSI as expected ?
Best wishes,
Desomond
It's strange that it crashes. I could expect it to "behave strangely", but it shouldn't crash. Can you post the code, perhaps?
It appears that the text starts with FF FE if it's unicode UCS-2 [16-bit Unicode].
--
Mats
Compilers can produce warnings - make the compiler programmers happy: Use them!
Please don't PM me for help - and no, I don't do help over instant messengers.
Ofcourse. It's quite lenghty so I'll post a link where I uploaded it instead:Originally Posted by matsp
http://www.andresraieste.pri.ee/ttu/...for2/CPeatus.c
Beware though, as the code is commented in my native language - estonian.
An example of input file it's supposed to eat:
http://www.andresraieste.pri.ee/ttu/...peatusdemo.txt
And executable (compiled under MinGW on Vista 64bit):
http://www.andresraieste.pri.ee/ttu/...r2/CPeatus.exe
The program is supposed to analyze a cerain text file with keys and values (given as time) and calculate a bit with those..
I have no ideas why it crashes.. What do you think ?
Best wishes, Desmond
Last edited by desmond5; 03-10-2008 at 10:33 AM.
Not sure.
--
Mats
Compilers can produce warnings - make the compiler programmers happy: Use them!
Please don't PM me for help - and no, I don't do help over instant messengers.
I am trying to reproduce the error by cutting out stuff from the program that aren't related to it. Maybe that will help.
Always a good plan. I didn't spend many seconds looking at it. The fact that I can just about GUESS what the different names mean and what the comments say could just as well be written in invisible ink for the use I have for them. I'm sure I could figure it out, but it's a bit harder when you don't get the language...
--
Mats
Compilers can produce warnings - make the compiler programmers happy: Use them!
Please don't PM me for help - and no, I don't do help over instant messengers.
You're probably banging into null bytes. UTF-8 is specifically designed not to use the value 0, because it would break C string handling.
Impossible to tell.How can I check if the file encoding is ANSI as expected ?
No question about that. I shrinked the program a lot and uploaded the source and the input file that produces the error here: http://www.andresraieste.pri.ee/ttu/IABB/err/
So..the workaround would be ? I need check if there are null bytes in the program and fail if there are any ? There has to be some workaround.
The solution is to use actual wide character processing functions.
How about checking byte order mark ? Is that doable ?
If you know there is a byte order mark, then you've already decided that it's Unicode, right?
The file could be in some encoding from Jupiter and it just happens to start with a few bytes that look like the Unicode byte order mark.
I think that if you see a byte order mark, the best thing is probably to assume it's Unicode, just be aware that it's not proof of anything. The only way to know for sure what the encoding is, is for somebody to tell you what it is.
I just wanted to produce a correct error message instead of the program crashing. How should I proceed with that, then ?
Well, the crash is a separate issue. Even if you wrongly guess the encoding, the presence of a zero byte shouldn't cause a crash. But since I haven't seen your code, I can't really guess what's actually wrong.
I'd go ahead and assume that if you see a Unicode byte order marker, then the encoding is Unicode -- otherwise assume ASCII. But you still need to be safe against the crash in either case. So first debug that issue and don't pollute your thinking with the Unicode vs. ASCII stuff.
Also see http://blogs.msdn.com/oldnewthing/ar.../24/95235.aspx
and http://blogs.msdn.com/oldnewthing/ar...7/2158334.aspx
for some problems with text file encoding markers...and the problems with them.
