Printable View
Hi, everyone.
I met some problem when processing some text in Chinese.The file has been preprocessed so that the words are separated from each other,so I want to retrieve each word and figure out the frequency of the words in the file.Since Chinese words often consists of several characters while char type in C language only holds one Byte,I wonder how I can handle it.
You need code that can handle UTF (Unicode) - "proper" UTF is 32-bit per character, but to save space, it's often stored in UTF-8 format, which consists of one or more 8-bit characters, where the highest bit indicates the number of bytes to follow (so 0..127 is single byte "ascii" characters, 128-255 is the first part of a multibyte set).
--
Mats
Thank you for your quick reply,but I am not so familiar with the code part,can you kindly explain it more specifically.How can I get the word in unicode?
I'm not quite sure what to explain - to read non-european text, you will need to read multiple characters, and then decode it into a Unicode character code - which is a 32-bit value. You will need to know how the original content is encoded (e.g. UTF-8, UTF-16 or UTF-32).
--
Mats
Thanks anyway! I will try it.
We can help you more - but we need to know how the file is encoded. Once we know how it is encoded, then we'll know how to look for the word separator. Then you could just use memcmp to determine word equality.
gg