Thread: Chinese unicode in console and getline wont work

Here's my github:

GitHub - jeremyduncanartificialintelligence/musical-artificial-intelligence

When you run the program it works fine for English, but for Chinese there's one big problem, and one small problem.

The big problem is getting the input sentence using getline and if it's a Chinese character, writing that to the file_out.txt.

Once I get that in there I can test if my hard coded Chinese characters work in my program like it does for English.

Smaller problem, the wcout might not work with both English and Chinese but I have to get the text into the file_out.txt first to run the program and take it from there is wcout is messing up somehow.

I tried a bunch of different things to get the getline to take wchar Chinese unicode, but it doesn't work. Can you help me please.

Here is my test code

Code:

#include <iostream>
#include <fstream>
#include <cstdlib> // exit()

using namespace std;

int main()
{
    // empty files
    ofstream fout("file_out.txt");
    fout.close();

    // get user input.
    ofstream holdInputSentence("file_out.txt");
    if (!holdInputSentence) { cerr<<"file error\n"; exit(1); }

    char inputSentence[4096];

    cout << "Enter your sentence, end it with a period: ";
    cin.getline (inputSentence,4096);
    holdInputSentence << inputSentence;
    holdInputSentence.close();

    ifstream read_the_counter_file("file_out.txt");
    string read_text = "";

    cout <<  "\n";

    while(read_the_counter_file >> read_text)
    {
        wcout << wstring(read_text.begin(), read_text.end()) << " ";
    }

    read_the_counter_file.close();

    cout <<  "\n\n";

    // I pause the console window so you can read the screen results
    system ("pause");
}

text to copy and paste into program when testing it:

Code:

嗄 1 x #

Have you tried using only the wide versions of all of those streams, ie not mixing things like cin and wcin?

Jim

Yes I tried with only the Chinese character all I see in the console is a question mark in wcout and a square box in the input:

I typed in: 嗄

Code:

Enter your sentence, end it with a period: ? 

?

Press any key to continue . . .

The 1 x # works fine

Code:

Enter your sentence, end it with a period: 1 x #

1 x #

Press any key to continue . . .

Here's with both Chinese character and ascii

Code:

Enter your sentence, end it with a period: ? 1 x #

? 1 x #

Press any key to continue . . .

My new code that prints to screen, in vs 2010 express c++:

Code:

#include <fcntl.h>
#include <io.h>
#include <cstdio>
#include <cwchar>
#include <iostream>
#include <fstream>
#include <locale>
#include <cstdlib> // exit()

using namespace std;
int main()
{
    _setmode(_fileno(stdout), _O_U16TEXT);
    
	// empty files
    ofstream fout("file_out.txt");
    fout.close();
 
    // get user input.
    ofstream holdInputSentence("file_out.txt");
    if (!holdInputSentence) { cerr<<"file error\n"; exit(1); }
 
    char inputSentence[4096];
 
    wcout << L"Enter your sentence, end it with a period: ";

	cin.getline (inputSentence,4096);
    holdInputSentence << inputSentence;
    holdInputSentence.close();
 
    wifstream read_the_counter_file("file_out.txt");
    wstring read_text;
 
    wcout <<  L"\n";
 
    while(read_the_counter_file >> read_text)
    {
        wcout << read_text << " ";
    }
 
    read_the_counter_file.close();
 
    wcout <<  L"\n\n";
 
    // I pause the console window so you can read the screen results
    system ("pause");
}

My output for english works when I use raster font, but that asian font gets english all different symbols like hearts.

Here is my output for my test Chinese font using raster font:

Code:

Enter your sentence, end it with a period: 绽湛栈蘸菚颤张章彰璋漳樟蟑鄣獐嫜餦麞
粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

?´ï?ÅÙ???³¹¹ü¼ýºs¼ÌÁ..ähº¼ák??ã^??ö¡´x?ÉSïA?»Ù¤V?¥M

Press any key to continue . . .

and using the asian font:

Code:

Enter your sentence, end it with a period: 绽湛栈蘸菚颤张章彰璋漳樟蟑鄣獐嫜餦麞
粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

?´ï?ÅÙ???³¹¹ü¼ýºs¼ÌÁ..ähº¼ák??ã^??ö¡´x?ÉSïA?»Ù¤V?¥M

Press any key to continue . . .

when I tried with wcin this code I got numbers instead of text;

Code:

wchar_t inputSentence[4096];
 
    cout << "Enter your sentence, end it with a period: ";
    wcin.getline (inputSentence,4096);

my code:

Code:

#include <fcntl.h>
#include <io.h>
#include <cstdio>
#include <cwchar>
#include <iostream>
#include <fstream>
#include <locale>
#include <cstdlib> // exit()
 
using namespace std;
int main()
{
    // empty files
    ofstream fout("file_out.txt");
    fout.close();
  
    // get user input.
    ofstream holdInputSentence("file_out.txt");
    if (!holdInputSentence) { cerr<<"file error\n"; exit(1); }
  
    char inputSentence[4096];
  
    wcout << L"Enter your sentence, end it with a period: ";
 
    cin.getline (inputSentence,4096);
    holdInputSentence << inputSentence;
    holdInputSentence.close();
  
	// display user input
    wifstream read_the_counter_file("file_out.txt");
    wstring read_text;
  
    wcout <<  L"\n";
  
    while(read_the_counter_file >> read_text)
    {
        wcout << read_text << " ";
    }
  
    read_the_counter_file.close();
  
    wcout <<  L"\n\ncopying file to new file then display new file contents below\n\n";

	// empty files
    ofstream fout4("file_out.txt");
    fout4.close();

	// copy file

	wofstream fout3("file_out.txt", wfstream::app);
	wfstream read_the_file("cn_font.txt" , ios::in);
    wstring read_text2;
 
    while(read_the_file >> read_text2)
    {
        fout3 << read_text2;
    }
 
    read_the_file.close();
    fout3.close();

	// display the copy file

	wifstream fin("file_out.txt");
    wstring read_text3;
 
    while(getline(fin, read_text3))
    {
        wcout << read_text3;
    }
 
    fin.close();

	wcout <<  L"\n\n";
  
    // I pause the console window so you can read the screen results
    system ("pause");
}

my cn_font file contents:

Code:

绽湛栈蘸菚颤张章彰璋漳樟蟑鄣獐嫜餦麞粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

What I copy and paste into the console when testing the program:

Code:

嗄 1 x #

my program results:

Code:

Enter your sentence, end it with a period: 嗄 1 x #

嗄 1 x #

copying file to new file then display new file contents below

嚜輻遢皝猹?═撘蝡敶啁?瞍單??煾??擗阡?蝎駁?阮?鍫隞厰?瘨券?銝?隞


Press any key to continue . . .

whats in file_out.txt after the program runs:

Code:

绽湛蘸菚颤彰璋漳樟蟑鄣獐嫜餦麞粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

I am using visual studio express c++ 2010, the console is using the asian font.
In the console options tab it says I'm using codepage 950 "Traditional Chinese big 5".

Of note is the Chinese characters I'm using are simplified, not traditional so some fonts aren't going to display for that reason.

How I was getting weird characters was I was using the two versions of unicode instead of utf-8.
When the files were unicode or unicode big endiad the read file looked like some weird fonts that aren't even letters, just gibberish.
And unicode big endian shows different gibberish than unicode does, only utf-8 for both files shows asian characters.

I tried with the _setmode(_fileno(stdout), _O_U16TEXT); code and a bunch of other code from all over the web then I just tried without it and was fortunate enough to be using utf-8 when I did this and saw asian characters.
I think when I installed the asian characters in the control panel regional setting I didn't need _setmode(_fileno(stdout), _O_U16TEXT); or something to be able to read the asian characters because with nothing there vs console said the codepage was 950.

Now I have to figure out how to change the codepage to 936, which is the Chinese simplified codepage.
I think MS made codepage 950 for xp and they didn't make a simplified version. So Iwill have to get a pro windows 7 and try to get codepage 936 there somehow.

So I guess my program is solved. First I had to get the Chinese fonts, then I had to change the fonts in regional settings, then I had to use the correct text file encoding utf-8.
Now I have to get the proper codepage which means I have to buy sme SW so I guess my problems solved for now.

Thanks Codeplug and Salem and jimblumberg.

my code:

Code:

#include <fcntl.h>
#include <io.h>
#include <cstdio>
#include <cwchar>
#include <iostream>
#include <fstream>
#include <locale>
#include <cstdlib> // exit()
 
using namespace std;
int main()
{
    // empty files
    ofstream fout("file_out.txt");
    fout.close();
  
    // get user input.
    ofstream holdInputSentence("file_out.txt");
    if (!holdInputSentence) { cerr<<"file error\n"; exit(1); }
  
    char inputSentence[4096];
  
    wcout << L"Enter your sentence, end it with a period: ";
 
    cin.getline (inputSentence,4096);
    holdInputSentence << inputSentence;
    holdInputSentence.close();
  
	// display user input
    wifstream read_the_counter_file("file_out.txt");
    wstring read_text;
  
    wcout <<  L"\n";
  
    while(read_the_counter_file >> read_text)
    {
        wcout << read_text << " ";
    }
  
    read_the_counter_file.close();
  
    wcout <<  L"\n\ncopying file to new file then display new file contents below\n\n";

	// empty files
    ofstream fout4("file_out.txt");
    fout4.close();

	// copy file

	wofstream fout3("file_out.txt", wfstream::app);
	wfstream read_the_file("cn_font.txt" , ios::in);
    wstring read_text2;
 
    while(read_the_file >> read_text2)
    {
        fout3 << read_text2;
    }
 
    read_the_file.close();
    fout3.close();

	// display the copy file

	wifstream fin("file_out.txt");
    wstring read_text3;
 
    while(getline(fin, read_text3))
    {
        wcout << read_text3;
    }
 
    fin.close();

	wcout <<  L"\n\n";
  
    // I pause the console window so you can read the screen results
    system ("pause");
}

my cn_font file contents:

Code:

绽湛栈蘸菚颤张章彰璋漳樟蟑鄣獐嫜餦麞粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

What I copy and paste into the console when testing the program:

Code:

嗄 1 x #

my program results:

Code:

Enter your sentence, end it with a period: 嗄 1 x #

嗄 1 x #

copying file to new file then display new file contents below

嚜輻遢皝猹?═撘蝡敶啁?瞍單??煾??擗阡?蝎駁?阮?鍫隞厰?瘨券?銝?隞


Press any key to continue . . .

whats in file_out.txt after the program runs:

Code:

绽湛蘸菚颤彰璋漳樟蟑鄣獐嫜餦麞粻鏱蔁騿掌鐣仉鞝涨障丈帐仗

I am using visual studio express c++ 2010, the console is using the asian font.
In the console options tab it says I'm using codepage 950 "Traditional Chinese big 5".

Of note is the Chinese characters I'm using are simplified, not traditional so some fonts aren't going to display for that reason.

How I was getting weird characters was I was using the two versions of unicode instead of utf-8.
When the files were unicode or unicode big endiad the read file looked like some weird fonts that aren't even letters, just gibberish.
And unicode big endian shows different gibberish than unicode does, only utf-8 for both files shows asian characters.

I tried with the _setmode(_fileno(stdout), _O_U16TEXT); code and a bunch of other code from all over the web then I just tried without it and was fortunate enough to be using utf-8 when I did this and saw asian characters.
I think when I installed the asian characters in the control panel regional setting I didn't need _setmode(_fileno(stdout), _O_U16TEXT); or something to be able to read the asian characters because with nothing there vs console said the codepage was 950.

Now I have to figure out how to change the codepage to 936, which is the Chinese simplified codepage.
I think MS made codepage 950 for xp and they didn't make a simplified version. So Iwill have to get a pro windows 7 and try to get codepage 936 there somehow.

So I guess my program is solved. First I had to get the Chinese fonts, then I had to change the fonts in regional settings, then I had to use the correct text file encoding utf-8.
Now I have to get the proper codepage which means I have to buy sme SW so I guess my problems solved for now.

Thanks Codeplug and Salem and jimblumberg.