Problem sending Unicode in C++

My application in C + + uses winsock2.h It use the send method to send the frames to the server developend on java.

My application send a string so:

#define MAXLONGITUD 10000
char bufEnviados[MAXLONGITUD + 1];
bufEnviados[0] = (0xff & (longitud >> 8));
bufEnviados[1] = (0xff & longitud);
send(sock, bufEnviados,strlen(bufEnviados), 0 );

I send the two first characters with UTF format so that the server in java could recognize them. Java Use readUTF to receive de string.

The application operated correctly it until received a string of 6.236 characters later from which it close the connection with the socket and it fall my application!

I have carried out several tests and I don't succeed in sending more than those 6.236 characters, however it achievement receibing whatever quantity of characters without problems!

Could somebody help me to resolve this problem? I am attempting of everything and I don't achieve it!

Quote:

strlen(bufEnviados)

That's wrong. You are encoding unicode characters into a char array. There can (and probably is) several null bytes in this array, but you are only sending up to the first one.

Since unicode characters are 2 bytes, you probably want to use an array of 2 byte elements. For instance:

Code:

unsigned short bufEnviados[MAXLONGITUD + 1];

Then you can just send however many elements of the array you fill.

As for why you're connection is being closed, that I don't know. Which side is closing the connection (sender or receiver)? What does send() return?

Your code and your post don't match up.

Code:

#define MAXLONGITUD 10000 char bufEnviados[MAXLONGITUD + 1]; bufEnviados[0] = (0xff & (longitud >> 8)); bufEnviados[1] = (0xff & longitud); send(sock, bufEnviados,strlen(bufEnviados), 0 );

Here, we appear to be taking some integral type, perhaps 16-bit, called 'longitud', and encoding it into a character array as a big endian 16 bit integer. This is a typical operation. However, the send() statement that follows calls strlen() on the buffer, which we have no gaurentee that the buffer is actually null terminated. Furthermore, the buffer doesn't appear to be a string. From what I can see... 2 would be more correct than strlen(), but I suspect we're not seeing the whole picture, because:

Your post talks about Unicode, and strings. However, your code does not appear to do anything with unicode or strings. What are you trying to do? What do you have to send, and how do you have to send it? UTF-8? UTF-16? And do you mean 6 thousand characters, and not six characters? You seem to be a non-native speaker of English -- your post is a bit incoherent. America, the UK, and other English speakers typically format 'one thousand and five tenths' as 1,000.5

Edit: Looking up readUTF in the Java reference, it seems to want a UTF-8 string, which this is probably not. What type is longitud, what does it store, and what protocol are you communicating with?

Problem C++

Only use Unicode for the server developed in java (readUTF) to recognize my string.

The code in my application is in C + + and winsock32.h use to connect to the socket level, the code in which I have the problem is:

Code:

#include <stdio.h> #include "stdafx.h" #include "connect.h" #include <string.h> #include <comdef.h> #define MAXLONGITUD 12000 char bufRecibidos[MAXLONGITUD + 1]; char bufEnviados[MAXLONGITUD + 1]; int bytesRecibidos; int bytesEnviados; long longitud; SOCKET sock; BSTR inXML; BSTR outXML; char * strCOut; //INICIA TRANSACCIONALIDAD while( (bytesRecibidos = recv(sock,bufRecibidos,sizeof(bufRecibidos),0))){ //RECEPCION DE TRAMA DEL SERVIDOR OLECHAR* oleChar = NULL; oleChar = (OLECHAR*)calloc(bytesRecibidos-1, sizeof(OLECHAR)); MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, bufRecibidos + 2, -1, oleChar, bytesRecibidos-1); inXML = SysAllocString(oleChar); free(oleChar); //TRANSACCIONALIDAD SERVICIO EXTERNO map1-> ejecutarTX_PRUEBA_DATOS(&inXML,&outXML); //ENVIO DE TRAMA AL SERVIDOR _bstr_t strOut(outXML); strCOut = (char *)strOut; longitud = strlen(strCOut); bufEnviados[0] = (longitud >> 8); bufEnviados[1] = (longitud & 0xff); strcpy(bufEnviados + 2 , strCOut); longitud = strlen(bufEnviados + 2) ; LOG4CPLUS_DEBUG(log, "Respuesta a enviar: " << bufEnviados + 2); bytesEnviados = send(sock, bufEnviados, longitud + 3,0); }//While LOG4CPLUS_INFO(log, "Conexion con el servidor perdida, terminando aplicacion"); closesocket(sock); CoUninitialize(); printf("EL CLIENTE CIERRA EL SOCKET.....\n\n"); WSACleanup(); exit(0);

My application is constantly waiting for strings sent by the java server (writeUTF) until the receive chains correctly 12,000.00 characters with no problems.

The section "//receiving TRAMA SERVER" change the data type to do this kind OLECHAR to use a function. "//EXTERNAL SERVICE Transactional" This function returns me a string of characters of type BSTR.

This character string can be up to 12,000.00 I do the conversion:

bufEnviados[0] = (longitud >> 8);
bufEnviados[1] = (longitud & 0xff);

Java server to send data readUTF that is used as the reference DataInputStream java UTF8 enconding modified format, in which first two characters correspond to bits signifcativos.

After generating the first two characters and set them on the fix, copy the string returned by the total ejecutarTX_PRUEBA_DATOS role in the settlement that sent.

The application worked well until he received a string returned by ejecutarTX_PRUEBA_DATOS more characters to my application 6.236.00 indicates that bytesEnviados are correct, but the server tells me java error and closes the socket ... it closes.

Try " char bufEnviados [MAXLONGITUD + 1] to "unsigned short bufEnviados [MAXLONGITUD + 1], but I had the same problem

I hope I can help and identity of all!

Thank you very much!

Quote:

This character string can be up to 12,000.00 I do the conversion:

bufEnviados[0] = (longitud >> 8);
bufEnviados[1] = (longitud & 0xff);

Java server to send data readUTF that is used as the reference DataInputStream java UTF8 enconding modified format, in which first two characters correspond to bits signifcativos.

(Is your word for length 'longitud'? It's very close to the English 'longitude', which means something entirely different.)
You're encoding the length OK, but not the string. Your length (longitud) is also wrong.

Your code:

Code:

//ENVIO DE TRAMA AL SERVIDOR _bstr_t strOut(outXML);

This is ok. (Merely converts a BSTR to a _bstr_t -- _bstr_t is a C++ class, sort of like std::string for BSTRs.

Code:

strCOut = (char *)strOut; longitud = strlen(strCOut);

No. You cannot cast strOut (which is a class of type _bstr_t) to a char *, and take a strlen().

Instead:

Code:

longitud = strOut.length()

Code:

bufEnviados[0] = (longitud >> 8); bufEnviados[1] = (longitud & 0xff);

This is good, but longitud is wrong.

Code:

strcpy(bufEnviados + 2 , strCOut); longitud = strlen(bufEnviados + 2) ;

No, again, because strCOut is invalid. You can get at the UTF-16 data of the string by doing:

Code:

((WCHAR *)outXML) + 2

You need to encode the data as Java-UTF-8 (which is not the same as UTF-8, for reasons unknown.) Windows probably does not have a function for this, you will have to write one.

Starting from here:

Code:

//ENVIO DE TRAMA AL SERVIDOR size_t characters = SysStringLen(outXML); LPCWSTR str = ((WCHAR *) outXML) + 2; // 4 byte skip over the length prefix. // Encode str into Java-UTF-8 std::vector<unsigned char> java_string; encode_java_string(str, java_string); size_t length_of_java_string = java_string.size(); unsigned char length_buffer[2]; length_buffer[0] = length_of_java_string >> 8; length_buffer[1] = length_of_java_string & 0xFF; send(sock, length_buffer, 2, 0); send(sock, &java_string[0], length_of_java_string, 0);

Just to get you started:

Code:

void encode_java_string(const WCHAR *input_string, std::vector<unsigned char> &output_string) { // Convert the string from UTF-16 to Java UTF-8 // Java docs contain details. }

Hi good day!

In my code the word "longitud" means the size of the string received.

I am sending the first 2-bit coded, but the string I'm sending is not encoded, thus sending the server until I came and 6236.00 characters came out of my loop and closed the application.

But I made the changes and indicate how I got:

Quote:

longitud = strOut.length();

I got the error "Unhandled exception Cliente.exe 0xc000005 - Access Violation"
Estoy trabajando en C++ con Visual 6.0

I am working with the sentence:

Quote:

LPCWSTR str = ((WCHAR *) outXML) + 2;

I am still solving my function to convert string to java string

But they also found a document indicating that the java is UTF16, not 8 as I thought then that the 2 bits are significant FAQ - UTF-8, UTF-16, UTF-32 & BOM

The document says:

Quote:

UTF-8 is most common on the web. UTF-16 is used by Java and Windows. UTF-32 is used by various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing. [AF] & [MD] UTS #6: Compression Scheme for Unicode

I hope the weekend finishing the string conversion function.

If you have more suggestions please do not hesitate to tell me

And now thank you very much!

Heeeeeelp!

Quote:

But they also found a document indicating that the java is UTF16, not 8 as I thought

Java probably stores the string in RAM as UTF-16 -- this is generally the most efficient way time/space-wise to work with Unicode text if you need to perform text transformations. However, Unicode text is probably more often serialized to UTF-8 for storage or network transfers. And, as you indicated in your original post, you're using readUTF to read the text you're trying to send, right? readUTF, according to the Java documentation:

Quote:

Originally Posted by Java Docs

Reads in a string that has been encoded using a modified UTF-8 format.

...however, the return value of readUTF is a string, which is probably storing the data, internally, as UTF-16. The UTF-16 here is irrelevant, as we must send the data as 'modified UTF-8'.

Quote:

I am sending the first 2-bit coded, but the string I'm sending is not encoded, thus sending the server until I came and 6236.00 characters came out of my loop and closed the application.

What is 2-bit coded?
I'm having a really hard time reading your post -- it's gotten a bit better, but this paragraph is difficult. (I'm a native English speaker.) Perhaps if you know someone who speaks it? Also, post your code as it is now. Then I can see what you've changed, and what is still going on. (And code isn't subject to language boundaries quite as much.)
Finally, "6236.00" characters? You can't have a .01 of a character...

Quote:

I got the error "Unhandled exception Cliente.exe 0xc000005 - Access Violation"

Strange. While I think that strOut is a little unneeded for this situtation, it should be valid. Is "ejecutarTX_PRUEBA_DATOS" giving a valid output BSTR?

Hola que tal?

Make some changes to my code now becomes:

Code:

while( (bytesRecibidos = recv(sock,bufRecibidos,sizeof(bufRecibidos),0))) { //RECEPCION DE TRAMA DEL SERVIDOR OLECHAR* oleChar = NULL; oleChar = (OLECHAR*)calloc(bytesRecibidos-1, sizeof(OLECHAR)); MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, bufRecibidos + 2, -1, oleChar, bytesRecibidos-1); inXML = SysAllocString(oleChar); free(oleChar); //TRANSACCIONALIDAD map1-> ejecutarTX_PRUEBA_DATOS(&inXML,&outXML); //ENVIO DE TRAMA AL SERVIDOR _bstr_t strOut(outXML); longitud = SysStringLen(outXML); bufEnviados[0] = (longitud >> 8); bufEnviados[1] = (longitud & 0xff); strcpy(bufEnviados + 2 , (char *) strOut); bufEnviados[6238] = 'a'; //Envio de la trama LOG4CPLUS_DEBUG(log, "Respuesta a enviar: " << (char *) strOut); bytesEnviados = send(sock,bufEnviados, longitud + 2,0); }

I was analyzing the plot of 8711 characters I get from "ejecutarTX_PRUEBA_DATOS" and found:

Quote:

bufEnviados[6238] = '¾';

I change manually:

Quote:

bufEnviados [6238] = 'a';

And send the correct data! The java server received the 8711 characters, but i don´t know why I got into a

Quote:

bufEnviados[6238] = '¾'

should have received only one character? I do not see that happening? should do a validation to not receive these types of characters ...

My native language is Spanish and has cost me to explain my problem here as you understand it!

Thanks for your help my problem I have published elsewhere in English and Spanish and I had no response.

Thanks for your help and even better because I have to investigate receipt of different characters as you receive it in java ... because these same code in C + + and Java, is not problems on java!

Thanks for your help and if you have ideas as to why my problem ... and thanks heeeeelp!

Code:

while( (bytesRecibidos = recv(sock,bufRecibidos,sizeof(bufRecibidos),0)))

Quick question. Is this correct:
1) Java server sends text with writeUTF
2) You read that text
3) You process that text
4) You send the result back
5) Java server reads result with readUTF
?

Code:

MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, bufRecibidos + 2, -1, oleChar, bytesRecibidos-1);

Since you are passing in -1, this function assumes that the passed buffer is NULL terminated. This is probably not the case though since your string will be received in chunks (that's why you are calling recv() in a loop). In other words, DO NOT assume that the buffer filled by recv() is NULL terminated.

this is correct, my application in c + + is a service and is constantly waiting for processing a text and send it again ....

1) Java server sends text with writeUTF
2) I read that text
3) I process that text
4) I send the result back
5) Java server reads result with readUTF

For now my application is running but still have to perform a function that validates me that process has no foreign or accented characters (ñ, á, é, í, ó, ú) ...

Also I'm doing more tests to see all the mistakes that I can find to interact with java.

Greetings Alfa.

Quote:

Originally Posted by alfaximena

this is correct, my application in c + + is a service and is constantly waiting for processing a text and send it again ....

1) Java server sends text with writeUTF
2) I read that text
3) I process that text
4) I send the result back
5) Java server reads result with readUTF

Then we need to do that.
#1 and #5 are done & handled by the Java server, or so I presume. #3 is up to you.
That leaves us with #2 and #4. #2 is essentially two steps:
1) Read the java-utf8 string that the server sends into a buffer.
2) Process that java-utf8 buffer into something more useful, like a WCHAR[].

Now, as bithub pointed out, you're probably not recv()ing enough data. A single call to recv() may or may not get everything you ask it to. (If you ask for 40 bytes, you might get 0, 20, 33, or 40, or anywhere in between.) Therefore, since we need so many characters, we need to but recv() in a loop. (You also can't assume that 1 recv() == 1 send()) Let's write a simple function:

Code:

int recv_all(SOCKET s, unsigned char *buffer, size_t size) { // how many bytes have we read? size_t bytes_read = 0; int ret; // while he haven't read enough data while(bytes_read < size) { // recv as much as possible, but no more than that ret = recv(s, buffer + bytes_read, size - bytes_read, 0); if(ret <= 0) { // ret == 0 means closed socket. ret < 0 means an error. std::cerr << "Error or socket closed in recv_full()" << std::endl; // handle errors / closed socket and abort. return -1; } bytes_read += ret; } return size; }

Now, we can use this to get the string the server is sending us, something like:

Code:

unsigned char length_buffer[2]; if(2 != recv_all(your_socket, length_buffer, 2)) { // handle errors } size_t length = (length_buffer[0] << 8) | length_buffer[1]; unsigned char *buffer; buffer = new unsigned char[length]; // buffer will contain the data that Java sends us. // THIS IS NOT A UTF8 STRING if(length != recv_all(your_socket, buffer, length)) { // handle errors } LPWSTR *str; str = java_utf8_to_utf16(buffer, length); delete [] buffer;

Of course, you NEED to add error handling to this code.
Let me reiterate something: The data the server is sending you is NOT UTF-8. You CANNOT pass it the MultiByteToWideChar() and expect it to work. Read the following:

writeUTF()
readUTF()

Those documents describe the string that Java is sending you -- you will have to decode it yourself or find code that does -- MultiByteToWideChar will not work here. Read the documents, they contain the data you need to know to write java_utf8_to_utf16().

Quote:

For now my application is running but still have to perform a function that validates me that process has no foreign or accented characters (ñ, á, é, í, ó, ú) ...

Those characters aren't foreign in your language. Are they? (My language uses them on rare occaision.)

Hi,

I could spend all sizes of chains, but still can not validate a pass ascii to UTF8 string I'm using:

Code:

WideCharToMultiByte (CP_UTF8, MB_ERR_INVALID_CHARS, oleChar1, -1, bufEnviados + 1, length - 1, NULL, NULL);

But some characters are changed to unicode as "½" in my language is very normal use á, é, í, ó, ú and ñ

It is so clear that no change to unicode.

It seems that I am very slow to solve my problems!

Greetings

Quote:

I could spend all sizes of chains, but still can not validate a pass ascii to UTF8 string I'm using:

This makes absolutely no sense. An ASCII string is a UTF-8 string. (Though a UTF-8 string may or may not be an ASCII one.)

Code:

WideCharToMultiByte (CP_UTF8, MB_ERR_INVALID_CHARS, oleChar1, -1, bufEnviados + 1, length - 1, NULL, NULL);

This does what? converts a UTF-16 to UTF-8. But as I've stated -- Java, if you're using DataOutput/DataInput with writeUTF/readUTF (as you've said you are), is not expecting UTF-8 -- it wants Java-UTF8 which is probably the reason you're seeing odd results.

They are apples and oranges. You have three things:

UTF-8
UTF-16
Java's Modified UTF-8

Java's modified UTF-8 is what you're getting, and what you should send. You need to read, and understand what you are doing.

Quote:

But some characters are changed to unicode as "½" in my language is very normal use á, é, í, ó, ú and ñ

No idea what you mean here. Seeing odd results is generally the result of data corruption. From what I've seen of your code thus far, you are corrupting data.

HELLO!

I could send and receive any type of string for my application .... but all I cause small problem ... I have memory leaks!

I could send and receive any type of string for my application .... but all I cause small problem ... I have memory leaks!

Code:

//INICIA TRANSACCIONALIDAD while( (bytesRecibidos = recv(sock,bufRecibidos,sizeof(bufRecibidos),0))){ //RECEPCION DE TRAMA DEL SERVIDOR OLECHAR* oleChar = NULL; oleChar = (OLECHAR*)calloc(bytesRecibidos-1, sizeof(OLECHAR)); MultiByteToWideChar(CP_ACP, MB_PRECOMPOSED, bufRecibidos + 2, -1, oleChar, bytesRecibidos-1); inXML = SysAllocString(oleChar); free(oleChar); //TRANSACCIONALIDAD map1-> ejecutarTX_PRUEBA_DATOS(&inXML,&outXML); //ENVIO DE TRAMA AL SERVIDOR - DEBE SER FORMATO UTF8 MODIFICADO -> ENVIO UTF16 _bstr_t strOut(outXML); LPWSTR str = ((WCHAR *) strOut); nInputStrLen = wcslen (str); nOutputStrLen = WideCharToMultiByte (CP_UTF8, 0, str, nInputStrLen, NULL, 0, 0, 0) + 2; UTFSring = new char [nOutputStrLen]; memset (UTFSring, 0x00, nOutputStrLen); WideCharToMultiByte(CP_UTF8, 0, str, nInputStrLen, UTFSring, nOutputStrLen, 0, 0); longitud = strlen((char *)UTFSring); bufEnviados[0] = (longitud >> 8); bufEnviados[1] = (longitud & 0xff); strcpy(bufEnviados + 2 , (char *) UTFSring ); delete [] UTFSring; UTFSring = NULL; str = NULL; bytesEnviados = send(sock,bufEnviados, longitud + 2,0); }

Help!

Thanks Cactus_Hugger for the information about UTF ... helped me understand how to get java!