Sockets... Sending Binary Data. Please help

I am working on a socket library in windows. I am able to send a file but there are 2 problems I am encountering.

Issue 1:

IF the file is small, say 120 bytes When i send the file from server to client the files does not send UNLESS I put in a sleep(1) after my send (server).

Issue 2:

The Client seems to be revieving 1 extra recieve and adding BUFF_SIZE amount of data extra to every single binary file that is sent. I have also noticed that at time the client does not recieve _BUFFSIZE bytes, the odd time (very odd) the number is less the _BUFFSIZE bytes recieved.

NOTES:

_BUFFSIZE = 100 for these example, I have played around w/ larger and smaller buffer sizes and I have same result.

I am atempting to send various files all w. same results (.mp3, .wma, .txt, .doc, .avi)

Here is my Send and Reveive from my socket Lib... I have made 100% certain my program only calls these 1 time each.

There is some trivial code (counters) in here for my debuging purposes.

Server

Code:

---

int ServerFileTransfer::sendFile( std::string const& strFileName )
{
        ifstream transferFile(strFileName.c_str(), ios::binary);

        int bytes = -1;

        if ( transferFile.is_open() )
        {
                ULONG filelength;
                transferFile.seekg(0, ios::end);
                filelength = transferFile.tellg();

                // put back to start of file for reading
                transferFile.seekg(0, ios::beg);

                TCHAR sendBuff[ _BUFFSIZE ] = {0};

                int count = 0;
                ULONG bytesSent = 0;
                for( bytesSent; bytesSent < filelength && !transferFile.eof(); bytesSent += _BUFFSIZE)
                {
                        transferFile.read( sendBuff, _BUFFSIZE );
                        bytes = send( _cliSocketData, sendBuff, _BUFFSIZE * sizeof( TCHAR ), 0 );
                        ++count;
                        if ( !isSocketOK( bytes ) )
                                return 0;
                }

                bytesSent -= _BUFFSIZE;

                if ( filelength > bytesSent )
                {
                        ULONG bytesRem = filelength - bytesSent;
                        transferFile.read( sendBuff, bytesRem);
                        bytes = send( _cliSocketData, sendBuff, bytesRem * sizeof( TCHAR ), 0 );
                        send
                        ++count;
                        if ( !isSocketOK( bytes ) )
                                return 0;
                }
                transferFile.close();
        }

        return true;
}

---

Client

Code:

---

int ClientFileTransfer::receiveFile( std::string& strFileName  )
{
        TCHAR recvBuff[ _BUFFSIZE ];
        int buffLen = _BUFFSIZE * sizeof( TCHAR );
        int bytes = 0;

        bool isOK = true;


        ofstream outFile(strFileName.c_str(), ios::binary);
        static ULONG bytesRecv = 0;
        int count = 0;
        while ((bytes = recv( _dataSock, recvBuff, buffLen, 0 )) > 0)
        {
                if ( bytes != _BUFFSIZE )
                        int x=0;
                bytesRecv += bytes;
                if (bytes != 100)
                        int x = 0;
                TransmitFile()
                outFile.write(recvBuff, bytes);
                if( !( isOK = isSocketOK( bytes ) ))
                        break;
                count++;
        }
        outFile.close();

        return isOK;
}

---

Feel free to make any suggestions, I will try them all and give feedback on how I make out.

Thanks,

EOF checking might very well be your problem here. I don't use the C++/STL I/O classes a lot, so I'm no expert there. But, here's the segment of concern:

Code:

---

transferFile.read( sendBuff, _BUFFSIZE );
bytes = send( _cliSocketData, sendBuff, _BUFFSIZE * sizeof( TCHAR ), 0 );

---

What if read() hits the end of the file, and that last chunk isn't exactly _BUFFSIZE bytes? read() fails, and the buffer is not completely filled. However, you send it as if it were full anyways. You seem to try to check for this after the for() loop, but it's too late by then.

Here's what I'd do:
Try to fill the buffer. Immediately after, check for eof. If you've hit one, see how many bytes you've read (is gcount() > 0?). If it ==0, break your loop. If it is >0, send, but only send gcount(). Something like:

Code:

---

do
{
  transferFile.read(sendbuf, _BUFFSIZE);
  if(transferFile.eof() && transferFile.gcount() == 0) break;
  bytes = send(your_socket, sendbuf, transferFile.gcount(), 0);
} while(!transferFile.eof());

---

Three assumptions not to make:
1) If the C++ eof() works like the C one (and I'm 95% sure it does), then it will only return true after the EOF has been read in, NOT if the next operation will return EOF.
2) _BUFSIZE == _BUFSIZE * sizeof(char) : You use different sizes in read() and send(). Don't. In fact, leave TCHAR out completely - use something like unsigned char (or my favourite, uint8_t) - it's a file, and you're just doing a binary transfer. TCHAR may not be 1 byte depending on your settings, and it needlessly reduces portability. (And are you really sending characters? Not for MP3, WMA, etc - raw data, better represented with unsigned char/uint8_t imho.)
3) Don't assume send() will actually send everything. I didn't correct this in my above snippet, but send() might not actually send() everything you pass it. Check send()'s return value - if it's less than the size you passed to it, you need to call it again, starting where it left off until it actually sends the whole buffer. You may want to wrap it in a small function to do this. See the man page.

And of course, error checking never hurts.
For odd data being sent, try WireShark or Ethereal - they'll allow you to see what data is really being sent, and are an invaluable tool when network programs go haywire. You may consider downloading one (they're both free). Get the server to send the correct stuff first, then work on recv'ing it.

Thank, I ll be sure to try everything out tomorrow before dinner. its a little late now and Im going to bed. One thing I wanted to point out that I may not have made clear is that when I say that in the client receive < _BUFFSIZE this does not happen at the EOF, it happens in the middle of the transfer.

And I think the eof is working correctly ( i may be wrong ) but if I send a file that is 30554 bytes and the _BUFFSIZE is 1000, the last 554 bytes do get sent fine. But again, for some reason I end up w/ an extrea 1000 bytes. So Cliend side the file would be 31554.

So if anyone else has suggestion or see's something that may be causing my issues please let me know. Again i will post an update w/ my changes tomorrow if the above suggestions help out any.

Thanks,

When the last read failed - the buffer stil contains 554 bytes of previous read and EOF
you send this buffer second time and it is appended to the file - removing the old EOF symbol and attaching the new one... So you get your strange size...

As was said - check the number of bytes actually read from the file and send exactly this number of bytes instead of your current approach

Thanks guys, this worked very well.

However the other problem of small files not sending any bytes exists.

Any idea why this could be.

Just guessing, couldn't it be a problem of flushing the transfer channel ?

Now might be a good time to repost what has changed in your code, since the problem seems to have moved to a different location.
Tools such as Wireshark/Ethereal are invaluable for seeing what's actually sent, but here's another thought: If you have Putty, you can connect to a server in Raw mode, and it'll show you what's being sent. Then just transfer a small text file, and see if it's transferring right.
Essentially, use a 3rd party tool that works to verify that the server is sending data as it should. Once you know the server to work, then tackle the issues in the client.

Hey I tried Putty out, works great and I'm sure it will help me out w/ this project. When i transfer the file Putty does display all the contents of my small text file, so it looks like the server is sending it ok, but the client is not receiving it. Here is my updated code... I have not yet gotten away from TCHAR, ill get to that, but that shouldnt have any effect on anything.

And again, keep in mind it sends larger files > 1MB fine, its only having a problem sending smaller files.

Server

Code:

---

int ServerFileTransfer::sendFile( std::string const& strFileName )
{
        ifstream transferFile(strFileName.c_str(), ios::binary);

        if ( transferFile.is_open() )
        {
                ULONG filelength;
                transferFile.seekg(0, ios::end);
                filelength = transferFile.tellg();

                // put back to start of file for reading
                transferFile.seekg(0, ios::beg);

                TCHAR sendBuff[ _FILEBUFFSIZE ] = {0};

                do
                {
                        transferFile.read(sendBuff, _FILEBUFFSIZE);
                        if( transferFile.eof() && transferFile.gcount() == 0 ) break;
                        if ( !isSocketOK( send( _cliSocketData, sendBuff, transferFile.gcount(), 0 ) ) )
                                return false;
                } while( !transferFile.eof() );
                transferFile.close();
        }

        return true;
}

---

Client

Code:

---

int ClientFileTransfer::receiveFile( std::string& strFileName  )
{
        TCHAR recvBuff[ _FILEBUFFSIZE ];
        int buffLen = _FILEBUFFSIZE * sizeof( TCHAR );
        int bytes = 0;

        bool isOK = true;

        ofstream outFile(strFileName.c_str(), ios::binary);
        static ULONG bytesRecv = 0;
        while ((bytes = recv( _dataSock, recvBuff, buffLen, 0 )) > 0)
        {
                bytesRecv += bytes;
                outFile.write(recvBuff, bytes);
                if( !( isOK = isSocketOK( bytes )) || bytes != _FILEBUFFSIZE )
                        break;
        }
        outFile.close();

        return isOK;
}

---

Thanks,

All this use of eof is bothersome - have you read the FAQ ?

> transferFile.read(sendBuff, _FILEBUFFSIZE);
Doesn't this return the number of characters actually read?

Code:

---

while ( (n=transferFile.read(sendBuff, _FILEBUFFSIZE)) != 0 ) {
  // now do something with 'n' bytes (NOT _FILEBUFFSIZE bytes or any other count)
  // Eg.
  totalBytes += n;
  // for example, try all your loops with just counting how many bytes there are in the file
  // Until you get consistent counts with respect to the file sizes, nothing else matters - GIGO
}
if ( !transferFile.eof() ) {
  // oops - reading didn't get to the end of the file
}
transferFile.close();

---

> send( _cliSocketData, sendBuff, transferFile.gcount(), 0 )
Just because you asked to send 'n' bytes doesn't mean it actually did it.
send() returns with the number of bytes actually sent - if this is less than you asked, you need to call send again with the bytes which didn't get sent.

Ditto on the recv(), it tells you how many were received, which for a TCP connection could be any number, not necessarily related to the sizes used in send() at the other end.

If I would have listened to the 1st reply I prob wouldnt have had the problem of small files not sending. The issue was that I was using TCHAR. After changing file transfer to use plain char it started working.

Thanks again for all who helped.

Quote:

---

All this use of eof is bothersome - have you read the FAQ ?

> transferFile.read(sendBuff, _FILEBUFFSIZE);
Doesn't this return the number of characters actually read?

---

That is not true. read actually returns *this.


If I would have listened to the 1st reply I prob wouldnt have had the problem of small files not sending. The issue was that I was using TCHAR. After changing file transfer to use plain char it started working.

Now I have disovered a new problem and it looks like its the way i receive.

Where I say (Client)

Code:

---

if( !( isOK = isSocketOK( bytes )) || bytes != _FILEBUFFSIZE )
        break;

---

This worked for smaller files so I thought all was good, but when I tried to send a file > 20MB There are times that my server does not always send _FILEBUFFSIZE bytes causing my client to break out of the loop when it should not. The reason i put bytes != _FILEBUFFSIZE into my if is because if that was not there the recv would hang after the last set of data was sent from the server.

Is there anouther way around this to I can ensure all of my data is succesfully sent? Possibly modify the code on server side to ensure _FILEBUFFSIZE bytes are always sent unless it is infact the last round of bytes that should be sent. Or is there anouther way I can modify the client so it accepts any number of bytes up to _FILEBUFFSIZE but still exits once the recv is complete?

I have an idea of what to do in the server and keep you posted if anythinng works, but any suggestions would be apriciated.

So how do you find out the number of bytes read then?

transferFile.gcount() tells you the number of bytes read

What I am thinking of doing is something like this:

Code:

---

                UINT bytesToSend = _FILEBUFFSIZE;
                do
                {
                        bytesToSend = _FILEBUFFSIZE;
                        do
                        {
                                transferFile.read(sendBuff, bytesToSend);
                                bytesToSend -= transferFile.gcount();
                                if( transferFile.eof() && transferFile.gcount() == 0) break;
                        } while( bytesToSend != 0 );

                        if ( !isSocketOK( send( _cliSocketData, sendBuff, transferFile.gcount(), 0 ) ) )
                                return false;                       
                } while( !transferFile.eof() && !transferFile.gcount() == 0 );
                transferFile.close();

---

So i ensure that _BUFFSIZE is always sent unless its the last set of data. On client side i can keep the recv loop going unless _BUFFSIZE is not revc indicating that its the end of file. The sending part of this seems to be working but the revc is hanging and not receiving anything for some reason.

Dang I just realized a stupid mistake in my solution. The way i made it, it will overwrite what was previously in the sendBuff when it reads.... this will not work.

Finally I think everything is resolved.

What I did is I went back to old way as suggested to read data from the buffer until 0 bytes are read and its the end of file. I send that data (is not always _BUFFSIZE) and I receieve. In my recevece I check to make sure there will be more data that is being sent using select(...) fuction. I specify a timeout time of 1 second... if it hits that, then i break out of the receieve loop. This seems to be working on all scenario's

Thanks again for everyone's help.

Quote:

---

I specify a timeout time of 1 second... if it hits that, then i break out of the receieve loop.

---

That seems a wee dangerous, should your connection slow temporarily.
If your situation is simple, you might not want to bother with the complexity of select() - just keep using recv(). For blocking sockets (the default) recv() will return 0 when the connection is closed, and -1 on any error. (Connection loss, etc.) Having the server close the connection after transfer seems it would suffice.
Should you need to keep the connection open for other data, then you should send the file's size before the file itself, so the client knows how much to recv.

What I have done is:

- I still use select but only to make sure there is data to be sent (to be same).. and this is before the recv loop

- To break out of the loop I send the file size expected, and when the amound of data recv == to what is expected i break out of the loop and return success message, if this is not true at any iteration of the loop I use select to check that there is still data to be sent.

For some reason If there is no data to be send the recv does not return 0 or an error code.. it just hangs.

Does the socket that is sending the data need to be closed in order to get out of the recv loop?