GLBasic forum

Main forum => Tutorials => Topic started by: FutureCow on 2009-Nov-10

Title: Networking - Transmitting files between two computers / peer communication
Post by: FutureCow on 2009-Nov-10
I had a reason with my current project to implement file transfers between two applications. After looking at the 3 "networking for cowards" tutorials that Kitty Hello had written I found that they didn't give the exact solution to my problem. After a lot of testing (and Kitty Hello resolving a NUL character bug in the networking routines) I thought I'd try and save people having to work out how to do it themselves.

Notes for the test program


Program Routines
What the code does is as follows :-



Main routine[/u]
The main routine does two things, firstly it initialises the socket networking. This MUST be done before you try to send data across the network.
Code (glbasic) Select

SOCK_INIT()


Next it determines whether you want this program to act as the Server or Client, and runs the appropriate Function.



Server
The server code starts by obtaining the size of the file to be transmitted
Code (glbasic) Select

FileSize=GETFILESIZE(SourceFile$)


We try to open the file (error if it can't be opened) so we can read its contents
Code (glbasic) Select

FileOpened=OPENFILE(FileHandle,SourceFile$,1)
IF FileOpened=TRUE


Next we tell the program to listen on the socket for any incoming connection requests with the SOCK_TCPLISTEN(Socket Number) command. In my code it  assigns the name "Sock" to the socket.
Code (glbasic) Select

Sock=SOCK_TCPLISTEN(SocketNumber) // Listen for connections


We then loop the following command continuously. The SOCK_TCPACCEPT command will return a value other than -1 when a connection attempt has been made to the open socket.
Code (glbasic) Select

EstablishedConnection=SOCK_TCPACCEPT(Sock,ClientIPAddress) // Check for established client connection
IF EstablishedConnection > 0


We now know that we have a successfully connected client and can start talking to them. Firstly we send them the size of the file we will be transmitting. This is only needed to give them a display saying how much has been downloaded so far, and how much is still to come. We could just start writing the data at this point (assuming they knew what to do with it). In my code, it's been written so that the first thing transmitted will always be the file size. We send the data with the SOCK_TCPSEND command. It will return the number of bytes transferred to the remote end, or <0 if there was an error.
Code (glbasic) Select

STDOUT "Sending filesize of " + FileSize + " to client\n"
CurrentPacketSizeSent=SOCK_TCPSEND(EstablishedConnection,FileSize) // Send size to transfer to client


We then wait for a confirmation message back to say the file size was received. This error checking may not end up necessary, but is good practice regardless. I've arbitrarily made my program wait to receive the confirmation string "Received-proceed". We listen for data on the same socket that we transmit on. In this instance we listen using the "SOCK_RECV" command. When it receives data the command will return either the number of bytes read, or -1
if there was an error. If it returns -2 it means the socket is not ready yet to give us data, so we sleep for 5ms before trying again. The data will be returned in the ReceiveMessage$ string.
The SOCK_RECV command will read bytes from the socket until its either read "PacketSize" number of bytes, or it encounters an "\0" character. I've configured my program to always read 1024 bytes from the socket, though again this is an arbitrary value. Without getting into how data actually travels across the network, suffice to say it's probably best to leave this under 1488 (look up MTU if you're interested) and I'd make it a fair bit smaller than that to be on the safe side.
So this is how we receive the message confirmation, and do some error checking :-
Code (glbasic) Select

STDOUT "Waiting on confirmation of receipt.\n" // Wait for confirmation that client received the filesize
LOCAL ReceivedMessage=0 // We haven't received the receipt yet
WHILE ReceivedMessage=0 // While we haven't had the receipt
ConfirmationReceived=SOCK_RECV(EstablishedConnection,ReceiveMessage$,PacketSize) // Read from the socket
SELECT ConfirmationReceived
CASE -1 // The socket returned an error
DEBUG NETGETLASTERROR$()+"\n" // Print out the error message
BREAK
CASE -2 // The socket's not ready to be read
SLEEP 5 // Wait a little while
CASE > 0
ReceivedMessage=1 // We've read some data, stop listening to socket
STDOUT "Message received -" + ReceiveMessage$ + ".\n" // Output the message received
ENDSELECT
WEND

// When the confirmation is received...
IF ReceiveMessage$="Received-proceed"
STDOUT "File size confirmed. Sending file data...\n"


We've now got the go-ahead to start sending the actual file data. I've decided that I'm going to send the data in "packets" of 1024 bytes so the first thing I do in my file reading loop is determine that there are at least 1024 bytes of data left to read. If not, I determine how many bytes are left. The result will be contained in the "Diff" variable.
Code (glbasic) Select

CurrentFilePos=FILEPOSITION(FileHandle) // Where are we in the file we're reading?
IF FileSize-CurrentFilePos < PacketSize // If there are less than "PacketSize" bytes left in the file
Diff = FileSize - CurrentFilePos // We'll make a smaller "packet" of size Diff
ELSE // Else
Diff=PacketSize // The number of chars to send (Diff) is the same as the "PacketSize"
ENDIF


Now we know how many bytes we're going to read in the next file read. The easiest way to grab the source data would be to do a string read of 1024 characters from the file, but unfortunately this will likely cause issues. The READSTR command used to read the data uses the NUL character (ASCII code 0 - also written as \0 or in GLBasic code as CHR$(0)) to determine when it's finished reading a string of text. Binary files unfortunately can include a NUL character as part of their standard data, so if you do a big READSTR and inadvertently read a NUL part way through, the file data will be truncated before it contains the full 1024 characters.

To get around the problem, we read the string 1 character at a time. If that character read is a NUL, I explicitly put a NUL character into the transmission string (DataString$).
Code (glbasic) Select

FOR Loop=1 TO Diff
IF ENDOFFILE(FileHandle) = FALSE // Precaution - should never occur
READSTR FileHandle, CurrentByte$, 1 // Read a byte from the file
IF CurrentByte$="\0" // NUL read
INC DataString$, CHR$(0) // Add NUL to transmission string
ELSE // Else
INC DataString$, CurrentByte$ // Add non-NUL character to transmission string
ENDIF
ELSE
BREAK
ENDIF
NEXT


We now have a data "packet" of 1024 bytes from the file (or less if there were less than 1024 bytes still to be read from the file), so we send it. We use the TCPSEND command to send the transmission string (DataString$). The value returned will be the amount of data transferred. If the value is -1 an error occurred and we need to print an error message. If no error occurred, we increment the number of bytes that have been sent (TotalAmountSent) and print out a status message.
Code (glbasic) Select

CurrentPacketSizeSent=SOCK_TCPSEND(EstablishedConnection,DataString$) // Send PacketSize bytes of data
IF CurrentPacketSizeSent = -1 // ERROR!
STDOUT "ERROR! - " + NETGETLASTERROR$()+"\n" // Print error string
DEBUG "ERROR! - " + NETGETLASTERROR$()+"\n" // Print error string
BREAK // Stop sending
ELSE
INC TotalAmountSent, CurrentPacketSizeSent // Increment amount of data transferred
STDOUT TotalAmountSent + " bytes of " + FileSize + " bytes sent\n" // Output amount of data transferred
ENDIF


Once we've transmitted all the packets we then close the file and the socket.
Code (glbasic) Select

SOCK_CLOSE(Sock) // Close socket
CLOSEFILE FileHandle // Close file




Client
The client code does pretty much the reverse of the server code. When the client is receiving, the server is sending and vice-versa.
We create the connection to the server with the TCP_CONNECT command. It will return -1 if there is an error opening the connection (e.g. the server hasn't created their connection yet), or -2 if the socket isn't ready for use.
Code (glbasic) Select

EstablishedConnection=SOCK_TCPCONNECT(ServerIP$,SocketNumber) // Try to connect to server
IF EstablishedConnection > -1 // Success


We then get the size of the file to be transmitted with the SOCK_RECV command and do error checking on the results.
Code (glbasic) Select

DataLength=SOCK_RECV(EstablishedConnection,ReceiveMessage$,PacketSize) // Try to read size of file to transmit


Assuming the filesize was successfully read, let the server know we got it and that we're ready for the data.
Code (glbasic) Select

CurrentPacketSizeSent=SOCK_TCPSEND(EstablishedConnection,"Received-proceed") // Send back a "Received-proceed" confirmation of message transmission


Open the destination file to be populated with the data.
Code (glbasic) Select
OPENFILE(FileHandle, DestFile$, 0)

Loop whilever data is being transmitted, reading data for the file with the SOCK_RECV function
Code (glbasic) Select

DataLength=SOCK_RECV(EstablishedConnection,ReceiveMessage$,PacketSize) // Read "PacketSize" bytes for the destination file


Increment our counter of how much has been read so we can keep the user appraised of the progress, then write the data (ReceiveMessage$) to the destination file (FileHandle). When the amount of data read is less than "PacketSize", we've read all the data.
Code (glbasic) Select

INC ReadFileSize, DataLength // We've read valid data. Increase our count of how much has been transferred
WRITESTR FileHandle, ReceiveMessage$ // Write transferred data to the destination file
IF DataLength < PacketSize THEN Finished=1 // When we read less than "PacketSize" bytes, we should have all the file.


Display how much data was transferred, close the file and the open socket.
Code (glbasic) Select

STDOUT "\n\nOriginal size : "+FileSize // Print out totals
STDOUT "\nRead size : " + ReadFileSize + "\n"
SOCK_CLOSE(EstablishedConnection) // Close socket
CLOSEFILE FileHandle // Close file


And we're done!
I've attached the complete code



So what else does the program need?
* There are two things this program needs. Firstly it could do with more error checking in a few places, reporting the error message back to the user when an error is encountered (send them the output of the NETGETLASTERROR command which works for both NET... and SOCK... commands)
* Secondly the program should also generate checksums for the file on both ends and compare them to check that the file transferred correctly.

I hope you find this useful!


[attachment deleted by admin]
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: Moru on 2009-Nov-10
Wow, thanks a lot, saves many hours :-) And who said programmers don't do documentation. Fine work :-)
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: Ian Price on 2009-Nov-10
This will come in mighty handy I'm sure. :)
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: FutureCow on 2009-Nov-10
Thanks for the kind remarks everyone!
There were a fair few hours work in working out how to use the networking commands and getting it all working properly (the bug with \0 characters was frustrating to find!!) Everyone here is really generous with their time when I ask questions so I was happy to be able to give something substantial back to the community for a change.
I hope you all get good use out of it!!!
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: Kitty Hello on 2009-Nov-10
Excellent. The next step would be to write a TFTP server/client. That's really easy to do now:
http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol (http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol)
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: codegit on 2009-Nov-10
Dude, this is an AWESOME tutorial......thanks.  :good:
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: Hemlos on 2009-Nov-10
Awesome  :good:

This will be handy for updating my MMO game directly through the game server.

Thanks!
Title: Re: Networking - Transmitting files between two computers / peer communication
Post by: bigsofty on 2009-Nov-10
This is a very nice gesture, many thanks!  :good: