\0 terminated SOCK_RECV command

[FutureCow]

The original SOCK_RECV() would chop your messages up nicely by stopping reading when a '\0' was read.
This was changed after I found I couldn't transmit binary files due to \0 characters through them.

Now I've found I could really do with two versions of the SOCK_RECV command.
1) The current one that will read \0 as part of the stream (for binary transfers)
2) One that terminates at \0 for non-binary transfers (i.e. the way SOCK_RECV used to work)

Currently if you send something like

Code (glbasic) Select Expand

SOCK_TCPSEND([socket],"SetLives=1")
SOCK_TCPSEND([socket],"SetLevel=14")
SOCK_TCPSEND([socket],"ResetHighScoreTable")

then do a

Code (glbasic) Select Expand


SOCK_RECV([socket],LivesString$,1023)
SOCK_RECV([socket],LevelString$,1023)
SOCK_RECV([socket],ResetString$,1023)

InputString will contain "SetLives=1SetLevel=14ResetHighScoreTable" and LevelString$/ResetString$ will probably be empty. And it's not always consistent that all the output will be in the first SOCK_RECV() output as it depends on the timing when you run the SOCK_RECV command.

Having a new command that terminates on the '\0' means I could do 3 separate SOCK_RECV() calls and get each parameter and its setting separately for processing. Without it, I'm currently having to do something similar to the following (this is just an example, mines slightly more efficient)

Code (glbasic) Select Expand


Server
=======
SOCK_TCPSEND([socket],"SetLives=1")

Client
======
while Message$=""
    SOCK_RECV([sock],Message$,1023)
wend
if Message$="SetLives"
    SOCK_TCPSEND([socket],"Acknowledged")

Server
=======
while Message$=""
    SOCK_RECV([sock],Message$,1023)
wend
if Message$="Acknowledged"
    SOCK_TCPSEND([socket],"SetLevel=14")

[...]

and so on which wastes time waiting for acknowledgements and causes excess network chatter, or some really annoying string manipulations to strip each different length string off the network stream.

[Kitty Hello]

You get an ascii stream that has \0 characters? What is that?
If you know that, you could try SPLITSTR(str$, chr$(0));

[FutureCow]

I thought from testing the old version that the tcp_send automatically inserted a "\0" at the end of each transmission string.     I assume from your reply that I'm probably incorrect though.

SPLITSTR isn't quite as neat, but it'll be easier than my current solution - thanks Gernot!

[Kitty Hello]

now wait - the TCP_SEND will _not_ append a '\0' character, because web browsers don't send that, either.
If you receive, I will always force a '\0' at the end of what I read, so you can have a const char* to that string. With GLBasic only, you should not worry, because I have a length variable for each sting.
If you receive a string that contains a '\0' character, however, I'd be glad to hear where that comes from. Don't get me wrong, I'm not saying you do something wrong. It might easily be me missunderstanding of the specs.

[FutureCow]

Okay now I'm getting confused! 
My problem was that I didn't know in advance (before changing my code anyway) how long a string I was going to be sending at various stages. I assumed I could send any string and a '\0' would automatically append. Therefore I thought I could send say 5 strings of various lengths (say <100 characters), and (in theory) do 5 sock_recv's of 1023 characters to read them and it would all work nicely.

I just made the assumption from what I saw that the strings would automatically have a '\0' on them from TCP_SEND.
I'll have to have a look tomorrow and see how people normally do efficient stream communication for separate messages they want to send. I'm currently sending a 8 char "header" and my code knows for each different that header I send how many chars follow. I then use MID$ to delete the total number of processed chars from the string, and reprocess the string until it's empty. It's not neat but the best solution to fit my needs at short notice.

[Kitty Hello]

OK. Here's what I think about that. I have no idea if I'm right:
- You send data to another computer.
  - I send() as much of that as I can send.
  - if there's more data, I send again and again

- You read from a socket
  -I read until the other end either disconnects, or there is nothing more to read (EWOULDBLOCK)

Can anyone shed light on this? Ocean, maybe?

[Moru]

One thing I have always wondered about is if my app sends data very fast in small packets, will I still get the packets one at a time when I read from the socket or will they be concatenated on the way and sent as one packet? I was using some library for Visual Basic a few years ago and in the documentation it said not to expect the packets to stay separated, they could get concatenated in transfer. You had to put your own start packet/end packet codes into your packets.

About the original question, a packet is binary data, I don't think it should stop reading on 0, you should read until EOF or similar.

[Moru]

I have tested this long time ago (before the new net commands) and there was no need for zero-termination. I used \0 as split between command and parameters instead since glbasic don't terminate a string on ascii 0. Only thing that is important is the length of the string, just as it's supposed to be in basic. Zero-termination is just there for compatibility with inline C I guess?

[Kitty Hello]

OK, I got enlightened now.

When you send strings (long ones) over the internet, usually you provide some sort of "header" for you data. That means, you read 4 bytes that describe the length of the coming package, and then read this length from the socket, checking for errors (connection broken e.g.)

If you read from a website or so, the thing acts differently. They have a sort of EOF by disconnecting after the sending. The server will send you the full page and then close its socket. That way you know that your data was received completely.

For fast interchange, I'd suggest going route #1. That's BTW the thing I do with the NETSENDMSG commands.

Did I miss a thing?

[FutureCow]

Here's a test program with some interesting results

NOTE: Compile as a console program then run 2 copies of it.

Code (glbasic) Select Expand

GLOBAL Net_ServerIP$
GLOBAL EstablishedConnection
GLOBAL Net_SocketNumber
GLOBAL Done
Net_SocketNumber=1234
Net_ServerIP$="127.0.0.1"

SOCK_INIT()

// MAIN
//-------------------------------------------------------------------------
Done=0

STDOUT "Press 1 to be server\n"
STDOUT "Press 2 to be client\n"

LOCAL Result$=INKEY$()
IF Result$ = "1" // pressed 1
STDOUT "Server\n"
DEBUG "Server-debug\n"
GOSUB server
ELSEIF Result$= "2" // pressed 2
STDOUT "Client\n"
DEBUG "Client-debug\n"
GOSUB Client
ENDIF


//-------------------------------------------------------------------------
SUB server:
LOCAL Done=0
LOCAL ClientIPAddress%
LOCAL Sock%
STDOUT "Waiting for connection...\n"

Sock=SOCK_TCPLISTEN(Net_SocketNumber) // Listen for connections on our socket

WHILE Done=0
EstablishedConnection=SOCK_TCPACCEPT(Sock,ClientIPAddress) // Check for established client connection
IF EstablishedConnection > 0 THEN Done=1 // If a connection was successful
WEND

// We now have a connection

SOCK_TCPSEND(EstablishedConnection,"1")
SOCK_TCPSEND(EstablishedConnection,"2\0")
SOCK_TCPSEND(EstablishedConnection,"3"+CHR$(0))
SOCK_TCPSEND(EstablishedConnection,"4")
SOCK_TCPSEND(EstablishedConnection,"5\0")
SOCK_TCPSEND(EstablishedConnection,"6"+CHR$(0))

DEBUG ("Server Done\n")
SLEEP 10000
ENDSUB

//-------------------------------------------------------------------------
SUB Client:
LOCAL Done=0
LOCAL ReturnCode
LOCAL ReceivedMessage$
LOCAL YLoc=50

STDOUT "Establishing connection with "+Net_ServerIP$+" on port "+Net_SocketNumber+"...\n"
EstablishedConnection=SOCK_TCPCONNECT(Net_ServerIP$,Net_SocketNumber) // Try to connect to server
IF EstablishedConnection = -1 // Failure
STDERR "Network error : client failed to create connection"
END
ENDIF

SLEEP 500 // allow messages to cache
LOCAL Done=0
WHILE Done=0
ReturnCode=SOCK_RECV(EstablishedConnection, ReceivedMessage$, 1023)
IF ReturnCode=0
STDERR "Remote end has disconnected\n"
END
ELSEIF ReturnCode=-1
STDERR "Network error has occurred"
STDERR "Error received was : "+NETGETLASTERROR$()+"\n"
END
ELSEIF ReturnCode=-2 // Nothing waiting in the queue
Done=1
ELSE
DEBUG "Message length received : "+ReturnCode+"   Message recieved = " + ReceivedMessage$ + "\n"
STDOUT "Message length received : "+ReturnCode+"   Message recieved = " + ReceivedMessage$ + "\n"
ENDIF
WEND
SLEEP 10000
ENDSUB



What you will see is that the client receives only one message - the output from the client is

Code (glbasic) Select Expand

Message length received : 8   Message recieved = 123

4,5 and 6 don't appear, as the CHR$(0) terminates the printing of the string, but you can tell by the message length (8 characters) that all the data was sent, then returned by one SOCK_RECV() command.

From everything I've found doing an internet search, it appears that all characters (including \0) should be sent with send() or recv() - the use of the NUL character as a message delimeter would be entirely a user implementation issue.

From what you wrote Gernot I assume that NETGETMSG and NETSENDMSG are appending a \0 and doing a SPLITSTR to separate the messages?

[Kitty Hello]

It acts as expected. You send data and the other side can read data.
So, if you want to send a string and do _not_ close the connection but want to write another string, you must append a string length header. That's what I do with the NET commands.

Here's what you want:

Code (glbasic) Select Expand


// --------------------------------- //
// Project: _moo
// Start: Friday, January 22, 2010
// IDE Version: 7.242


// SETCURRENTDIR("Media") // seperate media and binaries?
GLOBAL Net_ServerIP$
GLOBAL EstablishedConnection%
GLOBAL Net_SocketNumber%
Net_SocketNumber=1234
Net_ServerIP$="127.0.0.1"

SOCK_INIT()

// MAIN
//-------------------------------------------------------------------------
Done=0

STDOUT "Press 1 to be server\n"
STDOUT "Press 2 to be client\n"

LOCAL Result$=INKEY$()
IF Result$ = "1"      // pressed 1
   STDOUT "Server\n"
   GOSUB server
ELSEIF Result$= "2"   // pressed 2
   STDOUT "Client\n"
   GOSUB Client
ENDIF


FUNCTION SendString%: sock%, str$
// pack the length as a 10 byte string
SOCK_TCPSEND(sock%, FORMAT$(10,0,LEN(str$)) + str$)
ENDFUNCTION


FUNCTION ReceiveString$: sock%
LOCAL size%, str$, ok%
ok% = SOCK_RECV(sock%, str$, 10) // length package
IF ok% <= 0 THEN RETURN "" // nothing there

// now we must read the string - dead or alive, you're sending to me
size% = str$
// no need to loop - GLBasic does that for you
WHILE TRUE
ok% = SOCK_RECV(sock%, str$, size%)
IF ok% = -2 THEN CONTINUE // no data sent, yet. Retry
BREAK
WEND
RETURN str$
ENDFUNCTION



//-------------------------------------------------------------------------
SUB server:
   LOCAL Done=0
   LOCAL ClientIPAddress%
   LOCAL Sock%
   STDOUT "Waiting for connection...\n"
   
   Sock=SOCK_TCPLISTEN(Net_SocketNumber)                                    // Listen for connections on our socket
   
   WHILE Done=0
      EstablishedConnection=SOCK_TCPACCEPT(Sock,ClientIPAddress)               // Check for established client connection
      IF EstablishedConnection > 0 THEN Done=1                           // If a connection was successful
   WEND
   
   // We now have a connection
   STDOUT "Sending...\n"
   
   LOCAL names$[]
   DIMDATA names$[], "John Rambo", "Jim Knopf", "Kermit the Frog", "Dr. Shiver - buy now"
   
   WHILE TRUE
      SendString(EstablishedConnection, names$[RND(LEN(names$[])-1)] )
      SLEEP 3000
   WEND
ENDSUB

//-------------------------------------------------------------------------
SUB Client:
   LOCAL Done=0
   LOCAL ReturnCode
   LOCAL ReceivedMessage$
   LOCAL YLoc=50
   
   STDOUT "Establishing connection with "+Net_ServerIP$+" on port "+Net_SocketNumber+"...\n"
   EstablishedConnection=SOCK_TCPCONNECT(Net_ServerIP$,Net_SocketNumber)      // Try to connect to server
   IF EstablishedConnection = -1                              // Failure
      STDERR "Network error : client failed to create connection"
      END
   ENDIF

   WHILE TRUE
       ReceivedMessage$ = ReceiveString$(EstablishedConnection)
       IF LEN(ReceivedMessage$)
       STDOUT "Got: \"" + ReceivedMessage$ + "\" len="+LEN(ReceivedMessage$)+"\n"
   ENDIF
       SLEEP 500
   WEND

ENDSUB


[Kitty Hello]

Read it into a string then send. The LEN and SEND functions do not care about '\0' characters.

[Kitty Hello]

the UUENCODE is something you have to do yourself. The socket commands don't do that.
I have posted an article how to upload a png file to a webserver (HTTP), see that.

I think all problems are solved now, no?

[Moru]

I don't realy see the problem, GLB treats strings as binary data right up to the instant you try to print it to the screen, right? So just use it as binary data, don't print it to the screen.