I've been looking at the docs for chan but I can't figure out how to change this.
https://www.tcl.tk/man/tcl/TclCmd/chan.html

When I send a message from my tcl server to a python client the message comes out in 3 lines and when I send a message to a java client I have to read the channel 3 times as it reads the channel one line at a time.

This is what I'm sending:
chan puts $channel [json::dict2json [dict create "time" "to" "say" "goodbye"]]

And this is what is received on both clients:
{"time": to
,"say": goodbye
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is my channel configurations, I forgot to show it in my post:
chan configure $chan -blocking 0 -buffering line

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I forgot to add my channel configurations:

chan configure $chan -blocking 0 -buffering line

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

It turns out the json formatting did not look correct which distracted me from the idea that the dict2json was making the json in 3 lines.

It should look like this:
{
"time": "to",
"say": "goodbye"
}

not this:
{"time": to
,"say": goodbye
}

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I've decided to create the json messages my self so I can control the formatting.

"{\"command\": \"create_lobby\", \"status\" \"success\", \"lobby_id\": \"$id\"}"

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Thank you very much for that code, mapping the new lines is a very good idea.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/12/2023 3:50 PM, Shaun Kulesa wrote:

I've decided to create the json messages my self so I can control the formatting.

"{\"command\": \"create_lobby\", \"status\" \"success\", \"lobby_id\": \"$id\"}"

A couple of inputs:

a) If your channel is line-oriented, then yes, the clients will get the
data one line at a time. Many protocols have solved this by sending the
size of the data first, followed by the data itself. See if you can
adopt this convention.

b) You can avoid recreating your own json-to-dict algorithm. You can
replace newlines with a space character, for example. Of course, if you
have data items that span multiple lines, you may need something more
robust such as using something that will never occur in the data.

% puts [string map {\n { }} [json::dict2json [dict create "time" "to"
"say" "goodbye"]]]

{"time": to ,"say": goodbye }

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Shaun Kulesa <[email protected]> wrote:

I've been looking at the docs for chan but I can't figure out how to change this.
https://www.tcl.tk/man/tcl/TclCmd/chan.html

When I send a message from my tcl server to a python client the
message comes out in 3 lines and when I send a message to a java
client I have to read the channel 3 times as it reads the channel one
line at a time.

This is what I'm sending:
chan puts $channel [json::dict2json [dict create "time" "to" "say" "goodbye"]]

And this is what is received on both clients:
{"time": to
,"say": goodbye
}

TCP is not "message" oriented. There is no guarantee provided that the receiver sees the identical "blocks" written by the sender.

And, if you want to control the format, look at the json::write
package, and the "indented" and "aligned" flags.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

The json would do the whole line then do another line with the ending bracket but I think I found a suitable solution. I haven't tested it fully as I am busy but it seems to work fine.

chan puts $channel {command: create_lobby, status: success, lobby_id: 1}

The whole message goes through as one line in java and then I turn it into a hash map by looping through it.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* Shaun Kulesa <[email protected]>
| Thank you very much for that code, mapping the new lines is a very good idea.

Whenever I need to send arbitrary data on a line oriented connection,
I encode it via base64 as one line per message:

# sender:
package require base64 ; # from tcllib
set data "some random text\n with newlines and \000 binary data"
puts $chan [base64::encode -maxlen 0 [encoding convertto utf-8 $data]]

# receiver
package require base64 ; # from tcllib
if {[gets $chan line] > 0} {
set data [encoding convertfrom utf-8 [base64::decode $line]]
}

No guessing what might be a suitable replacement for \n etc.

HTH
R'

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

What are the benefits of sending the data as base64?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Shaun Kulesa <[email protected]> wrote:

What are the benefits of sending the data as base64?

All the 'characters' internal to the message are hidden, and the
message handlers only see the output base64 character-set. So your
handlers know when a message ends, because the first non-base64
character must be the end (or a corrupt message).

Base64 also encodes arbitrary binary data (which is its original
purpose, encoding binary data for transmission via a non-binary
transmission mechanism [i.e., email]).

You also don't have to 'write' any code, as Tcl already has base 64
encoding and decoding via the 'binary' command (provided you are using
at least 8.6+).

The disadvantage is your messages grow by about 33% in size due to the encoding.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

I'm making an online desktop application UNO game where there can be multiple lobbies, the server is running on a computer with open forward ports, the server handles input from the clients to do things like edit a database and then relay the new
database values to all the clients in that lobby such as deck size. I want to be able to send commands from the clients such as "create_lobby" and provide arguments with that command such as "lobby_name".

What would you suggest is the best protocol for that?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/12/23 18:32, Shaun Kulesa wrote:

The json would do the whole line then do another line with the ending bracket but I think I found a suitable solution. I haven't tested it fully as I am busy but it seems to work fine.

chan puts $channel {command: create_lobby, status: success, lobby_id: 1}

The whole message goes through as one line in java and then I turn it into a hash map by looping through it.

Actually, at best java waits until it gets a line before returning to
your code -- the message could "go" in several pieces.

It is a very, very, very bad protocol design to rely on EOLs like this.

You may want to go to a more proven protocol like http or websockets or
such.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/2023 10:39 AM, Shaun Kulesa wrote:

I'm making an online desktop application UNO game where there can be multiple lobbies, the server is running on a computer with open forward ports, the server handles input from the clients to do things like edit a database and then relay the new

database values to all the clients in that lobby such as deck size. I want to be able to send commands from the clients such as "create_lobby" and provide arguments with that command such as "lobby_name".

What would you suggest is the best protocol for that?

Given this, I'd suggest sticking to what you have developed so far.
Your message content (i.e., payload) is not internally complex as it
consists of simple mostly single words.

The only change you need, and my earlier suggestion should have
included, is that you need to explicitly include a newline character
after you do string map because we went from the issue of multiple lines
to no lines at all. Then any simple client will read it fine.


% set data [dict create "time" "to" "say" "goodbye"]
% chan puts $channel "[string map {\n { }} [json::dict2json $data]]\n"

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/2023 10:54 AM, Rich wrote:

Shaun Kulesa <[email protected]> wrote:

I want to be able to send commands from the clients such as
"create_lobby" and provide arguments with that command such as
"lobby_name".

What would you suggest is the best protocol for that?

Take a look at the Tcllib 'comm' module.

It provides exactly this "send a command to a remote Tcl interpreter" functionality.

I think the OP is already using at least two clients written in other languages: Python and Java. I am not sure comm will help much here as
the desire is to get a simple line-based text messaging protocol working.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/2023 8:02 AM, Rich wrote:

Shaun Kulesa <[email protected]> wrote:

What are the benefits of sending the data as base64?

...

The disadvantage is your messages grow by about 33% in size due to the encoding.

I can add a few more:

- First of all, this did nothing to resolve the issue you are seeing and
your Java client would still have the same problem: You need to include
a terminating newline character at the end of your message.

- you are increasing the complexity for all clients. They need to get
and/or write their own version of the base64 decoder and implement it.

- You are creating discrepancy between what you think you are sending
and what is actually being sent. There are wire-level network
applications that come in quite handy when you are debugging something.
This approach nullifies that.

- It goes against the unix way :-)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Shaun Kulesa <[email protected]> wrote:

I want to be able to send commands from the clients such as
"create_lobby" and provide arguments with that command such as
"lobby_name".

What would you suggest is the best protocol for that?

Take a look at the Tcllib 'comm' module.

It provides exactly this "send a command to a remote Tcl interpreter" functionality.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

* saitology9 <[email protected]>
| The only change you need, and my earlier suggestion should have
| included, is that you need to explicitly include a newline character
| after you do string map because we went from the issue of multiple
| lines to no lines at all. Then any simple client will read it fine.

Maybe I'm missing something, but [chan puts] already adds a newline
after the text, so why would another \n at the end of the text be required?

| % chan puts $channel "[string map {\n { }} [json::dict2json $data]]\n"

This blind replacement of \n by a space IMHO is a data-depending bug
waiting to happen (any \n in the actual payload will get lost).

R'

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thursday, April 13, 2023 at 4:59:15 PM UTC+2, saitology9 wrote:

On 4/13/2023 10:54 AM, Rich wrote:

Shaun Kulesa wrote:

I want to be able to send commands from the clients such as
"create_lobby" and provide arguments with that command such as
"lobby_name".

What would you suggest is the best protocol for that?

Take a look at the Tcllib 'comm' module.

It provides exactly this "send a command to a remote Tcl interpreter" functionality.

I think the OP is already using at least two clients written in other languages: Python and Java. I am not sure comm will help much here as
the desire is to get a simple line-based text messaging protocol working.

I think troubles started when the OP tried to workaround an issue (bug or wrong usage) in the receiving libraries.
Using JSON had been a decent choice, but why would one insist on the formatting/layout, if the parser was compliant?
From what I see in this thread, the receiver should keep parsing until the JSON object is complete.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/2023 12:13 PM, Ralf Fassel wrote:

* saitology9 <[email protected]>
| The only change you need, and my earlier suggestion should have
| included, is that you need to explicitly include a newline character
| after you do string map because we went from the issue of multiple
| lines to no lines at all. Then any simple client will read it fine.

Maybe I'm missing something, but [chan puts] already adds a newline
after the text, so why would another \n at the end of the text be required?

| % chan puts $channel "[string map {\n { }} [json::dict2json $data]]\n"

This blind replacement of \n by a space IMHO is a data-depending bug
waiting to happen (any \n in the actual payload will get lost).

Well, the title of this thread is: "Socket sends message in multiple
messages". Why? Because it has newline characters in the body. The
solution is simple, and given the OP's extra background info (i.e.,
channel being in line mode, his app requirements, etc.), it is quite
applicable and acceptable in real life.

I just read heinrichmartin's reply too and I agree with that post. The
client side could be more thorough in that it can keep reading and
checking to see if it is a complete json document. However, this also
has flaws in that it fails to recognize when a bad json doc is sent.

I think it was rich who pointed out that the OP could switch to a
"message" oriented scheme or use something like http. That would work,
but it may not be worth the trouble given the task.

A more "standard" way would be to send the size of the message first,
followed by the actual message. But the string-map solution is really ok
too.


A few short answers to some of your points:

No guessing what might be a suitable replacement for \n etc.

Your confusion may stem from the misconception the separator must be a
single character. This is not the case. You may want to read up on
mime and its message boundaries.

Whenever I need to send arbitrary data on a line oriented connection,
I encode it via base64 as one line per message:

I do this too. It is ok if you have control of the client as well as
the server components. Otherwise, it is not a good choice. I mentioned
some disadvantages of this approach earlier.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/2023 12:13 PM, Ralf Fassel wrote:

No guessing what might be a suitable replacement for \n etc.

...

This blind replacement of \n by a space IMHO is a data-depending bug
waiting to happen (any \n in the actual payload will get lost).

Only as buggy as the email protocol that has been in place since like
the 1970's or maybe even earlier.

Chances are you sent your reply through an email client which uses a
"suitable replacement for \n" :-)

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

saitology9 <[email protected]> wrote:

On 4/13/2023 10:54 AM, Rich wrote:

Shaun Kulesa <[email protected]> wrote:

I want to be able to send commands from the clients such as
"create_lobby" and provide arguments with that command such as
"lobby_name".

What would you suggest is the best protocol for that?

Take a look at the Tcllib 'comm' module.

It provides exactly this "send a command to a remote Tcl interpreter"
functionality.

I think the OP is already using at least two clients written in other languages: Python and Java. I am not sure comm will help much here as
the desire is to get a simple line-based text messaging protocol working.

If he is trying to send to non Tcl clients, or from non Tcl clients,
then yes, 'comm' won't help by itself.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Thursday, April 13, 2023 at 6:51:42 PM UTC+2, saitology9 wrote:

The
client side could be more thorough in that it can keep reading and
checking to see if it is a complete json document. However, this also
has flaws in that it fails to recognize when a bad json doc is sent.

This is exactly why one should not reinvent the wheel. Others have come across these issues,
and the library will hopefully arrange for a timeout and cleanup of the parser state.

I think it was rich who pointed out that the OP could switch to a
"message" oriented scheme or use something like http. That would work,
but it may not be worth the trouble given the task.

But it has the same issue as mentioned above.
What if an erroneous message does not end in \n\n?
Or if it is missing the content-length header but appends junk after the headers?

Error detection and recovering are essential for software that shall run longer than for a single batch job.
That may be built-in - or external monitoring software does the reset.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/13/23 09:39, Shaun Kulesa wrote:

I'm making an online desktop application UNO game where there can be multiple lobbies, the server is running on a computer with open forward ports, the server handles input from the clients to do things like edit a database and then relay the new

database values to all the clients in that lobby such as deck size. I want to be able to send commands from the clients such as "create_lobby" and provide arguments with that command such as "lobby_name".

What would you suggest is the best protocol for that?

HTTP

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Shaun Kulesa <[email protected]> wrote:

I'm making an online desktop application UNO game where there can be
multiple lobbies, the server is running on a computer with open forward ports, the server handles input from the clients to do things like edit a database and then relay the new database values to all the clients in
that lobby such as deck size. I want to be able to send commands from the clients such as "create_lobby" and provide arguments with that command
such as "lobby_name".

What would you suggest is the best protocol for that?

STOMP is what you are looking for:

https://wiki.tcl-lang.org/page/Stomp

It was made exactly for your purpose and there are client libraries for
every programming language you can (or can not) imagine, including JS (for things like angular & friends in web browsers). It can be transmitted via
pure TCP, websockets.. or enveloped in HTTP, JMS, SOAP, whatever. It is a bullet proof protocol used widely, with header mechanisms, authorization
etc.. So there is no need to reinvent all that ;)

--
EL

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)