Comments (15)
Hi @pha3z - yes, issue received and typing a (much longer than anticipated) response, wanted to let you know!
from supersimpletcp.
Hi @pha3z - yes, what you're describing is the need for framing (which SimpleTcp doesn't do). SimpleTcp just has a single thread continually reading from a stream and punting the byte[] data to your app via an event.
Here's what I recommend you do to implement a simplistic framing mechanism (pseudo-code). SimpleTcp won't be able to accomplish this as is, but we may be able to craft a workable solution for you. The main reason being is that the stream reader is continually reading and firing events, and not giving you any input on how many bytes you actually want (which I think is a useful enhancement).
Assume that level of control is available to your app while reviewing this pseudocode.
- Read two bytes from the network and convert to integer
i
- If
i == 65536
- True: read 65536 bytes from network, write into an output stream, and repeat
- False: read
i
bytes from network, write into an output stream
- Convert output stream to byte array and send to application
I believe something like this is what you want, correct?
from supersimpletcp.
Hi @pha3z - yes, what you're describing is the need for framing (which SimpleTcp doesn't do). SimpleTcp just has a single thread continually reading from a stream and punting the byte[] data to your app via an event.
Here's what I recommend you do to implement a simplistic framing mechanism (pseudo-code). SimpleTcp won't be able to accomplish this as is, but we may be able to craft a workable solution for you. The main reason being is that the stream reader is continually reading and firing events, and not giving you any input on how many bytes you actually want (which I think is a useful enhancement).
Assume that level of control is available to your app while reviewing this pseudocode.
Read two bytes from the network and convert to integer
i
If
i == 65536
- True: read 65536 bytes from network, write into an output stream, and repeat
- False: read
i
bytes from network, write into an output streamConvert output stream to byte array and send to application
I believe something like this is what you want, correct?
I could certainly use that.
from supersimpletcp.
The logic sounds exactly right to me. I think that would fix the problem.
But this network stuff really confuses me. Are you saying that if Simple TCP knew how many bytes to expect as a complete network block/frame, it would be able to ensure the data is provided to application in the correct order?
In order for that to even hold true, would Simple TCP at least need a guarantee that the first packet (with the first two bytes) arrives before any other packets. Or is the issue of "data framing" separate from the issue of packet arrangement; are they two separate layers? As I explained in my original question, I saw one occurrence where a small piece of a block was provided from DataReceived() before the first piece of the block was provided from a second firing of the event. I presume that happened because packets arrived out of order, and Simple TCP just handed over the packet that arrived first instead of waiting to assemble the packets together in order. Is that correct?
If I'm understanding this correctly and the issue is really an issue of packet ordering, it seems like a more "generalized" fix would be to offer a "Ordered Packets" option so that data is always pushed to the application in order the sender originally sent it. Is that feasible?
from supersimpletcp.
Hi @pha3z SimpleTcp has, on the client side, a single thread reading from the stream using a buffer of a fixed size. Once it reads data, it passes it to the application. It isn't aware of where an application-layer message begins, ends, etc. That's where framing comes in - SimpleTcp doesn't do it (WatsonTcp does, but it requires WatsonTcp on both ends). On the server side, SimpleTcp has a single thread per client and each thread reads only from the stream for that particular client.
TCP provides sequence numbers to ensure that data arrives in order, even if the packets containing the data do not. SimpleTcp doesn't get in the way of that, and leaves that to the underlying operating system. See: https://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/
So I'm not sure how anything would ever arrive out of order. Event invocations are also synchronous, meaning when DataReceived
is invoked, the DataReceiver
is blocked. If you're seeing data blocks out of order based on the invocations of your DataReceived
event handler, then they are either being a) sent out of order or b) the underlying OS is not re-ordering according to the sequence numbers.
Please take a look at this gist - I hacked down the SimpleTcp client to something that gives full control of how many bytes to read to the calling application. I haven't done it for the server (more complicated, unless you're only ever going to have a single client connected).
Let me know if something like this would be useful. Look primarily at the three public methods Connect
, Send
, and Read
. https://gist.github.com/jchristn/cf518c17d4fa5ebc4b9fe501d06fb1f7
As I mentioned this is very easy to do client-side, server-side is harder unless you're ok with only having a single connected client.
from supersimpletcp.
So I'm not sure how anything would ever arrive out of order. Event invocations are also synchronous, meaning when DataReceived is invoked, the DataReceiver is blocked. If you're seeing data blocks out of order based on the invocations of your DataReceived event handler, then they are either being a) sent out of order or b) the underlying OS is not re-ordering according to the sequence numbers. - jchristn
P > That's precisely what I'm describing and why I was very confused. To be absolutely clear, yes: I logged at least one event where the client should have sent a block of data in this form: ABC. DataReceived() fired on the server 3 different times, and the order of data I got from the firings was: BAC.
Are you 100% positive there isn't some bug that could cause DataReceived invocations to occur out of order? Its very hard to believe that the 3rd party client is sending packets out of order... and even harder to believe the OS is not re-ordering properly... its Ubuntu server.
from supersimpletcp.
I think I know what the problem is. Its in my handler:
private async Task<bool> DataReceived(string ipPort, byte[] data)
{
//_log?.Debug("[TCP_SERVER] Data received");
await _dataReceivedHandler(ipPort, new ArrayBufferReader<byte>(data));
return true;
}
Since I'm calling my own internal handler asynchronously, that means .net can divert control back to the original caller, which allows the DataReceiver to fire immediately again for the next packet. Then there's a race condition. And the second invocation happens to beat the first invocation.
In order to assemble the packets properly, all code called from the DataReceived() method has to be synchronous. Do you concur with this?
from supersimpletcp.
Hi @pha3z, to your question:
In order to assemble the packets properly, all code called from the DataReceived() method has to be synchronous. Do you concur with this?
Going synchronous would be one way to solve the problem. And in this type of case I would recommend it because there's a relationship between the data components (A, B, C) that you're reading from the network.
If I were to craft another NuGet package that provided a client and a server, exposing APIs similar to what I showed in the gist, would that be useful to you? This is probably something I could do in a pretty short amount of time.
from supersimpletcp.
I don't think it would be helpful to me, because I already have the entire application logic in place to do the Data Framing. It works perfectly fine when I get the data in the right order. I just wasn't 100% sure how the TCP packet reception worked under the hood and I thought that was the cause of me getting data out of order.
What seemed odd to me though is that the DataReceived event handler for Simple TCP server is expected to return a Task. But after some thought, I suppose sometimes someone might intentionally WANT to react asynchronously to event data, in which case the event handler would need to be defined as asynchronous. Maybe this is one of the reasons async programming often gets confusing. I think I mistakenly thought the event handler is async, so I should be calling things from it asynchronously as a pattern.
from supersimpletcp.
What version are you using? The latest (2.x) uses EventHandler instead of callbacks.
i.e.
public event EventHandler<DataReceivedFromClientEventArgs> DataReceived;
And the implementation:
_Server.DataReceived += DataReceived;
and
static void DataReceived(object sender, DataReceivedFromClientEventArgs e)
{
Console.WriteLine("[" + e.IpPort + "]: " + Encoding.UTF8.GetString(e.Data));
}
from supersimpletcp.
That makes way more sense! :)
My project is still on version 1.1.7.
I suppose I'll have to read the change releases now and catch up.
from supersimpletcp.
Ok - please give it a whirl and let me know if this helps at all. It shouldn't be a terribly painful upgrade.
But just note, this won't solve your framing problem...
Please let me know if this at least solves the out of order issue. Cheers, Joel
from supersimpletcp.
Succeeded with the upgrade. Code works solid now. 💯 Thank you so much for your help!
from supersimpletcp.
Glad to hear my friend! BTW I've started working on another library that will give more explicit control over reads etc. I'll post the link to it here when it's ready. Cheers @pha3z
from supersimpletcp.
Hi @pha3z not sure if this is of any use/value to you, but I modified SimpleTcp to give the consuming application explicit control over when data is read. This will allow much easier build outs of state machines but has some draw backs, one of which being less-than-immediate disconnect detection (since there is no background thread continually attempting to read data).
Please let me know if there is any interest in trying this out as potentially a better fit:
Repo: https://github.com/jchristn/cavemantcp
NuGet: https://www.nuget.org/packages/CavemanTcp/
from supersimpletcp.
Related Issues (20)
- Message Framing Question HOT 1
- Send(); fails on Windows 8/8.1 HOT 2
- SimpleTcpServer DataReceived method spawns a new task for each DataReceived event HOT 3
- receive data not fast after connect HOT 1
- Server.IsListening wrong behavior when MaxConnections reached HOT 1
- Client connection is lost with no obvious reason HOT 10
- how to include dll file in exe file to be a one exe file ?? HOT 1
- NoDelay option for listener HOT 1
- KeepAliveSettings for server are applied in seconds on connecting client instead of ms HOT 1
- An active client with keep alive that does not process incoming message leads memory buildup in the server HOT 1
- What metadata is being embeded when the data is send in chunks? HOT 1
- SimpleTcpClient.Events.Disconnected is not triggered when SimpleTcpServer.Close() is invoked. HOT 1
- Block Unwanted Participants HOT 1
- multiple IP address HOT 2
- Keep Alive in linux container HOT 1
- The server sends messages to all clients on a regular basis HOT 2
- The client connecting to the Server on Maui is immediately kicked out. HOT 17
- About _Server.Stop() HOT 5
- supersimpletcp is missing NuGet package README file
- Why doesn't Dispose call Stop? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from supersimpletcp.