Implement fragmentation support about rosbridge_suite

[dabertram]

just to let you know about progress on this issue..

I have prototyped a class that:

• takes messages with op-code "fragment",
• reconstructs the original message,
• passes the reconstructed message again to "protocoll.incoming(message)"

first tests look like it's working for huge messages (tested with 1.000.000 bytes) that were fragmented into chunks of 1000 bytes..
I was able to receive the original message by calling "rostopic echo /..."

The code is not yet well written/documented/..
as soon as I have more news, a class that is well tested and working, or even something suitable for a pull request I will update on this issue.

[rctoris]

👏

[dabertram]

Yesterday I successfully used my rosbridge (de)fragmentation class to send a message containing 100.000.000 bytes (chunk size 1000 bytes).
It took almost 90 minutes to send all fragments to rosbridge plus several hours (I had to leave it running over night..) for the message to arrive 'at rostopic'. But it arrived in the end..

I noticed that while rosbridge was processing this mass of data, rosbridge wasn't able to receive further messages.
I will try if using a 'dedicated' thread for defragmentation can solve this..
although I think this is not a 'real' problem since messages of this size shouldn't occure too often, let's say there is at least some room for improvements..
Another (ugly) option would be to set a maximum size for the reconstructed message..

type-checking and stuff is still missing from defragmentation class, but I think this can be added easily before merging into the main repository.. ;)

[baalexander]

Thanks for the update @ipa-fxm-db. What protocol are you using to transfer the data to rosbridge (TCP, WebSockets, TCP over WebSockets, etc.)?

Rosbridge_server contains the Tornado library, which adds some convenient non-blocking network I/O functionality. It could be worth taking a look at.

[dabertram]

I'm just using rosbridge with it's websocket support.
Sorry, I didn't describe the problem correctly..

After finishing reconstruction of the huge message, the missing messages appeared.
So they were not blocked by network layer or similar, but instead they were delayed until the huge message was reconstructed..

Currently I'm working on a mechanism that allows to have a timeout for received fragments and free up memory in case a fragmented message wasn't completed within a certain time (assuming rosbridge usually is running for a long time)

I will try to find out what is causing the delays after finishing the timeout thing... but I don't expect this to be a big issue..

[dabertram]

Update:

I turned Defragmentation class into a thread (rosbridge creates 1 instance of this class for every connected client).

• messages that are published with op-code "publish" are not delayed (..would need to check if they were before)
• reconstruction of messages is not delayed anymore if two different fragmented messages (different original messages, each with it's own 'sender' ) are being processed at the same.

next thing is to change string concatenation from using "+=" to String.join() to improve performance while reconstructing..
this requires an ordered list instead of the dict that I was using until now (lazy.. I know ;) ) ..so I will have to change some smaller parts.

We are going to test if my class fits our own use case by the next week.
You can expect a pull request after that (if everything is working as it should)

[baalexander]

@ipa-fxm-db - Another thing you may want to look at if the JSON encoding/decoding is taking a long time is using a better JSON library than the default Python one (which rosbridge currently uses).

Issue #7 mentions two libraries: simplejson and ultrajson.

Either library should provide substantial performance increases.

[dabertram]

Thanks for the recommendations! ..but read below..

I added some code to measure internal times.. and I think performance is good enough for the moment..
each fragment was about 1000 bytes.. see rosbridge output below:

(look at last line)
Reconstruction of a message of (10001 fragments * 1000 bytes) = ~10.000.000 bytes took about 1 minute.
Compared to the time it took for sending the fragments ( "10001 message fragments sent. [duration: 410.442198s]" ) I think it's okay to leave it like it is for now..

You can also see from rosbridge output that rosbridge is removing unfinished fragment lists after some timeout (in this example 600 seconds from last received fragment). Those fragment lists timed out because I stopped the clients that were sending them.. so that was expected ;-)

[INFO] [WallTime: 1370273146.585986] [Client 18] reconstructed message (ID:19782) from 501 fragments. [duration: 0.056502 s]
[WARN] [WallTime: 1370273151.130282] [Client 18] fragment list 43125 timed out.. -> removing it..
[INFO] [WallTime: 1370273164.030695] [Client 16] reconstructed message (ID:26702) from 501 fragments. [duration: 0.05615 s]
[INFO] [WallTime: 1370273167.155707] [Client 18] reconstructed message (ID:49371) from 501 fragments. [duration: 0.05425 s]
[WARN] [WallTime: 1370273173.643883] [Client 18] fragment list 58538 timed out.. -> removing it..
[INFO] [WallTime: 1370273178.576277] [Client 19] reconstructed message (ID:36115) from 1001 fragments. [duration: 0.217701 s]
[INFO] [WallTime: 1370273184.603897] [Client 16] reconstructed message (ID:56405) from 501 fragments. [duration: 0.05371 s]
[INFO] [WallTime: 1370273187.744155] [Client 18] reconstructed message (ID:51638) from 501 fragments. [duration: 0.056288 s]
[INFO] [WallTime: 1370273205.190626] [Client 16] reconstructed message (ID:21776) from 501 fragments. [duration: 0.056112 s]
[INFO] [WallTime: 1370273206.190306] Client disconnected. 3 clients total.
[INFO] [WallTime: 1370273207.320452] Client connected. 4 clients total.
[INFO] [WallTime: 1370273208.352296] [Client 18] reconstructed message (ID:23422) from 501 fragments. [duration: 0.052343 s]
[INFO] [WallTime: 1370273217.136774] Client disconnected. 3 clients total.
[INFO] [WallTime: 1370273219.816229] [Client 19] reconstructed message (ID:16441) from 1001 fragments. [duration: 0.22642 s]
[INFO] [WallTime: 1370273228.920640] [Client 18] reconstructed message (ID:56127) from 501 fragments. [duration: 0.052594 s]
[INFO] [WallTime: 1370273233.144734] Client connected. 4 clients total.
[INFO] [WallTime: 1370273303.519196] [Client 17] reconstructed message (ID:57722) from 10001 fragments. [duration: 59.186009 s]

[dabertram]

pull request sent..

if you need a test script, I can provide one.. (testing defragmentation AND tcp-server from 2nd pull request)

[dabertram]

I started to work on fragmentation support.

this time the other "direction":

• client can request fragments if any message from rosbridge would be larger than requested fragment_size
• fragment_size can be set to a new value by client at any time, by just adding field fragment_size to any message the client sends to rosbridge
• fragmentation can be disabled by setting adding { "fragment_size": None } to any message sent to rosbridge

it's basically working.. not yet tested much, but first tests run fine

see: d214c3f

[rctoris]

#54