bittorrent / libutp Goto Github PK

I want get it.

After running "make" in either utp_test or utp_file, all of the executables produce similar output to the following:

~/libutp/utp_file$ ./utp_send /tmp/utp.log localhost:1234 utp_config.h
logging to '/tmp/utp.log'
connecting to localhost:1234
sending 'utp_config.h'
creating socket 0x8c5c2d8
connecting socket 0x8c5c2d8
buffering packet: 1 Permission denied
[15135327] sent: 0/734 0.0 bytes/s buffering packet: 2 Permission denied
[15141355] sent: 0/734 0.0 bytes/s socket error: (110) Connection timed out

Nothing is written to the log file. This is on a default Ubuntu install, 11.04.

Things I've tried: sending/receiving to both localhost and a remote box, receiving via nc, running as root. Always the same.

Hi I am trying to write bindings for this library exposing a classical BSD-style socket interface (by doing some buffering in between). I have some questions about the semantics of the on_read callback and the get_read_buffer_size callback.

• AFAICS, the on_read callback is called each time there is available data on the socket. Is there any way to signal that the data passed to the on_read callback has not been completely consumed by the upper layer, or should one always make a copy when it is not consumed right away ?
• When is the get_read_buffer_size callback supposed to return ?
• When should one call the function utp_read_drained ?

Thanks!

Compiling libutp under a Linux running on a x86_64 architecture results in the error below:

g++ -Wall -ansi --shared -o libutp.so -g utp.cpp utp_utils.cpp -DPOSIX -I . -I utp_config_lib -lrt -fno-exceptions -fno-rtti
/usr/bin/ld: /tmp/ccglGf8u.o: relocation R_X86_64_32 against `.text' can not be used when making a shared object; recompile with -fPIC
/tmp/ccglGf8u.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [libutp.so] Error 1

The solution is adding -fPIC when using getting libutp.so in the root Makefile, as in the diff below:

--- makefile.old    2010-11-19 18:33:24.000000000 +0200
+++ makefile    2010-11-19 18:33:09.000000000 +0200
@@ -15 +15 @@
-   g++ -Wall -ansi --shared -o libutp.so -g $(utp) -DPOSIX -I . -I utp_config_lib $(libs) $(cflags)
+   g++ -Wall -ansi -fPIC --shared -o libutp.so -g $(utp) -DPOSIX -I . -I utp_config_lib $(libs) $(cflags)

Can we refactor libutp to support different congestion control flavours?

In part motivated by issues that are inherent to LEDBAT:
http://home.ifi.uio.no/michawe/research/publications/ledbat-impact-letters.pdf

Hi,
While executing libutp I am getting socket error.
Client and server works on same machine.

Reference from transmission:

https://trac.transmissionbt.com/ticket/5143

Quoting rb07 "Technically the problem is an alignment problem with SPARC processors."

This bug was submitted by waby38 at Transmission ticket #5128; I'm passing it upstream.

Some arch like SH4 don't like / support unaligned access, others like x86/ARM support it but with poor performance.

In this extracted part of code, in get_familly() call, "&_in._in6addr" can be unaligned, and will be used by the IN6_IS_ADDR_V4MAPPED macro who make uint32_t access. This generate unaligned memory access which end up with exception on SH4, and poor performance on others arch.

#define IN6_IS_ADDR_V4MAPPED(a) \
        ((((__const uint32_t *) (a))[0] == 0)                                 \
         && (((__const uint32_t *) (a))[1] == 0)                              \
         && (((__const uint32_t *) (a))[2] == htonl (0xffff)))

struct PACKED_ATTRIBUTE PackedSockAddr {

        // The values are always stored here in network byte order
        union {
                byte _in6[16];          // IPv6
                uint16 _in6w[8];        // IPv6, word based (for convenience)
                uint32 _in6d[4];        // Dword access
                in6_addr _in6addr;      // For convenience
        } _in;

        (...)

        byte get_family() const
        {
                return (IN6_IS_ADDR_V4MAPPED(&_in._in6addr) != 0) ? AF_INET : AF_INET6;
        }
};

I 've make a small patch to fix it. (gcc only ?)

--- third-party/libutp/utp.cpp~ 2012-10-24 15:01:08.000000000 +0200
+++ third-party/libutp/utp.cpp  2012-11-07 12:07:02.000000000 +0100
@@ -194,7 +194,7 @@ struct PACKED_ATTRIBUTE PackedSockAddr {
        snprintf(i, len - (i-s), ":%u", _port);
        return s;
    }
-};
+} __attribute__ ((aligned (4)));

 struct PACKED_ATTRIBUTE RST_Info {
    PackedSockAddr addr;

Similar to: https://tools.ietf.org/html/rfc7413

We already have extension bits, so this should be fairly similar to TCP. For the listening side, a new utp_setsockopt() option just like TCP_FASTOPEN can be used during the utp_call_on_accept callback. For the connecting side, utp_setsockupt() option like TCP_FASTOPEN_CONNECT tells the utp_connect() to "succeed" immediately and the fire utp_write() will initiate the connection with the initial data.

If a connection is accepted with ConnId 0xFFFF it will store send id as 0x10000 and will not be able to receive any data due to not being able to find a socket for 0x0000. Similar issues with ST_RESET trying to figure out what id it is but cant due to overflow/underflow.

I consider making a bug fix for this but I dunno what the right thing todo is for this lib.

Either just simple fix it and ignore the fact that there might be buggy versions of the lib in use or
the other idea I had was to make 0xFFFF reserved. As in sending ST_RESET for ST_SYN(connId=0xFFFF) and generating in the range of 0-0xFFFE instead of 0xFFFF.

Either way the overflow issues will have to be fixed by masking (id+/-1)&0xFFFF

Repro code:

utp_process_udp(ctx, (byte*)"\x41\x00\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x01\x00\x00", 20, (sockaddr*)&addr, sizeof(addr));
utp_process_udp(ctx, (byte*)"\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x10\x00\x00\x00\x02\x00\0Bug\x00", 24, (sockaddr*)&addr, sizeof(addr));

I'm attempting to build the test example on Mac OS X, however it appears that it wants to link with librt, which is not available (at least on my machine). Apparently libcsc has superseded librt, so presumably this ( http://sourceforge.net/projects/hackerlabs/ ) could be installed to provide the needed functionality. If this is the case, perhaps this should be added to the build instructions?

$ make
g++ -Wall -ansi -o utp_test -g utp_test.cpp ../utp.cpp ../utp_utils.cpp -DPOSIX -I . -I .. -lrt -fno-exceptions -fno-rtti
In file included from ../utp.h:4,
from utp_test.cpp:62:
../utypes.h:30:1: warning: "INT64_MAX" redefined
In file included from /usr/include/arpa/inet.h:90,
from utp_test.cpp:26:
/usr/lib/gcc/i686-apple-darwin10/4.2.1/include/stdint.h:122:1: warning: this is the location of the previous definition
../utp.cpp: In member function ‘void UTPSocket::send_ack(bool)’:
../utp.cpp:922: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘void UTPSocket::send_packet(OutgoingPacket_)’:
../utp.cpp:1019: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘bool UTPSocket::is_writable(size_t)’:
../utp.cpp:1054: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘void UTPSocket::update_send_quota()’:
../utp.cpp:1209: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘void UTPSocket::check_timeouts()’:
../utp.cpp:1364: warning: comparison between signed and unsigned integer expressions
../utp.cpp:1370: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘int UTPSocket::selective_ack_bytes(uint, const byte_, uint, int64&)’:
../utp.cpp:1489: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In member function ‘void UTPSocket::selective_ack(uint, const byte_, uint)’:
../utp.cpp:1559: warning: comparison between signed and unsigned integer expressions
../utp.cpp: In function ‘uint UTP_ProcessIncoming(UTPSocket_, const byte_, size_t, bool)’:
../utp.cpp:2036: warning: comparison between signed and unsigned integer expressions
ld: library not found for -lrt
collect2: ld returned 1 exit status
make: *_* [utp_test] Error 1

I would like to point out that identifiers like “__UTP_H__” and “__UTP_TYPES_H__” do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

Not really a feature request, just an idea that came up when discussing custom UDP transports:

At least on linux (edit: also on windows ) there is the IP_RECVTOS socket option that allow reading TOS and thus ECN flags for received packets, which could be used to implement ECN in user space. That way routers won't be forced to drop uTP packets more aggressively than they do for TCP connections.

http://www.scmagazineuk.com/bittorrent-moves-to-patch-reflective-ddos-attack-flaw/article/435396/

is the exploit described here filed already in the code here?

I took a while reviewing the execution flow for libutp as a drop in addition for our bittorrent application. Our application is pretty heavily multi-threaded at this point, so I have to keep an eye on issues of thread safety.

It seems that there are a number of possibilities for multiple threads using the libutp library to cause problems. For instance:

as utp connections are created and destroyed the g_utp_sockets Array<UTPSocket*> will grow and shrink moving around members of the array to maintain a tight list. However, loops such as the one in UTP_IsIncomingUTP expect the state of this list to be unmangled for the duration of the loop. Resulting bugs range from not matching incoming packets to legitimate sockets or to NULL dereferences and other random memory corruption/access.

In my current test case I am reading/writing packets to/from different UDP sockets on each thread so as to insulate the code from issues which may be caused by having multiple threads reading from the same system level socket in parallel.

Is it a current or eventual design goal to allow multi-threaded use of libutp?

This callback signature lacks any way of passing "userdata" through to the callback.

In our specific case, we have more than one UDP port listening for incoming data and we need to know which port a new connection came from. From a user perspective the UTPSocket is an opaque value and there is no externally available method for retrieving data other than stats and peer name).

From inspecting the code, it seems that the time span in which this callback needs to be valid is confined to the wrapping UTP_IsIncomingUTP call. So, I can store my userdata as a global during that call as a workaround (correct?).

However, It seems that this callback function signature (and the functionality surrounding its use) should provide for passing userdata through to the callback as this is the only callback that lacks that feature.

Despite readme saying that "the external interface is strictly C (ANSI C89)", newly added utp_callback_arguments struct (utp.h) contains two anonymous unions which weren't standardized up until C11. Please either make the header C89-compliant or adjust the readme to state the real standard you're aiming at.

I have made a easy test, and found that maybe not every packet been acked. But the program will work correct?

In my test,I use the log code in UTP_ProcessIncoming() to get the info of every package.

the log code in UTP_ProcessIncoming() like that

LOG_UTPV("0x%08x: Got %s. seq_nr:%u ack_nr:%u state:%s version:%u timestamp:"I64u" reply_micro:%u",
conn, flagnames[pk_flags], pk_seq_nr, pk_ack_nr, statenames[conn->state], conn->version,
conn->version == 0?(uint64)(pf->tv_sec) * 1000000 + pf->tv_usec:uint64(pf1->tv_usec),
conn->version == 0?(uint32)(pf->reply_micro):(uint32)(pf1->reply_micro));

and the test log is like this:

[76] 0x009c6730: Got ST_SYN. seq_nr:18467 ack_nr:0 state:CONNECTED version:1 timestamp:93 reply_micro:0
[170] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18467 state:SYN_SENT version:1 timestamp:77088 reply_micro:0
[170] 0x009c6730: Got ST_DATA. seq_nr:18468 ack_nr:40 state:CONNECTED version:1 timestamp:170878 reply_micro:93702
[370] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18468 state:CONNECTED_FULL version:1 timestamp:277095 reply_micro:201

[371] 0x009c6730: Got ST_DATA. seq_nr:18469 ack_nr:40 state:CONNECTED version:1 timestamp:371079 reply_micro:93893
[371] 0x009c6730: Got ST_DATA. seq_nr:18470 ack_nr:40 state:CONNECTED version:1 timestamp:371140 reply_micro:93893
[371] 0x009c6730: Got ST_DATA. seq_nr:18471 ack_nr:40 state:CONNECTED version:1 timestamp:371161 reply_micro:93893
[470] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18470 state:CONNECTED_FULL version:1 timestamp:371652 reply_micro:374

[470] 0x009c6730: Got ST_DATA. seq_nr:18472 ack_nr:40 state:CONNECTED version:1 timestamp:470701 reply_micro:98951
[471] 0x009c6730: Got ST_DATA. seq_nr:18473 ack_nr:40 state:CONNECTED version:1 timestamp:470776 reply_micro:98951
[471] 0x009c6730: Got ST_DATA. seq_nr:18474 ack_nr:40 state:CONNECTED version:1 timestamp:470806 reply_micro:98951
[571] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18472 state:CONNECTED_FULL version:1 timestamp:470997 reply_micro:286

[571] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18474 state:CONNECTED_FULL version:1 timestamp:471735 reply_micro:898

[572] 0x009c6730: Got ST_DATA. seq_nr:18475 ack_nr:40 state:CONNECTED version:1 timestamp:571560 reply_micro:100480
[572] 0x009c6730: Got ST_DATA. seq_nr:18476 ack_nr:40 state:CONNECTED version:1 timestamp:571627 reply_micro:100480
[572] 0x009c6730: Got ST_DATA. seq_nr:18477 ack_nr:40 state:CONNECTED version:1 timestamp:571649 reply_micro:100480
[670] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18476 state:FIN_SENT version:1 timestamp:572760 reply_micro:1058

[670] 0x009c6730: Got ST_DATA. seq_nr:18478 ack_nr:40 state:CONNECTED version:1 timestamp:572173 reply_micro:100064
[671] 0x009c6730: Got ST_DATA. seq_nr:18479 ack_nr:40 state:CONNECTED version:1 timestamp:572227 reply_micro:100064
[771] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18478 state:FIN_SENT version:1 timestamp:671062 reply_micro:98877

870] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18479 state:FIN_SENT version:1 timestamp:771687 reply_micro:99294

[871] 0x009c6730: Got ST_FIN. seq_nr:18480 ack_nr:40 state:CONNECTED version:1 timestamp:871286 reply_micro:99535
[970] 0x009c54d0: Got ST_STATE. seq_nr:41 ack_nr:18480 state:FIN_SENT version:1 timestamp:871496 reply_micro:201

cfpp2p@39d0263

This bug was submitted by mmain Transmission ticket #5232 I'm passing it upstream.

I have been successfully using Transmission on Centos 5.8, but now need to
 upgrade to Centos 6.3 and have started building new machine with 6.3 loaded
 and I am unable to build Transmission from source without following error
 during make(I also tried 2.75 with same
result)

tr-utp.c:65: error: conflicting types for ‘UTP_Write’
../third-party/libutp/utp.h:116: note:
 previous declaration of ‘UTP_Write’ was here

make[1]: * [tr-utp.o] Error 1
make[1]: Leaving directory `/root/transmission-2.76/libtransmission'
make: * [all-recursive] Error 1

...

Changed 16 hours ago by x190

UTP_Write() is type bool. Maybe utp.h needs the following bit of voodoo that
 was added to transmission.h.

#if !defined (__cplusplus)
 #ifdef HAVE_STDBOOL_H
  #include <stdbool.h>
 #elif !defined (__bool_true_false_are_defined)
  #define bool uint8_t
  #define true 1
  #define false 0
 #endif
#endif


comment:5 Changed 13 hours ago by mmain

Yes, this worked a treat. Many Thanks


comment:6 Changed 12 hours ago by jordan

x190: that's the great idea, you ought to submit that upstream to the
 libutp project on github.

It seems to me that the counters in utp_context_stats needs to be bigger than uint32 (the current type).
I was playing with a seeding application using the libutp. I noticed that for a single 1.4GB file the count of the smallest received packets (<= 23 Bytes, ACKS) reached over a million. Same was true for the count of biggest sent packets (> 1400 Bytes). I can imagine that for 24/7 application these 32 bit counters will overflow in few (1-3) weeks.

hello :

can you give me a sample about using the lastest version of libutp (master version)?

I have writen a wrapper class for the head version ...

but using the master version ,the client can send one packages, then it can't send another packages...

the log is :

callback_log 988 006CC330 127.0.0.1:10000 000041 MTU [TIMEOUT]
callback_log 988 006CC330 127.0.0.1:10000 000041 Packet timeout. Resend. seq_nr:18468. timeout:4000 max_window:1382 cur_window_packets:2
callback_log 988 006CC330 127.0.0.1:10000 000041 MTU [RESET] floor:576 ceiling:1402 current:-842150451
callback_log 988 006CC330 127.0.0.1:10000 000041 UTP_Connect conn_seed:41 packet_size:1435 (B) target_delay:100 (ms) delay_history:3 delay_base_history:13 (minutes)
callback_log 988 006CC330 127.0.0.1:10000 000041 Packet timeout. Resend. seq_nr:18468. timeout:2000 max_window:1382 cur_window_packets:2

Thanks!

a single-threaded can work as both client and server at the same time?

in the introduce of uTP,in means that the libutp interface is not thread-safe,but now,I want to use uTP for a p2p mode and work as both client and server at the same time,how can I do it?

An extra null pointer check is not needed in the function “utp_destroy”.

I've been looking over a lot of the example code provided by the uTP library and would suggest that the documentation surrounding UTPGetRBSize and the UTPOnReadProc be made more clear. The name of the callback is a bit misleading as the value you return is the not really the size of the read buffer, but the number of bytes occupied in the read buffer. This is further confused by the rather cryptic comment in pyutp/utp/utp_twisted.py line 73. Also it should be made clearer that the actual size in bytes of the read and write buffers have to be specified by using the UTP_SetSockopt function. I think there is unequal attention given in the examples using the UTPOnWriteProc callback with a buffer and not even one example using the UTPOnReadProc callback utilizing a buffer. Also it would be helpful to document where in this scenario it is appropriate to use UTP_RBDrained. Hopefully this feedback will prove helpful.

I'm seeing a case where if I leave SO_SNDBUF on the sending socket to the default for version 1 (~3MB) or set it to a to 1MB, a utp transfer hangs and never. I was able to reproduce with utp_send and utp_recv by just commenting out this line:

https://github.com/bittorrent/libutp/blob/master/utp_file/utp_send.cpp#L367

and transferring a large file ~300MB. If I set the SO_SNDBUF to 100*300 the problem goes away. Here are the log entries from utp_send right before the hang:

[3625] 0x00851b88: rtt:19 avg:23 var:3 rto:500
[3625] 0x00851b88: acks:2 acked_bytes:2764 seq_nr:44962 cur_window:1382 cur_window_packets:1 quota:-21293032
[3625] 0x00851b88: CheckTimeouts timeout:500 max_window:464223 cur_window:1382 quota:-21293032 state:CONNECTED_FULL cur_window_packets:1 bytes_since_ack:0 ack_time:-1879048172
[3686] 0x00851b88: CheckTimeouts timeout:439 max_window:464223 cur_window:1382 quota:-21293032 state:CONNECTED_FULL cur_window_packets:1 bytes_since_ack:0 ack_time:-1879048111
[3718] recv 127.0.0.1:7777 len:20 id:41
[3718] recv id:41 seq_nr:41 ack_nr:44961
[3718] 0x00851b88: recv processing
[3718] 0x00851b88: Got ST_STATE. seq_nr:41 ack_nr:44961 state:CONNECTED_FULL version:1 timestamp:7658418 reply_micro:3940078
[3718] 0x00851b88: acks:1 acked_bytes:1382 seq_nr:44962 cur_window:1382 cur_window_packets:1 relative_seqnr:0 max_window:464223 min_rtt:112 rtt:23
[3718] 0x00851b88: actual_delay:3940078 our_delay:0 their_delay:0 off_target:99 max_window:464231 delay_base:3939998 delay_sum:0 target_delay:100 acked_bytes:1382 cur_window:0 scaled_gain:8.923907 rtt:23 rate:9284620 quota:-20991287 wnduser:3670016 rto:500 timeout:407 get_microseconds:3718642 cur_window_packets:1 packet_size:1382 their_delay_base:4291027386 their_actual_delay:4291027503
[3718] 0x00851b88: fast_resend_seq_nr:44962
[3718] 0x00851b88: got ack for:44961 (pkt_size:1382 need_resend:0)
[3718] 0x00851b88: rtt:112 avg:35 var:24 rto:500
[3718] 0x00851b88: acks:1 acked_bytes:1382 seq_nr:44962 cur_window:0 cur_window_packets:0 quota:-20991287
[3718] 0x00851b88: CheckTimeouts timeout:500 max_window:464231 cur_window:0 quota:-20991287 state:CONNECTED_FULL cur_window_packets:0 bytes_since_ack:0 ack_time:-1879048079

I'm not sure I follow why this is the case. Should we always set SO_SNDBUF?

ticket mirrored from downstream:
https://trac.transmissionbt.com/ticket/4260

At least on the NAS devices Conceptronic CH3SNAS and DNS-323, µTP is not working. These devices run libuClibc 0.9.29, and Transmission was compiled with gcc 4.1.3.

Transmission does not crash, but the log is filled with "Unexpected UDP packet" messages.

The reason for this is that the structs defined in libutp don't have the correct size on this platform.

I've attached a patch to fix the problem, although I can't speak for other platforms, so this should be tested beforehand.

We actually found this issue a while ago during an IRC session (I believe livings was involved), and we found the solution right then and there. But apparently it was not checked into the trunk / reported upstream.

I captured uTP traffic through wireshark and saw

data part of UDP, either it starts with -

01 00 e2 18 7f 64 .Lv.x1.K .......d
0030 6a e0 56 9d 87 84 00 38 00 00 38 cd 31 45 4b df j.V....8 ..8.1EK.
0040 45 94 54 af f0 57 89 a6 e3 f4 3f 9d 0b 6d 3f 5b E.T..W.. ..?..m?[
0050 d0 7d 90 99 db 45 4c 34 72 4a d5 c9 03 08 57 c3 .}...EL4 rJ....W.
0060 8d 93 84 d0 62 77 87 29 1b ba ef 10 6c ....bw.) ....l

or

21 00 42 ff 7f 55 .Lv.x1.. ..!.B..U
0030 c0 e1 56 9d 85 1d 00 38 00 00 33 46 f0 98 ..V....8 ..3F..

Now,

According to the uTP draft, version is of 4 bits but here its says 01 that is 1 byte.

Most of the packets starts with 01.But what are those packets which not start with 01?

In UTP_IsIncomingUTP incoming messages are routed by source sockaddr.

In the case where our client has access to multiple outbound devices for load balancing we may have multiple utp connections to/from the same address with different local IP's. These messages all end up being pushed to the first created UTPSocket for a given peer.

The TCP case works as the kernel (in linux at least) routes based on a 5-tuple of source/dest IP/port + protocol. The protocol is redundant in this case.

I'm currently trying to test your implementation of uTP using the Python wrapper you provided. In doing so I encountered two problems: the first being that the struct.unpack call would throw this exception:

IPAddr(struct.unpack("L", socket.inet_aton(ip))[0])
struct.error: unpack requires a string argument of length 8

This at "sockaddr_types.py" line 80.

However, I managed to solve that by changing the format of the unpack (fmt) to network(big endian) "!L".

After that, I did not get that message anymore BUT python crashes due to a segmentation fault in the underlying utp implementation. I'm particularly expert in
programming python applications which interface with C/C++ but I guess a wrong pointer is passed to the C uTP library. The library crashes when calling:

utp.UTP_SetCallbacks(self.utp_socket,
ctypes.byref(f), ctypes.py_object(self))

at line 92 in the Socket class in file utp_socket.py. I attach to this e-mail the stack trace of the error.

I'm currently using the pre-installed Python 2.6 under Mac OS Snow Leopard latest version. I tried also the last Python 2.7 from the official releases and I got
the same kind of behavior. Then I tried to install it in a Linux machine and all went fine. However, I do really need to get this working on all platforms, windows
included.

Thank you very much. I really appreciate your work.

in utp.cpp, get_value() returns a uint32, but there are three cases where it's asserted that it returns >= 0:

utp.cpp: assert(our_hist.get_value() >= 0);
utp.cpp: assert(conn->our_hist.get_value() >= 0);
utp.cpp: assert(conn->our_hist.get_value() >= 0);

Which isn't a big deal, but it's probably not what you intended.

utp.cpp: In member function ‘void UTPSocket::apply_ledbat_ccontrol(size_t, uint32, int64)’:
utp.cpp:1631: warning: comparison of unsigned expression >= 0 is always true
utp.cpp: In function ‘size_t UTP_ProcessIncoming(UTPSocket_, const byte_, size_t, bool)’:
utp.cpp:1945: warning: comparison of unsigned expression >= 0 is always true
utp.cpp:1952: warning: comparison of unsigned expression >= 0 is always true

printed by ucat:

debug: Allocatd 4096 buffer
debug: Bound to local 0.0.0.0:23333
debug: UTP context 0xab5c80
debug: Received 20 byte UDP packet from 127.0.0.1:45886
log: recv 127.0.0.1:45886 len:20 id:17767
log: recv id:17767 seq_nr:9158 ack_nr:0
log: Incoming connection from 127.0.0.1:45886
debug: Firewall allowing inbound connection
log: 0xab61d0 127.0.0.1:45886 017768 MTU [RESET] floor:576 ceiling:1402 current:0
log: 0xab61d0 127.0.0.1:45886 017768 UTP socket initialized
log: 0xab61d0 127.0.0.1:45886 017768 Got ST_SYN. seq_nr:9158 ack_nr:0 state:SYN_RECV timestamp:2970548573 reply_micro:0
log: recv send connect ACK
log: 0xab61d0 127.0.0.1:45886 017768 Sending ACK 9158 [17767]
log: 0xab61d0 127.0.0.1:45886 017768 send 127.0.0.1:45886 len:20 id:17767 timestamp:37330287042 reply_micro:0 flags:ST_STATE seq_nr:17767 ack_nr:9158
debug: sendto: 16 byte packet to 127.0.0.1:45886
debug: Accepted inbound socket 0xab61d0
log: 0xab61d0 127.0.0.1:45886 017768 CheckTimeouts timeout:-37330287 max_window:1382 cur_window:0 state:SYN_RECV cur_window_packets:0

Is that an expected behaviour? According to the source it seems that when timeout in CS_SYN_RECV state the connection should be closed.

The README.md file mentions tests in a utp_test subdirectory, but there is no such directory and no tests.

I bisected the history and found out the sub-directory was deleted in dc14938 without the README being updated.

As a matter of interest, why were the tests removed in the first place? There also seems to have been a Python binding (a pyutp sub-directory) which has also been removed. Why in the name of all that is Pythonic? :)

Using the latest version of Transmission (2.77+ rev.14042).

Testing a 64-bit build done with MinGW-w64 compiler.

With no change to Windows 7 64-bit (SP1 + all patches) the application fails to transfer anything using uTP protocol. It does connect to peers, but after some time peers seem to drop the connection, and libutp starts working with closed sockets, then Transmission drops the peer.

Running the utp_test (built with the current git code, and using the just built library, not the one from the code in Transmission) results in a time-out (on one side).

Changing Windows to use the hardware clock (probably HPET, but there is no setting in the BIOS so I just used "bcdedit /set useplatformclock true" and the QueryPerformanceFrequency went from around 3 to around 14 MHz) and running the application again results in normal operation, transfers are using uTP protocol.

Processor are Xeon X5460 with SpeedStep enabled.

The problem sounds very similar to the one fixed some time ago, about using QueryPerformanceCounter and timing not being progressive.

Same application built for 32-bit works with no problem like this (in fact has been working with all versions that have uTP, for years, about 10,000 users download each version from SourceForge). The 32-bit build uses the MinGW compiler/runtime/headers, not the MinGW-w64 ones.

This is coming from:

http://trac.transmissionbt.com/ticket/4915
http://bugs.gentoo.org/400929

Build log of the failure (both are same links, but in case you can't connect to one of them, I'll post both):

http://trac.transmissionbt.com/attachment/ticket/4915/attachment.cgi%3Fid%3D312311
http://bugs.gentoo.org/attachment.cgi?id=311957

Patch solving the build problems on FreeBSD. It's simple addition of 2 #include lines. The patch works for Linux too (as in, doesn't cause problems).

http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-p2p/transmission/files/transmission-2.51-fbsd.patch?view=markup

shared library, grab environment AR, CXX, *FLAGS, ..., all for packaging purposes so I can make the transmission p2p client use the system copy of libutp

would help a much if this was committed. grab the patch from here:

http://dev.gentoo.org/~ssuominen/libutp-Makefile.patch

The pack #pragma statements are non-portable (and not supported even by gcc on Solaris). Using pack(1) / pack(0) instead works.

Hey,

please remove librt from the test-suite, so that it compiles with OSX. I tried removing it, but didn't have time investigating on the segfaults that were caused by its removal.

It appears that the resends[] buffer in selective_ack() is either mis-sized or not appropriately constrained. It is hard-coded to a stack of 32 entries, but no check is made that we won't run past this limit. The code simply relies on the declared size of the selective_ack in the packet itself (ugh)

For instance:

// Resend segments
// if count is less than our re-send limit, we haven't seen enough
// acked packets in front of this one to warrant a re-send.
// if count == 0, we're still going through the tail of zeroes
if (((v - fast_resend_seq_nr) & ACK_NR_MASK) <= OUTGOING_BUFFER_MAX_SIZE &&
    count >= DUPLICATE_ACKS_BEFORE_RESEND &&
    duplicate_ack < DUPLICATE_ACKS_BEFORE_RESEND) {
    resends[nr++] = v;
    LOG_UTPV("0x%08x: no ack for %u", this, v);
}

This is tripped regularly in large windows.

Hello,

As in the subject, there seems to be a bug in libutp: if a connection is opened between a client and a server and the client closes the connection without sending any data, then the server does not get a STATE_EOF state change notification so the socket is kept around forever.

To reproduce, is enough to use ucat on the server as follows:

ucat -d -l -p 5678

and on the client as follows:

ucat -d 127.0.0.1 5678

and close (CTRL-D) the client connection without sending any data. The client will fail with ETIMEDOUT and the server will not get the STATE_EOF notification.

If utp_create_socket is called, and then utp_connect is not called, there's no way to reclaim the utp_socket memory. If the parent utp_context is destroyed, the memory for the unconnected utp_socket is not reclaimed, and it's no longer possible to call utp_connect.

sock = utp_create_socket(ctx)
utp_destroy(ctx)
# segfaults, no way to reclaim sock memory
utp_connect(sock)

Here's another sequence:

sock = utp_create_socket(ctx)
utp_destroy(ctx)
# asserts
utp_close(sock)

I think utp_connect should not segfault if the parent utp_context is destroyed, and utp_close should probably delete the utp_socket if it's uninitialized.

Where to find examples for the current source version?
The files reffered in the readme are not longer present in this repository.

libutp currently keeps quite a lot of state in global variables, and as a result a single instance of the libutp library can only be used for a single host. This makes it difficult to integrate libutp in simulators like Shadow [1] which implement multiple hosts in a single address space.

One approach is to move the global state into a single variable, and switch a pointer. This is the approach taken in sjmurdoch@d6377a7

The advantage of this approach is that existing API calls remain unchanged; only new ones are added and it would be possible to keep existing code working. The disadvantage is that it is not thread-safe; even when different threads are handling different nodes.

Thread safety would be very desirable, but to achieve this the API would need to be changed to have a state object passed into most calls to the library. As a result code using the existing API would need to be updated.

So my question is whether either of these options would be considered for libutp, or whether the approach of using global variables is decided on for the foreseeable future.

[1] https://shadow.cs.umn.edu/

Sorry if this is the wrong place for this, but I think I may not be understanding the API of libutp, but my UTP_Write call is not writing. My general flow in the program is:

UTP_Create with my send_to function registered. send_to calls a udp sendto function, using the provided const byte *p as the data sent by the sendto function.
UTP_SetCallbacks with a utp_write function registered. utp_write only does housekeeping/ data statistics
UTP_Write with the socket and number of bytes I want to send.

Where does UTP_Write get the specific bytes that I want to write?

The test applications use a recv loop with a UTP_IsIncomingUTP, but I don't understand what the latter function does, or why I would need to receive if all I want to do is send.

Thanks.

utp.cpp has following warning:

QA Notice: Package has poor programming practices which may compile
fine but exhibit random runtime failures.
utp.cpp:123:59: warning: dereferencing type-punned pointer will break strict-aliasing rules 

These are known to break with recent compiler. If interested, take a look at this blog posts:

http://blog.llvm.org/2011/05/what-every-c-programmer-should-know.html

Or there are much more clarifications in google...

The -ansi parameter caused snprintf() to not be defined (in a very strict environment)... and compilation of utp.cpp fails.

snprintf() is defined in C99 but that is not a valid option for g++ (i.e. -std=c99 is the same as using no -ansi, but gets a warning).

Is there mayba an utp multi-path implementation that can accept multiple udp connection and gives in-order delivery? Or maybe a similar implementation?