Message stream encryption about bt

[atomashpolskiy]

@the8472 , I have a question regarding MSE, that, I believe, you might help me with.

The question is:
Is there any alternative of crypto_flags (which is used to indicate encryption support on a per-peer basis in HTTP tracker response) for UDP trackers? http://bittorrent.org/beps/bep_0015.html states that additional fields may be appended to the end of the message (i.e. for announce response that would be after the list of address/port pairs). And http://bittorrent.org/beps/bep_0041.html provides support for passing URL query parameters to UDP trackers (in this case supportcrypto=1). So it should be trivial to add support for crypto_flags for UDP trackers, but surprisingly it's not present in popular clients/trackers (for now I've looked in libtorrent's code only). Am I correct?

Thanks in advance!

[the8472]

Is there any alternative of crypto_flags (which is used to indicate encryption support on a per-peer basis in HTTP tracker response) for UDP trackers?

No. Currently the most reliable source for encryption preference flags is the extended handshake which is then propagated through PEX.

So it should be trivial to add support for crypto_flags for UDP trackers, but surprisingly it's not present in popular clients/trackers (for now I've looked in libtorrent's code only). Am I correct?

Since the UDP tracker protocol is a binary protocol it would require the definition of an new packet layout to get such flags into an announce response.

[atomashpolskiy]

@the8472 , thanks for a quick response!

No. Currently the most reliable source for encryption preference flags is the extended handshake which is then propagated through PEX.

Do you happen to know how the encryption support should be advertised in the extended handshake? It's mentioned in http://bittorrent.org/beps/bep_0011.html:

Flags are defined as follows:
0x01 prefers encryption, as indicated by e field in extension handshake

I assume this e field should be in the message root (not in the m dictionary), but what are the allowed values?

Since the UDP tracker protocol is a binary protocol it would require the definition of an new packet layout to get such flags into an announce response.

Yeah, certainly, but it seems like the future changes were expected:

Do not expect packets to be exactly of a certain size. Future extensions could increase the size of packets.

Extension bits or a version field are not included. Clients and trackers should not assume packets to be of a certain size. This way, additional fields can be added without breaking compatibility.

Basically, possibility of adding something to the end of the announce response depends on whether the length of the list of peers (address/port pairs) is a well-defined function of numwant and IP address family. Say, numwant * 6 and numwant * 18 with zero-padding (in case there are insufficient peers) for IPv4 and IPv6 respectively. Though I'm not sure that existing clients/trackers use this math and not just treat all trailing bytes as a list of peers.

By the way, why MSE hasn't been officially made a BEP?

[atomashpolskiy]

Do you happen to know how the encryption support should be advertised in the extended handshake?

Found this:
https://trac.transmissionbt.com/browser/trunk/extras/extended-messaging.txt
So e flag is just an int with possible values being 1 or 0. Nothing fancy here.

However it's not clear whether it makes sense for a client that participates in a plaintext connection to drop it and re-connect in case of receiving an extended handshake with e being set to 1. One obvious condition is whether the peer is reachable (i.e. client's connection is outbound). Less obvious is peer's "ban" strategy: abruptly dropping a connection might cause the client to be "blacklisted" by the peer for some time. I guess I need to experiment with different approaches.

[the8472]

By the way, why MSE hasn't been officially made a BEP?

Well, it's not deemed as important anymore as it used to be. It was mostly designed in the wake of comcast injecting RST packets to terminate bittorrent connections. Aggressive traffic shaping vs. obfuscation is an arms race. The companies that profit from such boxes have invested additional time, so after a while they - presumably - switched to traffic flow matching and similar heuristics. On the bittorrent side interest in fighting that arms race wasn't that big. So the usefulness of encryption has declined over time.

µTP is seen as carrot approach instead of a stick.

However it's not clear whether it makes sense for a client that participates in a plaintext connection to drop it and re-connect in case of receiving an extended handshake with e being set to 1.

It's not really required. If the peer that wants encryption wants to do it they can do it. The peer that merely supports encryption but does not require it shouldn't waste resources on reestablishing connections.

[atomashpolskiy]

Finally got everything working, no bittorrent traffic in Wireshark, only TCP PSHs and ACKs, and occasional SMTP packets in between 😸 Doesn't seem to have affected the throughput at all.
Still need to brush things up a bit and write more tests and docs. Hopefully I'll be able to add this to master by the end of the next week and follow up with creating a dedicated tutorial page on the project's website. Phew!

[atomashpolskiy]

Added to master and posted a tutorial at http://atomashpolskiy.github.io/bt/encryption/