elektito / bttools

A set of Python tools for experimenting with the BitTorrent protocol.

• I am moving to Sourcehut.
• I might leave my older projects, and even mirror some newer ones, here. That said, Sourcehut is going to be my main home for source code.

bttools's Issues

Use logging instead of print.

Use the Python logging facilities instead of print.

Timeout flows properly.

The flows are not being timed out at the moment at all. If a flow remains open for a relatively long period of time without any communication in it, it must be removed from the flow table.

Refactor tracer common code into a separate module.

Refactor utptrace and tcptrace code so that their common code is moved into a separate, shared module.

Better logging in btparser.

In order to make btparser log machine-parsable, make the following changes to it:

• When a new file is going to be parsed (whether with the -d or -f option) its path should be logged:

  [NEW FILE] /home/foobar/stream-0-192.168.1.100-33465-192-168.1.200-54654

• When an error or warning is logged, it should be clearly marked:

  [ERROR] Invalid BitTorrent stream.

Write a log analyzer for the tracers.

Write a log analyzer for UtpTracer and TcpTracer. The analyzer should report the following:

• What error messages were encountered? How many times each?
• How much traffic was thrown away in total?
• How much traffic is pending at the end?

Also, for each flow, the following should be reported:

• How many packets were thrown away for this particular flow in total.
• How many packets were pending for that flow when it was being closed or at the end.

Flows with too many pending packets should be blacklisted.

Flows with too many pending packets should be blacklisted, that is all memory belonging to them should be freed except for the minimum amount needed to track the flow until it is either properly closed or timed out.

Always raise an exception for unexpected end of stream in BitTorrentParser.

Currently, only parse_message_request checks for an unexpected end of stream. Do this in parse_message for all message types. Simply check if there are enough bytes as claimed, and if not, raise an exception.

Write a log analyzer for btparser.

For each file btparser parses, the log analyzer should output:

• How much "piece data" was there (in bytes)?
• How much header data was there (in bytes)?
• Was there an error in the file (for example a very large block length)?

Also at the end the following should be output:

• What infohashes were encountered?
• What pieces for each infohash were encountered (even partially).
• How many blocks for each piece were observed? How many unique blocks.

utp extensions are not taken into account.

Currently if a utp packet contains extensions, we will read the packet erroneously. At the very least, we should skip the extension headers.

There are way too many "Invalid connid" errors .

There are too many "Invalid connid" errors. An inspection shows that sometimes a connection is shortly established and half-closed in a "SYN=>STATE=>FIN" sequence. When this happens, an entry is kept in the flow table for the flow and anything afterwards is ignored, even though a new connection can be established with a new connid.

Making connid a part of the flow tuple-id can fix this.

Implement command-line options for btparser.

btparser.py should accept the following command line arguments:

• --torrent or -t: a torrent file from which torrent information is read and associated with its infohash. Can be specified multiple times.
• --file or -f: a file containing a single BitTorrent stream to parse.
• --directory or -d: all the files in the given directory are processed.

Either -d or -f must be specified.

Better formatting for tracer logs.

Make UtpTracer and TcpTracer to format their logs like this: "[packet number][flow] message". For example when a new flow created, something like this should be logged:

[1681][192.168.1.100:53456 => 192.168.1.200:38765] New flow.

Sequence numbers are not correctly handled.

Sequence numbers in the uTP and TCP tracers are not handled correctly. The addition operations used are correct but the comparisons are not. See this wikipedia article and also RFC 1982 for more information.

IP fragmentation is not handled.

Some of the input files have been corrupt for no apparent reason. Inspection showed that this is due to IP packet fragmentation. Implement an IPv4 defragmenter in order to fix this.

Add some tests.

We sorely need tests. Add some.