Compartmentalization/files API simplification about taipei-torrent

[Zilog8]

Modification of active pieces & FileStore

I like the idea from a code perspective, but I'm still a bit unsure about the memory-use implications. Memory use would basically be unbounded (depending on how many open ActivePieces there are, and the size of the torrent's pieces). Although I guess if that were to become a problem, it could be mitigated through other means (off the top of my head, maybe choose at runtime if ActivePiece should use a []byte or file for storage), and we would still have the benefit of simpler code.

If the client is purely leeching, there is no need for a cache, even if the FileStore is remote

BitTorrent file distribution "works" because no client can/should be "purely leeching".

Otherwise, looks good to me.

[rakoo]

100% agree, this proposal makes memory usage unbounded. Here's another proposal: keep an upper boundary of memory used for all ActivePieces, and in RequestBlock, prevent from creating a new ActivePiece if we have already reached the limit. Somewhere around here, check if the sum of active pieces' content + the potential new piece goes beyond the threshold; if true, don't request a new piece, just wait for the other ones to be retrieved, committed to disk and then discarded for memory to be available.

As you can guess, my goal is to avoid mixing parts as much as possible to keep everything simple (and, as you raised, under control).

BitTorrent file distribution "works" because no client can/should be "purely leeching".

If you're talking about "bittorrent the community", then I totally agree with you. However I'd like to see bittorrent used beyond the "community" use case, because the transport itself is extremely powerful; I'd like bittorrent to be the default transport protocol for distributing Wikipedia dumps, or Internet Archive items, or OSM tiles (or raw data), or my distribution packages. For those cases I expect users to be purely leeching content until completion, then leave the swarm (which would be kind of ok because there will always be an available seed, the upstream project). I'd expect them to be barely seeding content. Basically, I'd like bittorrent to be used when a client can reliably get the torrent (or even just the magnet) to retrieve data wherever it comes from, without having to seed just to improve its choking score.

Anyway, point taken, TT should remain "community" oriented in that it should not be directed towards pure leeching (that's a fair assumption).

[Zilog8]

Here's another proposal: keep an upper boundary of memory used for all ActivePieces, and in RequestBlock, prevent from creating a new ActivePiece if we have already reached the limit...

I like this, it seems like the most correct approach: "Don't []byte off more than we can chew" :-) .