jackpal / taipei-torrent

A(nother) Bittorrent client written in the go programming language

Taipei Torrent

This is a simple command-line-interface BitTorrent client coded in the go programming language.

Features:

• Supports multiple torrent files
• Magnet links
• DHT
• IPv6
• UDP trackers
• UPnP / NAT-PMP automatic firewall configuration
• Socks5 proxy support

Additional Features:

• It can act as a tracker if you start it with the --createTracker flag
• SFTP file system proxying enables torrenting large torrents on systems with limited local file storage.

FAQ:

Q: Why is it named Taipei Torrent?

A: I started writing it while visiting beautiful Taipei, Taiwan

Q: What is the license?

A: See the LICENSE file.

Current Status

• Tested on Go 1.4.2 and tip.
• Tested on Windows, Linux and Mac OS X.
• People tell me they've run it on Android, too.

Development Roadmap

• Full UPnP support (need to be able to search for an unused listener port, detect we have already acquired the port, defend the report against router reboots, release the listener port when we quit.)
• Clean up source code
• Deal with TODOs
• Perhaps a web-based status UI.

Download, Install, and Build Instructions

1. Download and install the Go tools from http://golang.org

2. Use the "go" command to download, install, and build the Taipei-Torrent app:

   go get github.com/jackpal/Taipei-Torrent

Usage Instructions

Taipei-Torrent mydownload.torrent
Taipei-Torrent --useDHT "magnet:?xt=urn:btih:bbb6db69965af769f664b6636e7914f8735141b3"

or

Taipei-Torrent -help

Third-party Packages

https://github.com/jackpal/bencode-go - Bencode encoder/decoder

http://github.com/jackpal/gateway - LAN gateway discovery

http://github.com/jackpal/go-nat-pmp - NAT-PMP firewall client

https://github.com/nictuku/dht - Distributed Hash Table

https://github.com/nictuku/nettools - Network utilities

https://github.com/pkg/sftp - SFTP protocol

Google+ Community

https://plus.google.com/u/0/communities/100997865549971977580

Other Notable Go BitTorrent Implementations

I haven't used these, but they may be worth checking out:

https://github.com/anacrolix/torrent

taipei-torrent's Issues

Feature request: run program on download completion

It's be nice to run a program when a download is complete -- e.g.

TaipeiTorrent -execOnComplete="/usr/local/bin/download_completed" myFiles.torrent

Before the -execOnComplete program is run, some environment variables could be set, like TORRENT (path to the torrent) etc.

How do I get list of peers for a given swarm?

I'm trying to create ui for toy torrent client and so far everything is good, except I cannot locate where peers live.

Torrent file created from magnet link can't be loaded.

So, when you run TT with a magnet link, it generates a .torrent file in the running directory. Unfortunately, this file can't be subsequently loaded in place of the magnet link. This is because the .torrent file created is of the contents of the 'info' dictionary only ( https://wiki.theory.org/BitTorrentSpecification#Metainfo_File_Structure ).

What should be done is to generate a proper .torrent file by:
-Encapsulating the data written now inside an 'info' dictionary.
-Add in the announce and announce-list (if any) that were provided by the magnet-link

(I'm putting up this issue in case anyone else wants to take a crack at it, or failing that, to remind myself of the details when I have free time to program later)

Set up an IRC channel.

@jackpal: It would be nice if you registered an IRC channel (maybe #taipei-torrent?)

have it implementation bep_0009？

http://www.bittorrent.org/beps/bep_0009.html

Incorrect download size reported after client-side restart during the download process

I have added this code to torrent.go method DoTorrent

case <-heartbeatChan:
...
...
            for _, p := range t.peers {
                msg := fmt.Sprintf("%s;%s;%s;%s/%s", t.Ac, t.Name, p.id, strconv.Itoa(int(p.downloadedSize)), strconv.Itoa(int(t.TotalSize)))
                if p.downloadedSize != 0 && p.downloadedSize <= uint64(t.TotalSize) {
                    fmt.Println(msg)
                    PeerStatus <- msg
                }
            }

PeerStatus is a channel and I'm reading this channel and saving it on a database so I can show to the user a real-time download progress.

One of my tests consists in stopping the download process in the client and also removing the ethernet cable and later continuing the download.

On my server this is what the channel shows me: (please be aware of the last part: xxxx/xxxx)

--> Starting download
nw1;terminal1;999-999-999,3.33;18432/1116818
nw1;terminal1;999-999-999,3.33;49152/1116818
nw1;terminal1;999-999-999,3.33;70656/1116818
nw1;terminal1;999-999-999,3.33;98304/1116818
nw1;terminal1;999-999-999,3.33;121856/1116818
nw1;terminal1;999-999-999,3.33;147456/1116818
nw1;terminal1;999-999-999,3.33;173056/1116818
nw1;terminal1;999-999-999,3.33;196608/1116818
nw1;terminal1;999-999-999,3.33;222208/1116818
nw1;terminal1;999-999-999,3.33;245760/1116818
--> Stopped download
--> Starting download
nw1;terminal1;999-999-999,3.33;17408/1116818
nw1;terminal1;999-999-999,3.33;41984/1116818
nw1;terminal1;999-999-999,3.33;65536/1116818
nw1;terminal1;999-999-999,3.33;91136/1116818
nw1;terminal1;999-999-999,3.33;114688/1116818
nw1;terminal1;999-999-999,3.33;140288/1116818
nw1;terminal1;999-999-999,3.33;163840/1116818
nw1;terminal1;999-999-999,3.33;189440/1116818
nw1;terminal1;999-999-999,3.33;212992/1116818
nw1;terminal1;999-999-999,3.33;238592/1116818
--> Stopped download
--> Starting download
nw1;terminal1;999-999-999,3.33;16384/1116818
nw1;terminal1;999-999-999,3.33;38912/1116818
nw1;terminal1;999-999-999,3.33;65536/1116818
nw1;terminal1;999-999-999,3.33;88064/1116818
nw1;terminal1;999-999-999,3.33;114688/1116818
nw1;terminal1;999-999-999,3.33;137216/1116818
nw1;terminal1;999-999-999,3.33;163840/1116818
nw1;terminal1;999-999-999,3.33;186368/1116818
--> Stopped download
--> Starting download
nw1;terminal1;999-999-999,3.33;15360/1116818
nw1;terminal1;999-999-999,3.33;33792/1116818
nw1;terminal1;999-999-999,3.33;64512/1116818
nw1;terminal1;999-999-999,3.33;82944/1116818
nw1;terminal1;999-999-999,3.33;113664/1116818
nw1;terminal1;999-999-999,3.33;132096/1116818
nw1;terminal1;999-999-999,3.33;161792/1116818
nw1;terminal1;999-999-999,3.33;180224/1116818
nw1;terminal1;999-999-999,3.33;209920/1116818
nw1;terminal1;999-999-999,3.33;229376/1116818
nw1;terminal1;999-999-999,3.33;259072/1116818
nw1;terminal1;999-999-999,3.33;278528/1116818
nw1;terminal1;999-999-999,3.33;308224/1116818
nw1;terminal1;999-999-999,3.33;327680/1116818
nw1;terminal1;999-999-999,3.33;355328/1116818
nw1;terminal1;999-999-999,3.33;362496/1116818
nw1;terminal1;999-999-999,3.33;389120/1116818
nw1;terminal1;999-999-999,3.33;409600/1116818
--> Done

Since the p.downloadedSize is reset when I start the download, I should be able to sum the last value of p.downloadedSize and at the end it should be equal to 1116818 but that's not happening. If we sum all the last results from each start/stop, it won't be equal to o.TotalSize:

409600 + 186368 + 238592 + 245760 = 1080320

When I get that information on my web application it is not consistent because it'll show something like this:

Terminal 1: 1080320/1116818

But on the client side the download is complete.

If I don't stop the download process it shows the data correctly:

Terminal 1: 1116818/1116818

Compartmentalization/files API simplification

Hey there,

While working on the files subset of the code (specifically for putting everything files related, so various *files.go, various *cache.go, sftp.go) I realized that there is room to make the code simpler, more understandable. Here's a rough list of things I have in mind:

Modification of active pieces

A torrent.ActivePiece would keep a []byte with the whole piece content. In https://github.com/jackpal/Taipei-Torrent/blob/master/torrent/torrent.go#L1061 we don't write the chunk to the FileStore but to the active piece's buffer; then in https://github.com/jackpal/Taipei-Torrent/blob/master/torrent/torrent.go#L844 we don't read content back from the FileStore but directly from the active piece's buffer.

• Instead of many little writes (16k at a time), we only have 1 "big" write (pieceLength)
• If the piece is not complete, TT (or any torrent client) restarts the piece from scratch, even if all but one chunk have already been downloaded, because there is no way to know if the chunks are correct or not. So we might as well not write any chunk until the piece is complete
• pieceLength shouldn't be huge and TT actively tries to have as few active pieces as possible, so it's not going to take a lot of memory. With the repo's dsl iso a memprofile shows 2MB of memory for the whole download
• Instead of reading back from the FileStore (which could be slow if it's SFTP and you don't have a cache for some reason), the piece is verified directly from memory. Also, checkPiece and computePieceSum aren't needed anymore.
• If the client is purely leeching, there is no need for a cache, even if the FileStore is remote
• We'd need to specify in the FileStore doc that io.WriteAt is only called for full, verified pieces, and that they aren't read back unless asked by another peer

FileStore as the only relevant outside interface

FileStore would get back to what it was before, ie

type FileStore interface {
  // Write is always called for full, verified pieces only;
  // once a piece is written it's not read back until another client needs it
  io.WriterAt

  io.ReaderAt
  io.Closer
}

A TorrentCache would actually be a specific implementation of the FileStore interface that wraps another FileStore; there would be no specific interface just for it. From the point of view of the user, there's no indication that the FileStore is actually a cache. CacheProvider would be changed so that NewCache also has an underlying FileStore as argument. All reads are forwarded (if necessary), all writes are forwarded.

Combined with the previous change in active pieces, there is no need for a specific Commit phase for FileStore, because all writes are "clean" ie they're only full pieces. Any write of a piece to cache should be immediately forwarded to the underlying FileStore. This will also get rid of all the unfulfilled, because the cache wraps the FileStore; the cache would just read the whole piece from FileStore and store it locally, then write only the relevant part for the caller to use.

I am working on this in my file branch (see the storage sub-package) if you want to have a look.

Compartmentalization

Instead of having pretty much everything in the torrent package, I'd like to move the files stuff in its own package (I have a storage package). I also externalized the bitset.

Thoughts ?

bencode can't unmarshal nested dictionaries?

I can't make the bencode code unmarshal strings like:

d1:ad2:id20:abcdefghij0123456789e1:q4:ping1:t2:aa1:y1:qe

I tried this for hours already, thinking I was doing something wrong, but couldn't find a solution. I'm thinking this might be a bug, but I'm not sure. I would appreciate any help.

Unit test to reproduce it:

$ git diff --cached bencode_test.go 
diff --git a/bencode/bencode_test.go b/bencode/bencode_test.go
index cb683af..b48fc79 100644
--- a/bencode/bencode_test.go
+++ b/bencode/bencode_test.go
@@ -3,7 +3,7 @@ package bencode
 import (
        "bytes"
        "fmt"
-//     "jackpal/bencode"
+       //      "jackpal/bencode"
        "os"
        "reflect"
        "testing"
@@ -206,6 +206,15 @@ type SVPair struct {
        v any
 }
 
+var x = map[string]string{"id": "abcdefghij0123456789"}
+var y = map[string]any{
+       "t": "aa",
+       "y": "q",
+       "q": "ping",
+       "a": x,
+}
+
func TestDecode(t *testing.T) {
        tests := []SVPair{
                SVPair{"i0e", int64(0)},
@@ -221,6 +230,7 @@ func TestDecode(t *testing.T) {
                SVPair{"li42e3:abce", []any{42, "abc"}},
                SVPair{"de", map[string]any{}},
                SVPair{"d3:cati1e3:dogi2ee", map[string]any{"cat": 1, "dog": 2}},
+               SVPair{"d1:ad2:id20:abcdefghij0123456789e1:q4:ping1:t2:aa1:y1:qe", y},
        }
        for _, sv := range tests {
                if err := check(sv.s, sv.v); err != nil {
@@ -250,6 +260,7 @@ func TestUnmarshal(t *testing.T) {
                SVPair{"de", map[string]any{}},
                SVPair{"d3:cati1e3:dogi2ee", map[string]any{"cat": 1, "dog": 2}},
                SVPair{"d1:ai10e1:b3:fooe", structA{10, "foo"}},
+               SVPair{"d1:ad2:id20:abcdefghij0123456789e1:q4:ping1:t2:aa1:y1:qe", y},
        }
        for _, sv := range tests {
                if err := checkUnmarshal(sv.s, sv.v); err != nil {

Feature Request: Support WebTorrent

Consider adding support for WebTorrent.

WebTorrent is a BitTorrent variation designed to be implementable by web browsers. It works over WebRTC.

https://webtorrent.io

Not sure webtorrent will become popular, but supporting it would be an interesting exercise. Presumably we could use interfaces to cleanly encapsulate the WebRTC code from the normal networking code.

make UPnP a separate reusable go module

A suggestion from [email protected]

keepalived msg, empty msg, also read msg len: 1, why do this?

Taipei-Torrent/torrent/peer.go

Line 288 in 3696397

buf = make([]byte, 1)

DHT won't work without torrent?

I've noticed TT only connects to peers found using the DHT if we already have the torrent file. But what if we don't have the torrent file and only the infohash?

eg of the restriction: https://github.com/jackpal/Taipei-Torrent/blob/master/torrent/torrent.go#L520

Even if I remove all such restrictions like this (I think there are 2 more of them), I still can't download using just an infohash, for example "magnet:?xt=urn:btih:deca7a89a1dbdc4b213de1c0d5351e92582f31fb", even though I can connect to peers found over the dht for it (and there are several).

What's the deal?

add http stream Features pls

add http stream Features pls

Integration with blockchain (tendermint)

I think integration of Taipei-Torrent with Tendermint blockchain technology(https://github.com/tendermint/) would be great.
Both technology is written in go-lang. I'm about to launch Tendermint based blockchain(luna-blockchain) .if it meets with Taipei-Torrent, then It can solve the economic incentive problem of seeder. ( Through the way that seeder submit proof of seeding to blockchain). Although I need to read and understand Taipei-Torrnet code further, but since Tendermint consensus engine can generate decentralized immutable consensus about almost anything, It can also generate decentralized consensus about which torrent file is honest one. So it can also solve malicious/fake torrent problem too I think. Any idea?

Feature request: RPC support

It'd be really nice to be able to control the program remotely via RPC of some sort (JSON-RPC over WebSocket, for instance). Methods and features that would be handy are:

• add a .torrent file, either directly (as an encoded argument) or via a URL
• remove a torrent from the downloads/seeds
• get notified when download is complete and seeding
• check transfer status

Usually have "send a bad piece" error

When I use Taipei-Torrent to download file, I have 1 seeder and 10 leechers. I usually meet the problem "Closing peer that sent a bad piece 9 -tt15217_12515306877 reference sha1: [217 254 129 37 191 21 13 134 0 108 169 60 11 41 131 173 232 147 204 164] != piece sha1: [179 220 206 190 193 88 161 163 18 120 145 202 124 252 13 151 217 222 148 52]". What is the reason?

Torrent session not properly torn-down on loading error.

When starting a torrent errors out due to a malformed resume file, the created torrent session isn't torn-down properly. This mostly manifests itself when using the SFTP backend, as the SFTP connection is left open instead of being properly closed

Panic in Bitset.IsSet

This happened 1 out of 3 tries when trying to download the Ubuntu 13.10 iso.

From inspecting the source it looks like somehow the Bitset.b field is nil.

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x18 pc=0x23d9]

goroutine 1 [running]:
runtime.panic(0x2ab860, 0x603db9)
/Users/jack/code/go/src/pkg/runtime/panic.c:266 +0xb6
main.(_Bitset).IsSet(0x0, 0x559, 0xfe000)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/bitset.go:51 +0x29
main.(_TorrentSession).checkRange(0xc210063120, 0xc21000fdc0, 0x559, 0x6e6, 0xc21044b150)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:668 +0x6f
main.(_TorrentSession).ChoosePiece(0xc210063120, 0xc21000fdc0, 0x7573c0)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:659 +0x61
main.(_TorrentSession).RequestBlock(0xc210063120, 0xc21000fdc0, 0x0, 0x0)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:630 +0x192
main.(_TorrentSession).generalMessage(0xc210063120, 0xc21010bfa0, 0x1, 0x1, 0xc21000fdc0, ...)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:877 +0x25c
main.(_TorrentSession).DoMessage(0xc210063120, 0xc21000fdc0, 0xc21010bfa0, 0x1, 0x1, ...)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:842 +0x321
main.(*TorrentSession).DoTorrent(0xc210063120, 0x0, 0x0)
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent.go:559 +0xf9a
main.main()
/Users/jack/code/gocode/src/github.com/jackpal/Taipei-Torrent/main.go:58 +0x4d7

unsupported protocol scheme "udp"

Error: Could not fetch tracker info: Get udp://open.demonii.com:1337?compact=1&downloaded=0&info_hash=bR%F8%CEYl%F1%7B%3F%2Fi%E5r%166%17%E2%E2%85y&left=0&peer_id=-tt3244_557700679194&port=7777&uploaded=0: unsupported protocol scheme "udp"

TLS-Supprt

Is the torrent connection via TLS or something like that encrypted, or is this one of the TODos?

Killed process can leave corrupted resume file.

If Taipei-Torrent process is killed (say, because the system is out of memory) at a precisely unfortunate time, the *-haveBitset file for a torrent might end up being empty (0 bytes long). We should change the way the resume file is updated to prevent that.

Embedding Taipei-Torrent

A couple functions like ListenForPeerConnections and (*TorrentSession).AcceptNewPeer take a btConn struct, but this struct is not exported, while these functions are public. Change btConn to BtConn, even with all of its fields filtered would be fine?

A data race reading/writing t.si.HaveTorrent

A data race reading/writing t.si.HaveTorrent

Noticed while testing the debian magnet link in issue #63

WARNING: DATA RACE
Write by goroutine 10:
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).load()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:283 +0x147b
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).reload()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:201 +0x356
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).DoMetadata()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:1206 +0xe08
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).DoExtension()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:1129 +0xa1b
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).extensionMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:910 +0xf5
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).DoMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:903 +0x138
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:602 +0x1f66
github.com/jackpal/Taipei-Torrent/torrent.func·004()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrentLoop.go:56 +0x3d

Previous read by goroutine 44:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).connectToPeer()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:340 +0x65

Goroutine 10 (running) created at:
github.com/jackpal/Taipei-Torrent/torrent.RunTorrents()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrentLoop.go:58 +0x69d
main.main()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/main.go:114 +0x9ca

Goroutine 44 (finished) created at:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:563 +0x1363
github.com/jackpal/Taipei-Torrent/torrent.func·004()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrentLoop.go:56 +0x3d

Is PEX anywhere on the roadmap?

Nice project, I'm just curious will PEX be implemented?

Leachers seem to prefer seeders, is that correct behavior?

Someone asked by email:

When I use one node as seeder and another 10 nodes for leecher. I found that all these leecheres mainly get data from seeder and do not exchange data with other leechers. Why?

bencode tests don't pass

Some problem with TestUnmarshal.

need help!!!

hi today i saw this repo and the first question that came to my mind was that do it support heroku or not pleasereply ASAP!!!
Thanks in advance

Transmission-gtk connection error

I have this error with transmission :

2015/02/12 02:06:15 Error reading header:  Magic string is not correct: #WW-XX#@77
2015/02/12 02:06:16 Peers: 0 downloaded: 116555776 uploaded: 83918552 ratio 0.7199862150117725
2015/02/12 02:06:16 good, total 860 958
2015/02/12 02:06:16 Error reading header:  First byte is not 19
2015/02/12 02:06:17 Peers: 0 downloaded: 116555776 uploaded: 83918552 ratio 0.7199862150117725
2015/02/12 02:06:17 good, total 860 958
2015/02/12 02:06:17 New bt connection for ih 597c5b4f8c7f255e1e7112849ef98ef10c634b11

Maybe, transmission implement a feature not available in Taipei-Torrent ?

Could uTP be a solution ?
http://www.bittorrent.org/beps/bep_0029.html

Note : ktorrent work well..

Here, is my setup : rakshasa/libtorrent#77
(Yes, I have same bug with rtorrent.. )

Skip files

Hi,

Is there a simple way of skipping certain files? I can't seem to find it anywhere. Is there a way to parse a file into the pieces it has so that I could mark them as complete when the torrent is beng loaded for the first time.

This project does not implement "rarest first" algorithm

I have read all the codes and drawn a mind map. I found that when choosing piece to request, TT has implemented the "Strict Priority", "Random first piece" and "Endgame mode", but it does not implement "Rarest first" algorithm. Am I right?

sorry for a problem

I have made 4 pull requests, but I find an error with the first pulling with title"fix error in downloading pieces continue ". And I do not know have to undo the pulling. So I make another pull again.

Too many requests are made for metadata

Currently, Taipei-Torrent checks if a peer has advertised metadata support here. If the peer has, it kicks off the metadata-request processing by asking for the first piece of the metadata.

Whenever Taipei-Torrent receives a metadata response, DoMetadata. The problem is that each time DoMetadata is called with a metadata response, it attempts to fetch all missing metadata pieces.

Consider this scenario:

1. The client connects to a peer advertising metadata support, and requests the first piece of the metadata (piece 0).
2. The client receives piece 0 from the peer, and requests all of the missing pieces (in this case, 10).
3. The peer sends back metadata piece 1. However, since none of the other metadata pieces have been received, the client will send another request for every piece besides 0 and 1, even though requests for those pieces are already in-flight.
4. Step 3 continues for each received piece, until the client has all of the pieces (after issuing many duplicate requests).

I believe a good solution is to poll the peer for a piece if a response for it is not received, similar to how a tracker is polled for a response.

Race in connectToPeer

WARNING: DATA RACE
Read by goroutine 17:
syscall.raceReadRange()
/usr/local/go/src/pkg/syscall/race.go:25 +0x3f
syscall.Write()
/usr/local/go/src/pkg/syscall/syscall_unix.go:153 +0xb9
net.(_netFD).Write()
/usr/local/go/src/pkg/net/fd_unix.go:289 +0x335
net.(_conn).Write()
/usr/local/go/src/pkg/net/net.go:130 +0xfe
net.(_TCPConn).Write()
/usr/local/go/src/pkg/net/dial.go:1 +0x62
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).connectToPeer()
/Users/yves/go/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:315 +0xed

Previous write by goroutine 20:
runtime.copy()
/usr/local/go/src/pkg/runtime/slice.c:120 +0x0
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).Header()
/Users/yves/go/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:294 +0x437
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).connectToPeer()
/Users/yves/go/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:315 +0xab

Goroutine 17 (running) created at:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()
/Users/yves/go/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:497 +0xe42

Goroutine 20 (running) created at:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()

/Users/yves/go/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:497 +0xe42

This is pretty easy to reproduce, at least from my current slow network, using test.bash.

Swarms of size 20,50,100 don't complete

Repro: Uncomment the commented-out-tests in tracker_test.go

Might be some sort of OS-level resource starvation or race condition. Tested on OSX. Fails reproducibly. (Which tends to indicate that it's not a race condition.)

Race condition in `AddPeer` and `ClosePeer`

In AddPeer, the map t.peers is looped over to compare the ID of the peer being added with the current peer IDs. However, in ClosePeer, the peer being closed is deleted from t.peers. Since AddPeer and ClosePeer can be run by separate go routines, this can create a race condition if ClosePeer deletes the peer that AddPeer is inspecting in the loop.

Update Taipei Torrent for the latest Go release

There were several language and package changes since the last commit.

This issue is just an FYI that I'm updating everything now.

Binary releases

Hi there,

This is a very interesting project and i've been running it on both mac and
raspberry pi platforms. Everything works great. However, it lacks a section
to download precompiled binaries. Would be great to have a place (like Github releases)
to just go and download a binary.

I can work on that if you're interested.

install error

src/github.com/jackpal/bencode-go/parse.go:205: undefined: sync.Pool

"Rejecting peer because already have a peer with the same id" after node restart

A bug report by email:

When one node has downloaded a file and removes the file.
After this, when this node reboots Taipei-Torrent and downloads same file again.
It prints "Rejecting peer because already have a peer with the same id" and will create a empty file that same size with original file.

IPv6 DHT issues.

Greetings,

I added IPv6 to Nictuku's DHT implementation. It's a reasonable implementation, worked well in my testing, and involved minimal lines of code because Go hides most of the IPv4/IPV6 differences.

Because of BEP 32's recommendation, I kept the IPv4 and IPv6 routing tables completely seperate. This was most easily done by just adding v4/v6 flag to the config and generating a DHT for each. As a result you won't get IPv4 and IPv6 peers without some changes to clients using Nictuku's DHT.

There's various ways to fix this:

1. tweak Taipei-Torrent to launch two go threads one for IPv6 one for IPv4, just like dht/example/main6.go
2. rework DHT to maintain 2 internal routing tables and add logic for tracking which message updates which DHT
3. ignore the BEP and have one routing table for all peers (v4 and v6). I can't think of any downside. Seems like contacting the N nearest peers to your target is best, regardless of which are IPv4 and which are IPv6. After all if a size X routing table is optimal for Y peers why care about the mix of IPv4 vs IPv6 over time?

I'm willing to help, figured as a user of the DHT you might have an opinion on the above.

is that possible to provide io.ReadSeekCloser interface to bittorent file?

Hello. I'm try to find bittorent library for go, but i need strange things - i don't need to store file to fs, but i need to have io.ReadSeekCloser interface to it to able to read file (for example tar archive or squashfs image) and extract it contents into memory.
Does it possible ?

Crash in (*BitSet).IsSet torrent/bitset.go:52

2014/05/12 21:22:55 Closing connection to <..>
panic: Index out of range.

goroutine 27 [running]:
runtime.panic(0x69f940, 0xc2082a5500)
        /usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/jackpal/Taipei-Torrent/torrent.(*Bitset).IsSet(0xc2082420c0, 0x235, 0xc208357448)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/bitset.go:52 +0xda
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).RecordBlock(0xc20802e000, 0xc2082c6340, 0x1800000000235, 0x4000, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:812 +0x721
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).generalMessage(0xc20802e000, 0xc20848a000, 0x4009, 0x4009, 0xc2082c6340, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:1005 +0x1840
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoMessage(0xc20802e000, 0xc2082c6340, 0xc20848a000, 0x4009, 0x4009, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:878 +0xb6
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent(0xc20802e000)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:588 +0x1685
main.func·002()
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/main.go:96 +0x37
created by main.main
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/main.go:98 +0xbe4

goroutine 16 [select]:
main.main()
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/main.go:108 +0x1550

goroutine 19 [finalizer wait]:
runtime.park(0x4158d0, 0x96d280, 0x96b4e9)
        /usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x96d280, 0x96b4e9)
        /usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
        /usr/local/go/src/pkg/runtime/mgc0.c:2626 +0xcf
runtime.goexit()
        /usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 20 [syscall]:
os/signal.loop()
        /usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
        /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x32

goroutine 21 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x96eea0)
        /home/nictuku/src/github.com/golang/glog/glog.go:839 +0x75
created by github.com/golang/glog.init·1
        /home/nictuku/src/github.com/golang/glog/glog.go:406 +0x28a

goroutine 17 [syscall]:
runtime.goexit()
        /usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 23 [IO wait]:
net.runtime_pollWait(0x7f818a43eac0, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080521b0, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080521b0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).accept(0xc208052150, 0x83c680, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:409 +0x2fe
net.(*TCPListener).AcceptTCP(0xc20803e020, 0x6928a0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/tcpsock_posix.go:233 +0x59
net.(*TCPListener).Accept(0xc20803e020, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/tcpsock_posix.go:243 +0x4b
github.com/jackpal/Taipei-Torrent/torrent.func·001()
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/listen.go:52 +0x4c
created by github.com/jackpal/Taipei-Torrent/torrent.ListenForPeerConnections
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/listen.go:71 +0x43a

goroutine 24 [select]:
github.com/nictuku/nettools.(*ClientThrottle).cleanup(0xc208044da0)
        /home/nictuku/src/github.com/nictuku/nettools/ratelimit.go:76 +0x220
created by github.com/nictuku/nettools.NewThrottler
        /home/nictuku/src/github.com/nictuku/nettools/ratelimit.go:20 +0xcb

goroutine 28 [IO wait]:
net.runtime_pollWait(0x7f818a43ea10, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2080525a0, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2080525a0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).readFrom(0xc208052540, 0xc208017000, 0x1000, 0x1000, 0x0, 0x0, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:259 +0x39d
net.(*UDPConn).ReadFromUDP(0xc20803e058, 0xc208017000, 0x1000, 0x1000, 0x1e, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/udpsock_posix.go:67 +0x141
github.com/nictuku/dht.readFromSocket(0xc20803e058, 0xc208004780, 0xc208054140, 0xc208004660)
        /home/nictuku/src/github.com/nictuku/dht/krpc.go:215 +0xbb
created by github.com/nictuku/dht.(*DHT).Run
        /home/nictuku/src/github.com/nictuku/dht/dht.go:300 +0x182

goroutine 26 [runnable]:
github.com/nictuku/dht.(*DHT).Run(0xc208066100, 0x0, 0x0)
        /home/nictuku/src/github.com/nictuku/dht/dht.go:331 +0x1086
created by github.com/jackpal/Taipei-Torrent/torrent.NewTorrentSession
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:190 +0x4ee

goroutine 29 [select]:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).deadlockDetector(0xc20802e000)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:446 +0x48b
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:475 +0x5f

goroutine 30 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.func·004()
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/trackerClient.go:47 +0x6c
created by github.com/jackpal/Taipei-Torrent/torrent.startTrackerClient
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/trackerClient.go:59 +0x247

goroutine 31 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.func·005()
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/trackerClient.go:62 +0x7b
created by github.com/jackpal/Taipei-Torrent/torrent.startTrackerClient
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/trackerClient.go:68 +0x286

goroutine 40 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.queueingWriter(0xc2082cb260, 0xc2082cb2c0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:65 +0xbc
created by github.com/jackpal/Taipei-Torrent/torrent.NewPeerState
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:94 +0x74

goroutine 52 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.queueingWriter(0xc2082dbd40, 0xc2082dbda0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:65 +0xbc
created by github.com/jackpal/Taipei-Torrent/torrent.NewPeerState
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:94 +0x74

goroutine 42 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerWriter(0xc2082c6820, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:240 +0xa1
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:418 +0x41e

goroutine 43 [runnable]:
net.runtime_pollWait(0x7f818a43e8b0, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208132290, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208132290, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208132230, 0xc20818bc7c, 0x4, 0x4, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc20803e040, 0xc20818bc7c, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/jackpal/Taipei-Torrent/torrent.readNBOUint32(0x7f818a43f048, 0xc20803e040, 0x7f8100000000, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:225 +0xcd
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerReader(0xc2082c6820, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:276 +0x47
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:419 +0x447

goroutine 64 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.queueingWriter(0xc208301f20, 0xc208301f80)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:65 +0xbc
created by github.com/jackpal/Taipei-Torrent/torrent.NewPeerState
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:94 +0x74

goroutine 54 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerWriter(0xc2082c7c70, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:240 +0xa1
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:418 +0x41e

goroutine 55 [runnable]:
net.runtime_pollWait(0x7f818a43e750, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2081333a0, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2081333a0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208133340, 0xc2081ad120, 0x4, 0x4, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc20803e060, 0xc2081ad120, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/jackpal/Taipei-Torrent/torrent.readNBOUint32(0x7f818a43f048, 0xc20803e060, 0x7f8100000000, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:225 +0xcd
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerReader(0xc2082c7c70, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:276 +0x47
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:419 +0x447

goroutine 76 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.queueingWriter(0xc208300600, 0xc208300660)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:65 +0xbc
created by github.com/jackpal/Taipei-Torrent/torrent.NewPeerState
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:94 +0x74

goroutine 66 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerWriter(0xc208304d00, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:240 +0xa1
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:418 +0x41e

goroutine 67 [runnable]:
net.runtime_pollWait(0x7f818a43e490, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc208052d80, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc208052d80, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc208052d20, 0xc2082ba75c, 0x4, 0x4, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc20803e0a0, 0xc2082ba75c, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/jackpal/Taipei-Torrent/torrent.readNBOUint32(0x7f818a43f048, 0xc20803e0a0, 0x7f8100000000, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:225 +0xcd
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerReader(0xc208304d00, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:276 +0x47
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:419 +0x447

goroutine 78 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerWriter(0xc2082c6340, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:240 +0xa1
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:418 +0x41e

goroutine 91 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.queueingWriter(0xc2082da5a0, 0xc2082da600)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:65 +0xbc
created by github.com/jackpal/Taipei-Torrent/torrent.NewPeerState
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:94 +0x74

goroutine 79 [runnable]:
net.runtime_pollWait(0x7f818a43e5f0, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2081bc530, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2081bc530, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc2081bc4d0, 0xc2084ae22d, 0x3ddc, 0x3ddc, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc20803e0e0, 0xc2084ae22d, 0x3ddc, 0x3ddc, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/net.go:122 +0xe7
io.ReadAtLeast(0x7f818a448798, 0xc20803e0e0, 0xc2084ae000, 0x4009, 0x4009, 0x4009, 0x22d, 0x0, 0x0)
        /usr/local/go/src/pkg/io/io.go:289 +0x11e
io.ReadFull(0x7f818a448798, 0xc20803e0e0, 0xc2084ae000, 0x4009, 0x4009, 0x4009, 0x0, 0x0)
        /usr/local/go/src/pkg/io/io.go:307 +0x89
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerReader(0xc2082c6340, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:293 +0x16f
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:419 +0x447

goroutine 93 [chan receive]:
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerWriter(0xc2082c71e0, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:240 +0xa1
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:418 +0x41e

goroutine 94 [runnable]:
net.runtime_pollWait(0x7f818a43e280, 0x72, 0x0)
        /usr/local/go/src/pkg/runtime/netpoll.goc:146 +0x66
net.(*pollDesc).Wait(0xc2081bd6b0, 0x72, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:84 +0x46
net.(*pollDesc).WaitRead(0xc2081bd6b0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/fd_poll_runtime.go:89 +0x42
net.(*netFD).Read(0xc2081bd650, 0xc2082a53f8, 0x4, 0x4, 0x0, 0x7f818a43d440, 0xb)
        /usr/local/go/src/pkg/net/fd_unix.go:232 +0x30e
net.(*conn).Read(0xc20803e128, 0xc2082a53f8, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/pkg/net/net.go:122 +0xe7
github.com/jackpal/Taipei-Torrent/torrent.readNBOUint32(0x7f818a43f048, 0xc20803e128, 0x7f8100000000, 0x0, 0x0)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:225 +0xcd
github.com/jackpal/Taipei-Torrent/torrent.(*peerState).peerReader(0xc2082c71e0, 0xc208004420)
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:276 +0x47
created by github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).addPeerImp
        /home/nictuku/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:419 +0x447

I can work on this but it may take me some time.

Metadata downloader conflicts with the deadlock detector

See the discussion in #63.
The metadata downloader triggers a load() and a checkPieces() call that gets stuck for a while. This makes the deadlock detector unhappy.

I don't know how to fix this without doing some surgery on the detector.

Should we maybe make the deadlock detector print a stackdump without throwing a panic?

torrent server in go ?

Wondering if this code can be leveraged for a torrent server ?

Or maybe someone knows of a go torrent server ?

Data race: peer.go:180 p.lastWriteTime = time.Now()

Saw this when running test.bash. Reproduces 3/3

WARNING: DATA RACE
Write by goroutine 10:
github.com/jackpal/Taipei-Torrent/torrent.(_peerState).sendMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:180 +0xb1
github.com/jackpal/Taipei-Torrent/torrent.(_peerState).SendBitfield()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:146 +0x1a6
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).generalMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:965 +0x320a
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).DoMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:791 +0xf3
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:533 +0x176c

Previous write by goroutine 52:
github.com/jackpal/Taipei-Torrent/torrent.(_peerState).sendMessage()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:180 +0xb1
github.com/jackpal/Taipei-Torrent/torrent.(_peerState).SendExtensions()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/peer.go:170 +0x497
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).AddPeer()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:381 +0x6ca
github.com/jackpal/Taipei-Torrent/torrent.(_TorrentSession).connectToPeer()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:333 +0x468

Goroutine 10 (running) created at:
main.main()
/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/main.go:73 +0x9c5

Goroutine 52 (finished) created at:
github.com/jackpal/Taipei-Torrent/torrent.(*TorrentSession).DoTorrent()

/Users/jackpal/code/gocode/src/github.com/jackpal/Taipei-Torrent/torrent/torrent.go:494 +0xe42

Could not "go get"

go get github.com/jackpal/Taipei-Torrent

./torrent.go:306: undefined: torrentHeader
./torrent.go:307: undefined: torrentHeader
./torrent.go:321: undefined: torrentHeader
./torrent.go:321: cannot assign to torrentHeader

I looked at torrent.go and there are no other places using torrentHeader.

Please advise....

Thanks.

Reconsider peer connection strategy

Currently, we accept all incoming connections and try to connect to all new peers we learn of. According to this blog post from libtorrent main dev, the standard behavior is to cap connectivity to 50 peers.

The outcome of this strategy is that global connectivity is low, and it is long until a new entrant can find anyone willing to share in the swarm.

The proposed alternative is to have a commutative function that depends on both peers' address and outputs a priority, and select peers with highest priority first (even going as far as disconnecting peers we are currently connected to if another one turns out to be better).

Here's the function (from the simulator code):

def prio(n1, n2):
        if n1 > n2:
                t = n2
                n2 = n1
                n1 = t

        if (n1, n2) in prio_cache: return prio_cache[(n1, n2)]

        h = hashlib.sha1()
        h.update('%d%d' % (n1, n2))
        p = h.hexdigest()
        prio_cache[(n1, n2)] = p
        return p

Fairly simple, could be interesting to consider.

Request: Status update in-place

I've seen plenty of Go packages update status in-place, rather nicely. There's a new package that even gives you a fancy terminal dashboard from cli (https://plus.google.com/+WebMediaHouseNet/posts/3SKBzkGi3Wh), so I'd think getting at least a one or two-liner in-place output should be doable.

decode.go:101: v.Data undefined (type vector.Vector has no field Data)

Can't compile bencode package, what version of Go should I use?

String 2 Bytes

Ref: https://github.com/jackpal/Taipei-Torrent/blob/master/torrent/torrent.go#L66
Code:

func string2Bytes(s string) []byte { return bytes.NewBufferString(s).Bytes() }

Why not just

[]byte(YourString)