royger / wgo

BitTorrent client in Go

wgo - Simple BitTorrent client in Go

Roger Pau Monné (2010 - 2011)

Introduction

This project is based on the previous work of jackpal, Taipei-Torrent: http://github.com/jackpal/Taipei-Torrent

Since Go is (or should become) a easy to use concurrent system programming language I've decided to use it to develop a simple BitTorrent client. Some of the functions are from the Taipei-Torrent project, and others are from the gobit implementation found here: http://github.com/jessta/gobit

Installation

Simply run:

gomake

Tests

To run all the tests:

gomake test

If you just want to run a single test, enter the corresponding folder and run gomake test.

Usage

wgo is still in a VERY early phase, but you can try it, here are the flags:

./wgo -torrent="path.to.torrent" -folder="/where/to/create/files" -procs=2 -port="6868" -up_limit=20 -down_limit=100

The up_limit and down_limit options are to limit the maximum upload/download, and should be specified in KB/s. If ommited or set to 0, no limit is applied.

The procs option reflects the maximum number of processes the program can use, this is almost only used when checking the hash, and can mean a big improvement in the time needed to check the hash of a torrent. If you have more than one processor, don't hesitate to set this to your number of processors, or your number of processors minus one.

Other options are self explaining I think.

Source code Hierarchy

Since this client aims to make heavy use of concurrency, the approach I've taken is the same as Haskell-Torrent, this is just a copy of the module description made by Jesper Louis Andersen

• Process: Process definitions for the different processes comprising Haskell Torrent

  • ChokeMgr: Manages choking and unchoking of peers, based upon the current speed of the peer and its current state. Global for multiple torrents.
  • Console: Simple console process. Only responds to 'quit' at the moment.
  • Files: Process managing the file system.
  • Listen: Not used at the moment. Step towards listening sockets.
  • Peer: Several process definitions for handling peers. Two for sending, one for receiving and one for controlling the peer and handle the state.
  • PeerMgr: Management of a set of peers for a single torrent.
  • PieceMgr: Keeps track of what pieces have been downloaded and what are missing. Also hands out blocks for downloading to the peers.
  • Status: Keeps track of uploaded/downloaded/left bytes for a single torrent. Could be globalized.
  • Timer: Timer events.
  • Limiter: Limits the maximum upload and download speed of the program.
  • Tracker: Communication with the tracker.

• Protocol: Modules for interacting with the various bittorrent protocols.

  • Wire: The protocol used for communication between peers.

• Top Level:

  • Const: Several fine-tunning options, untill we are able to read them from a configuration file
  • Torrent: Various helpers and types for Torrents.
  • Test: Currently it holds the main part of the program

There's a nice graph that shows the proccess comunications: http://jlouisramblings.blogspot.com/2010/01/thoughts-on-process-hierarchies-in.html http://jlouisramblings.blogspot.com/2009/12/concurrency-bittorrent-clients-and.html http://jlouisramblings.blogspot.com/2009/12/on-peer-processes.html

wgo's Issues

Can never run: "Incorrect Info.Pieces length"

I tried several torrents and I always get:

% ./wgo -torrent=/home/stephane/Downloads/i386cd-5.0.iso.torrent
2010/04/05 18:49:55 [i386cd-5.0.iso]
2010/04/05 18:49:55 File i386cd-5.0.iso
2010/04/05 18:49:55 Total size: 0
2010/04/05 18:49:55 totalLength: 0 pieceLength: 262144
2010/04/05 18:49:55 SHA1 Length: 20 Len ref: 18300 Num pieces: 0
2010/04/05 18:49:55 Incorrect Info.Pieces length

(Here, the official NetBSD torrent, see http://www.netbsd.org/mirrors/torrents/#5.0.2-ports

Cannot compile: "can't find import: wgo/bencode"

I don't know how the Makefile is supposed to work but it does not:

% make
/home/stephane/bin/8g -o go.8 const.go Torrent.go Tracker.go Files.go Wire.go Bitfield.go Peer.go PeerMgr.go PieceMgr.go PeerQueue.go PieceData.go Stats.go logger.go test.go
Torrent.go:14: fatal error: can't find import: wgo/bencode
make: *** [go.8] Error 1

Debian "stable" OS, golang at tip

README.md not up to date

It mentions a "make.all" which does not exist.

gomake failed

Hi,
I got these errors when running 'gomake'
gomake
Makefile:17: warning: overriding commands for target clean' /home/ibk/go/src/Make.common:6: warning: ignoring old commands for targetclean'
rm -rf _.o *.a *.[568vq] [568vq].out wgo _test _testmain.go test.out build.out
for i in Bitfield bencode wgo_io Stats Files Limiter Peers Choke Listener Tracker; do make -C $i clean; done
make[1]: Entering directory /home/ibk/sinaugo/bittorrent/wgo/Bitfield' rm -rf *.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Bitfield'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/bencode' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/bencode'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/wgo_io' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/wgo_io'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Stats' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Stats'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Files' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Files'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Limiter' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Limiter'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Peers' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Peers'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Choke' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Choke'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Listener' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Listener'
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Tracker' rm -rf _.o *.a *.[568vq] [568vq].out *.so _obj _test _testmain.go *.exe _cgo_ _.cgo[12]._ test.out build.out make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Tracker'
make -C Bitfield install
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/Bitfield' 8g -o _go_.8 Bitfield.go rm -f _obj/wgo/bit_field.a gopack grc _obj/wgo/bit_field.a _go_.8 cp _obj/wgo/bit_field.a "/home/ibk/go/pkg/linux_386/wgo/bit_field.a" make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/Bitfield'
make -C bencode install
make[1]: Entering directory/home/ibk/sinaugo/bittorrent/wgo/bencode' 8g -o _go_.8 decode.go parse.go struct.go struct.go:25: undefined: reflect.MapValue struct.go:32: cannot type switch on non-interface value v (type reflect.Value) struct.go:33: undefined: reflect.FloatValue struct.go:40: cannot type switch on non-interface value v (type reflect.Value) struct.go:41: undefined: reflect.FloatValue struct.go:42: cannot use f (type float64) as type reflect.Value in function argument struct.go:47: cannot type switch on non-interface value val (type reflect.Value) struct.go:48: undefined: reflect.IntValue struct.go:49: cannot use int64(i) (type int64) as type reflect.Value in function argument struct.go:50: undefined: reflect.UintValue struct.go:51: cannot use uint64(i) (type uint64) as type reflect.Value in function argument struct.go:52: undefined: reflect.InterfaceValue struct.go:53: undefined: reflect.NewValue struct.go:63: b.map_ undefined (type _structBuilder has no field or method map_) struct.go:64: b.map_ undefined (type *structBuilder has no field or method map_) struct.go:64: b.key undefined (type *structBuilder has no field or method key) struct.go:64: b.val undefined (type *structBuilder has no field or method val) struct.go:72: b.val undefined (type *structBuilder has no field or method val) struct.go:84: b.val undefined (type *structBuilder has no field or method val) struct.go:96: b.val undefined (type *structBuilder has no field or method val) struct.go:109: b.val undefined (type *structBuilder has no field or method val) struct.go:110: undefined: reflect.StringValue struct.go:112: undefined: reflect.InterfaceValue struct.go:113: undefined: reflect.NewValue struct.go:121: b.val undefined (type *structBuilder has no field or method val) struct.go:132: b.val undefined (type *structBuilder has no field or method val) struct.go:133: undefined: reflect.ArrayValue struct.go:135: unknown structBuilder field 'val' in struct literal struct.go:137: undefined: reflect.SliceValue struct.go:154: unknown structBuilder field 'val' in struct literal struct.go:164: b.val undefined (type *structBuilder has no field or method val) struct.go:166: undefined: reflect.MakeZero struct.go:169: b.map_ undefined (type *structBuilder has no field or method map_) struct.go:170: b.val undefined (type *structBuilder has no field or method val) struct.go:172: b.val undefined (type *structBuilder has no field or method val) struct.go:181: b.val undefined (type *structBuilder has no field or method val) struct.go:181: cannot type switch on non-interface value reflect.Indirect(b.val) (type reflect.Value) struct.go:182: undefined: reflect.StructValue struct.go:183: undefined: reflect.StructType struct.go:190: unknown structBuilder field 'val' in struct literal struct.go:193: undefined: reflect.MapValue struct.go:194: undefined: reflect.MapType struct.go:195: undefined: reflect.Typeof struct.go:198: undefined: reflect.NewValue struct.go:199: too many arguments in call to v.Elem struct.go:301: undefined: reflect.ArrayOrSliceValue struct.go:355: undefined: reflect.MapValue struct.go:387: undefined: reflect.StructValue struct.go:199: too many errors make[1]: *_\* [_go_.8] Error 1 make[1]: Leaving directory /home/ibk/sinaugo/bittorrent/wgo/bencode'
make: *** [bencode.make] Error 2

Cannot compile: "PeerMgr.go:162: peer declared and not used"

% ./make.bash
rm -rf _.o *.a *.[568vq] [568vq].out *.cgo1.go *.cgo2.c _cgo_defun.c _cgo_gotypes.go *.so _obj _test _testmain.go
rm -rf *.o *.a *.[568vq] [568vq].out wgo
/home/stephane/bin/8g -o go.8 decode.go parse.go struct.go
rm -f _obj/wgo/bencode.a
/home/stephane/bin/gopack grc _obj/wgo/bencode.a go.8
cp obj/wgo/bencode.a "/local/go/pkg/linux_386/wgo/bencode.a"
/home/stephane/bin/8g -o go.8 const.go Torrent.go Tracker.go Files.go Wire.go Bitfield.go Peer.go PeerMgr.go PieceMgr.go PeerQueue.go PieceData.go Stats.go logger.go test.go
PeerMgr.go:162: peer declared and not used
PeerMgr.go:165: peer declared and not used
PeerMgr.go:180: undefined: ok
PeerMgr.go:180: cannot assign to ok
make: ** [go.8] Error 1

Debian "stable", golang at tip