Comments (3)
Hey,
that has been done before, although using v1. Still, shouldn't be too difficult with v2. I'll give you a few pointers, let me know what you think and if there's any questions:
- In general, you'd package your software/update/whatever in some way, then create a torrent from that (the torrent is metadata => small), distribute the torrent among your... machines or whatever, via some other channel. The machines all run a torrent client and start exchanging the file. You need a few (at least one) machine with a whole copy of the file, usually the one creating the torrent.
- You probably want to configure the torrent clients to not use the DHT, and instead use just the one tracker listed in the torrent, which would be an instance of Chihaya reachable by all the machines.
- The clients interact with Chihaya to get a list of peers, i.e., addresses of other machines they can get the data from/to.
On the Chihaya-side, you'd have to consider things like:
- Torrent approval: Depending on where or how you run this, you probably want to restrict which torrents are tracked. We have the torrent approval middleware for that, which takes a list of infohashes to whitelist. This is configured directly in the Chihaya config file, but worry not: We support reloading the config via signals (SIGUSR1, I believe), which will rebuild all the middleware according to the new config.
- Client approval: Probably easiest to do this via firewall or reverse proxy. On the application layer, you could whitelist individual peer IDs, if you know those (and if they're static), but that's probably not the best solution.
- As an optimization, in v1 we had a modified storage implementation that would return peers with low network distance, i.e., within the same subnet or whatnot. Basically it returned peers ordered by common IP prefix.
That's of course pretty nice w.r.t. performance, especially if you can keep it in-rack or whatnot, depending on your architecture.
We don't have that storage for v2, but you could implement it, if you really wanted. Or maybe we can figure something out.
If you do end up doing this, we'd appreciate any feedback on the process :) Any problems you ran into, etc. pp.
from chihaya.
- Chihaya
Hey, thank you @mrd0ll4r!
I was wondering if there are specific libraries for integration with other deployment tools.
For example, I guess one option could be to distribute your docker images with BitTorrent clients and use docker save and docker load. You would need a thin BitTorrent client to "pull" the images. That way, you would not need to use a docker registry, and your servers would also act like a distributed docker registry.
But my question is more about already-built solutions than custom deployment processes. For example, maybe if you are using K8S you can setup a local docker registry and load images into the local registry with a "thin" BitTorrent client β just and idea.
Ideally, there should be an easy way to use the BitTorrent protocol in your standard deployment solution.
But I'm not a sysadmin, I'm just curious about use cases for BitTorrent. I'm also contributing to a BitTorrent tracker, and I want to know more about real use cases.
By the way I found this client https://github.com/mandreyel/cratetorrent that I suppose it could be use to write a service that you can run on your servers to download your artifacts.
I'm going to close the issue since this is more a discussion that an issue.
from chihaya.
Hmm, I'm not sure if there are ready-made solutions... I know twitter and facebook used to do some bittorrent stuff for their deployments, but not sure if that's still the case. I wonder if @jzelinskie knows more, since he's much close to what's actually happening in the industry :)
from chihaya.
Related Issues (20)
- Explore serving UDP without standard lib HOT 3
- Bencoded dictionaries' keys are not always sorted HOT 5
- Explore using freecache for memory storage HOT 1
- Chihaya has reached a GoLang Limitation: concurrent operations on a single file or socket HOT 8
- Load config from dotfiles & create default config HOT 4
- Migrate chat off freenode HOT 2
- Explore replacing net.IP with inet.af/netaddr.IP HOT 2
- Explore using shoco for peer compression HOT 2
- Generational Garbage Collection
- State of Chihaya? No release since 2016? Will a new release be published? HOT 3
- Adopt goreleaser
- Change storage interface to take a slice of peers as input
- Does chihaya not support redis cluster modeοΌ HOT 2
- How to read metrics number? HOT 2
- Explore garbage collection using golang.org/x/exp/maps
- How to connect to redis with unix domain socket? HOT 2
- Ideas regarding limiting concurrency and reducing allocations
- Data Race in UDP Frontend
- Investigate zeropool
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chihaya.