Pinning rings about ipfs-cluster

[meyerzinn]

I think what we really need are two different projects, ipfs-cluster for managed clusters (i.e. companies trying to make a hosting service like pay-per-pin) and ipfs-opencluster for what #28 and similar describes; it should function as a permutation of BitTorrent and IPNS. It's a DHT that links the hash of an opencluster metadata file (https://gist.github.com/jbenet/deda429fae2e5af9a86a01b0cbb614f7) to peers cooperating to replicate the data. The "cluster" would then operate similar to a normal cluster (like how BitTorrent or IPFS work) except that their efforts are coordinated; the cluster does its best to divide up the responsibility for data among peers (with a preference for more reliable peers).

[meyerzinn]

This would allow teams like ClimateMirror.org to put together a pack with all of the datasets we are concerned about into a pack and have everybody and his grandmother help replicate data.

An interesting feature idea: allow propagation of an "updated" pack, which happens by initially declaring in the pack description an RSA public key for the "manager" of the cluster. The "manager" then signs a new manifest and sends it to a peer on the cluster. That peer sends it to all connected peers before dropping out and joining the new cluster. This would be seamless because the data, still content-addressed, can just shimmy on over.

[hsanjuan]

Yes, it is clear that there are two different main use-cases of what users expect from cluster.

But also, some cluster features/code serve (or would serve) both use-cases. I.e.:

• How content is [re]distributed
• How content is tracked
• How cluster nodes communicate
• APIs
• Composability of cluster nodes

So at this point I'm not ready to say that two projects are needed. For me, ideally, if the cluster core does not fulfill both use-cases (something it aims to), it should come down to a question of tooling over the core (or specific plugins). This is though something that will become clearer as cluster develops.

[NatoBoram]

Instead of administrators, some peers could have write permission and thus be part of the consensus, while the rest just follow and pin accordingly.

[hsanjuan]

All of the points in the original description of the issue are now addressed with --consensus crdt mode. Users can now join clusters, autodiscover other peers and track content as indicated by administrators.