Pins should support a replication factor about ipfs-cluster

[meyerzinn]

I'd like to be clear. Is this pinning blocks or pinning merkel dags that describe files? I.e., say I use ipfs add shakespeare.txt and get a hash there representing the file--if I add that to the cluster, will the cluster distribute the file in the way you describe or distribute the individual blocks?

Also, can this support clusters where nodes have limits that vary?

[hsanjuan]

@20zinnm right now its not pinning blocsk individually. It's equivalent to ipfs pin add. It will distribute the file.

Also, can this support clusters where nodes have limits that vary?

yes, it should not be difficult to implement custom strategies for different limits.

---

// fixed formatting ~ Kubuxu

[meyerzinn]

@hsanjuan I've been doing some reading concerning the Raft consensus algorithm you use in ipfs-cluster, and I wonder why a model like this wouldn't work:

1. Leader receives request to pin IPFS file.
2. Leader commits a message like 'ADD [hash]' or something.
3. Followers receive the request and break it down to individual blocks.
4. Followers then decide whether or not they have the capacity to keep specific blocks.
5. Should a follower decide to keep a block, it proposes a change to the log with the block hash (and if in an untrusted environment a proof of retrievability ¹).
6. The leader will commit it (or, if in an untrusted environment, will commit it after verifying the proof).
7. Followers prioritize blocks to keep based upon how many others are included in the state as keeping the block.

This would allow for fine-grained distribution and replication at the block level to maximize the utilization of peers and allow them the ability to chose whether or not to take blocks. This is advantageous because it allows for heterogeneous peers (my raspberry pi with a gigabyte and a rack in the cloud with a petabyte can participate on the same cluster).

[hsanjuan]

Hello @20zinnm ,

As it was pointed out on irc, while your proposal offloads the tracking decision to the nodes, it does not allow to provide balancing strategies where comparison between nodes are needed (i.e. geographically or network-topology based decisions).

Regarding block-level tracking, as I said, on this first approach we will not do it. A number of reasons were also provided on irc. I think you can accommodate your use case by sending the block hashes to cluster rather than the main hash, as cluster does not make distinctions. Would that work for you?

[hsanjuan]

PR is in.The re-pinning on host failure has been moved to #45.

[0zAND1z]

What is the default replication factor in an ipfs-cluster?

Would love to know that.

[lanzafame]

It's defined in the config file, found by default at `~/.ipfs-cluster/service.json`

…

On Wed., 13 Feb. 2019, 16:33 Ganesh Prasad Kumble, ***@***.***> wrote: What is the default replication factor in an ipfs-cluster? Would love to know that. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFpnaOGxAQWHWbawQpeHGeKEWbgJCWbiks5vM7HCgaJpZM4L5yPX> .

[0zAND1z]

Thanks.

What's the range for these replication factor variables here?

I see the https://cluster.ipfs.io/documentation/configuration/#the-cluster-main-section describing the default values for replication factor, but not the range of values.

[lanzafame]

Min is 0, max is whatever fits inside a golang int, and -1 has the special meaning of all peers.

…

On Wed., 13 Feb. 2019, 22:05 Ganesh Prasad Kumble, ***@***.***> wrote: Thanks. What's the range for these replication factor variables here? I see the https://cluster.ipfs.io/documentation/configuration/#the-cluster-main-section describing the default values for replication factor, but not the range of values. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFpnaCywRK-Fr8samhm_hwTFAzv-JgGGks5vM_-egaJpZM4L5yPX> .