Our current shard transfer algorithm goes like this:
- A node with an ID close to the shard Id is found
- That node is pinged to make sure it's still alive
- If it's alive, initiate shard payment and transfer
- If it's not alive, remove node from contacts and re-search
The "shard payment and transfer" subroutine goes like this:
- Given a target node's address, ask for its Stellar account id
- Send a payment to that Stellar account id
- If the payment is successful, send the shard to the target node for storage
The problem with this is that the data owner cannot trust the target node to host their file. Nor can the data owner trust the inherent volatility of network connections. It is possible that a host node, upon receiving a payment, disconnects from the network. Finally, it is possible that the host node simply deletes the file right after receiving it, keeping the payment but freeing up storage.
We therefore need to "batch" file storage and payment for file storage such that the failure of one entails the failure of the other. To further prevent deletion of file storage immediately after receiving the file, the two nodes must agree on a duration for which the host will store the data owner's shard.
To ensure that this agreement is honored, the host node must be able to prove that it still has the file at the end of the agreed-upon duration. It must therefore pass a data availability and integrity audit immediately prior to receiving payment.
To ensure that the host node is actually paid by the data owner, an escrow account is set up with the funds to pay the host node at the end of the agreed-upon duration.
Edge cases:
- What if the host node is generally online and available, but happens to be offline at the time of the final audit that verifies that it is storing the data it agreed to store?
- What if the host node is offline for the entire duration, but then gets online immediately preceding the audit in order to get paid? They haven't really satisfied their end of the bargain in this case.
We can redefine the agreement in order to account for these edge cases. We can say that the host node agrees to host the data owner's file and also be available a given percentage of the time. Every time the host node is audited between the time of initial data storage and agreed-upon duration to host the file, the result of that audit is stored. At the end of the agreed-upon duration, the ratio of passed audits/total audits is calculated, and if that ratio is >= the agreed upon ratio of availability, then the host node is paid, otherwise, they are not paid.
Edge cases of the new agreement:
- The host node cannot trust the data owner (who is also the auditor) to keep an honest record of the results of the audit. Depending on where these records are stored, it may be possible for the data owner to manipulate these records so that the data owner doesn't have to pay the host at the end of the agreement.