filecoin-project / fips Goto Github PK
View Code? Open in Web Editor NEWThe Filecoin Improvement Proposal repository
The Filecoin Improvement Proposal repository
Client can make an off-chain deal to miner. The miner will use existing or new CC sector to storage client's real data. If client want its off-chain data on chain, miner should upgrade the CC sectors to Regular Sectors with resealing.
Right now, the total network power is more than 6EiB,but the real storage is just only 15PB(https://storage.filecoin.io/). There are a lot of CC sectors with the zero deal. The miner should have an easy way to use the existing CC sectors to store the real data from the clients, no more resealing. So that it can solve the problem about the idle hardware equipment.
The currently method is that the pledged sector can be replaced with a new sector containing deals.(https://docs.filecoin.io/mine/lotus/sector-pledging/#upgrading-pledged-sectors). However, it requires the deal to publish and new sector to seal. It will take up more resources of the miners and also occupy more resources of the network. Client storage costs will also increase because of it.
The client can choose whether to publish its own data and deal on chain or not.
Storing the off-chain data will save the gas fee, which is so expensive when the base fee is high especially. Like Verified data, the miners can provide the client another lower storage price than on-chain mode. The clients also can download files from the miner by the file CID.
If the client needs, it can change its off-chain deal on chain. The miner will migrate the data into the regular sector.
Especially, when the client finds that the miner do not provide services any more, it can create an arbitration or appeal. If the miner lost the data, the miner needs to pay the customer compensation for the loss of data. Meanwhile, the power and pledge will also be slashed. (Perhaps we can discuss another FIP in detail)
Optimistic WindowPoSt largely obviates the need for FIP-0009, so we can consider reverting it. We could also wait until FIP-0013 drives down costs even further.
Placeholder issue, content to come.
Is your feature request related to a problem? Please describe.
Unable to extend the life of a sector that has deals in it from a client side.
Describe the solution you'd like
Extending a sector with deals expiration time with the amount of weeks needed.
On the client side it would be great to be able to specify it's an active deal and request a reseal with a new expiration date.
Resealing it as a new sector at the same miner would be fine, the data gets copied in a new sector and sealed again with new collateral and the rest of the process. Probably the easiest way to do it, as everything is available at the miner.
Describe alternatives you've considered
I considered copying the data from the old sector back to my client, and requesting a new deal..
Additional context
Add any other context or screenshots about the feature request here.
Background
Both miner and market actors have WithdrawBalance
methods.
However, either method is actually an attempt to withdraw a specified amount, it could be successful even when the available balance is less than required amount, that means, the actual amount withdrawn is equal to or less than the amount specified in the method parameters, and the method always return nil
, so that, there is no way to get how much amount actually withdrawn if we check the chain status and message info, e.g. from CLI or explorer.
Proposal
Simply add a return value to the method to indicate the actual withdrawn amount.
This will improve the visibility and traceability of FIL flow. This is important especially for miners who need to have a very clear balance sheet and financial report.
Currently, network upgrades (and incidents like https://filecoin.statuspage.io/incidents/ffhr434cd14c) can temporarily reduce chain quality and, in some cases, cause miners to miss Window PoSts. This is mitigated by the fact that no penalties are paid for missed Window PoSt. Unfortunately, power that wasn't proven is still lost until the next Window PoSt.
Introduce a "maintenance window" flag that can be set for a range of epochs. Any deadlines that end within a maintenance window will not be penalized for missed Window PoSts.
MaintenanceWindow() bool
function on the runtime. Implementations will return true
from this function when a maintenance window is in-progress.MaintenanceWindow()
returns true, assume that all partitions were correctly proven and move on without applying any penalties.MaintenenceWindow() bool
will return true starting at the upgrade epoch and ending some time after the upgrade epoch (likely 1 hour).Posting this FIP proposal here to open a discussion before pushing a PR. Would love the feedback! 🙏
---
fip:
title: Zero Knowledge Contingent Payment (ZKCP) over Payment Channels for fair retrievals.
author: @adlrocha
discussions-to: #73
status: Draft
type: Technical
category: Networking
created: 2020-02-22
spec-sections:
Filecoin retrieval deals currently happen primarily offline. Clients send periodic payment vouchers over a payment channel to pay for the retrieval of blocks of data to a provider. The provider sends blocks until it requires payment. When the provider requires payment, it pauses the data transfer until the client sends its next payment. This setup produces a clear unbalance in the exchange: if the provider requests a payment before the blocks paid for are sent, it can escape with the payment voucher and redeem it without sending any data to the client; and the other way around, if the provider sends the blocks before the client sends its payment, the client is able to run with the blocks without having to pay for them.
The aim of this FIP is to remove this unbalance in retrieval deals implementing all the primitives required to run a Zero Knowledge Contingent Payment protocol over Filecoin payment channels. This opens the door to the coexistence of different retrieval exchange protocols in the retrieval market, and the implementation of fair exchange of data over retrieval deals.
The unbalances in the current implementation of retrieval deals is vulnerable to unfair exchanges, where a client or a provider can take advantage of the other party. In the case of clients, they would be able to run with some blocks of data without paying the provider; while in the case of providers, they would be able to request a payment and run with the payment voucher without sending the corresponding blocks paid for.
A retrieval of data is currently conformed by the following steps:
Add a parameter in the negotiation of the retrieval deal (for instance in the '
RetrievalDealProposal
) to agree on the type of retrieval to be performed: mainly, pay-per-chunk
or zkcp
(at least for now).
When a retrieval deal of type zkcp
is accepted by a provider, the provider performs the following steps before sending the data:
- It generates a symmetric key k
, and uses k
to encrypt the chunk of blocks to be sent in the retrieval, c = Enc_k (data)
.
- It computes a hash of the key, y = hash(k)
, and builds a Zero Knowledge Proof, v
, that verifies E^-1_{hash-1(y)} == c
, i.e. that the ciphertext c
has indeed been encrypted using the key whose hash is y
.
The provider then sends over the data channel the encrypted data, c
along with the hash of the key, y
, and the proof v
, and requests a payment. In order for the provider to reveal the key, the client needs to send a payment voucher.
The client sends a payment voucher setting the field SecretPreimage = y
in the voucher. For the provider to be able to redeem the payment it needs to reveal the key, k
, for the client to be able to recover the data.
If the accepted retrieval is of type pay-per-chunk
, client and provider run the current implementation of the retrieval exchange seamlessly.
The basic implementation of the zkcp
retrieval assumes a single payment for all the blocks of data requested. To support an exchange with n
partial payments, the provider needs to generate n
different keys, encrypt each of the n
set of blocks with a different key, build n
proofs, and redeem the n
vouchers to reveal the keys to the client.
This proposal adds overheads to the current implementation of retrieval deals, and it may not suit the needs of every retrieval exchange. The rationale behind the proposal is to put in place all the constructions required to run a ZKCP protocol to perform fair-exchanges over Filecoin payment channels. This opens the door to the coexistence of several retrieval protocols so clients and providers can agree on the most suitable one to fulfill their needs in a retrieval deal.
There are already some proposals being discussed for alternative implementations of the ZKCP protocol with optimizations and modifications to circumvent some of its limitations. Having a basic implementation of ZKCP in Filecoin can open the door to the design and implementation of more retrieval exchange protocols for fair-exchange based in this basic construction. A few examples of the aforementioned alternative implementation of ZKCP retrievals can be found here:
This proposal is backward compatible. What's more, one of its goal is to enable the coexistence of different retrieval exchange protocols over the Filecoin retrieval network.
TBD
The main security considerations for this proposal involve:
v
in order to verify E^-1_{hash-1(y)} == c
, not leaking any information about the data or the key.Supporting the coexistence of several retrieval exchange protocols and fair-exchanges of data based on a ZKCP protocol gives flexibility to the retrieval network and retrieval deals, as clients and applications are able to choose the retrieval scheme that better suit their needs.
Moreover, this paves the way for the design and implementation of new retrieval exchange protocols, and further innovations in the retrieval market.
pay-per-chunk
exchange for fast retrievals, and a zkcp
fair retrieval for costly ones).TBD
Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
ProveCommits and PreCommits are creating network congestion leading to high base fee, leading to high cost for SubmitWindowPoSt and PublishStorageDeal.
Processing multiple ProveCommits at the same time can drastically reduce the gas used per sector, leading to a much lower gas usage from ProveCommits. We propose to add a new method: "ProveCommitBatched".
There are two parts that can be amortized:
From now on, I will call "Batching saving factor" the factor of gas saved by doing this.
This change should be done in conjuction with #25
With this mechanism, miners will prefer to batch multiple proofs together since they would be substantially reduce their costs.
This issue is intended for discussion. There are a number of details to work out before drafting a FIP.
The following table describes the proof size and the batch verification times.
#proofs | #snarks | size | verification time | efficiency* | savings** |
---|---|---|---|---|---|
1 | 10 | 1,920 bytes |
3ms |
1 | 1x |
10 | 100 | 19,200 bytes |
10ms |
3.3 | 3x |
20 | 200 | 38,400 bytes |
15ms |
5 | 4x |
30 | 300 | 57,600 bytes |
23ms |
7.66 | 3.91x |
50 | 500 | 96,000 bytes |
40ms |
13.3 | 3.75x |
100 | 1000 | 192,000 bytes |
53ms |
17.76 | 5.6x |
Here are some back of the envelope calculation to understand the advantages and disadvantages of this proposal:
More work needs to go into this, but preliminary:
ByteArrayMaxLen
Currently Filecoin does not support a full life cycle of an account from creating to termination, i.e. once an account is created, it will be always living in the network with a state, no matter it is obsoleted by the creator or not. This leads to two issues:
Add a termination method for non-singleton actors for vailable balance transferring and account termination.
The termination method has one parameter as beneficiary to get all available balance
Introduce collateral for heavy actors, e.g. miners, and the collateral will be return when it is terminated, to provide higher security guard.
This can be done step by step. For example:
This issue is intended for discussion.
This proposal is first published in a WeChat article and have done a survey, got the following vote data:
https://mp.weixin.qq.com/s/9GSvuuPb73TVIoA_foXlTQ
Do you support adding account termination method in Filecoin:
Overall, there are 81% voter support this proposal.
- One can not withdraw, or a miner can not mine; For this, we may consider the account user should take the full responsibility to check before termination
- The program might panic due to the corresponding state does not exist any more, this should be carefully taken care of
Or, we may not allow an account to be terminated when it is associated to another one, e.g. Owner, miner, controller, multisig signer, etc.
We may do Bitcoin like transferring, that is, 3 addresses are included is in a send message, From, To, Change addresses. But this makes send message heavier, and in most cases, the change address will be set to as From address. At the same time, it does not solve the the issue that obsoleted address still ocuppy network resource.
Facts:
Potential proposals that could evolve into a FIP:
There may be other solutions that we did not consider. Let us know what are your thoughts on the three proposals.
Once this gets more solidified we can turn it into an FIP proposal.
Read: FIP Draft
This proposal allows a substantial fraction of block rewards to be immediately available for withdrawal to allow for greater liquidity and decision freedom for storage miners. The majority of mining rewards continue to vest over 180 days as collateral to reduce initial pledge, to align long-term incentives, and to guarantee storage reliability.
Maximum sector lifetime is 540 days for now, there is no way to extend a deal lifetime now.
It is a little early to discuss about the implementation, but a design for it is needed.
extend a deal lifetime
someday, and plan it in the roadmap.Summary
It is currently not possible to extend the life cycle of the V1 proof sectors. This has caused a huge loss to the miners who sealed the V1 proof sectors in early days. Sealed sectors should only take windowPoSt challenges, security issues would not be involved. So we propose the V1 proof sectors' life cycle could be extended.
If the life cycle of these sectors cannot be extended, there will be a lot of sealing computation resources wasted to seal these sectors again. If miners have to seal sectors again, it will take much more collaterals and gas. More messages will make the chain congested, and other miners will have to pay more high gas fees too.
If the life cycle of V1 proof sectors is set to 180 days, miners would experience horrible loss and may cause even greater losses due to gas fee issues.
Improvement Goal
Allow the life cycle of V1 proof sectors to be able to extend like v8 proof sectors. Please see the detailed improvement plan: [https://github.com/filecoin-project/specs-actors/pull/1310]
There is a simple price manipulation strategy exploiting round trips to attack EIP-1559, which I have discussed extensively in the Ethereum community but is relevant to Filecoin as well. You can read more about this here. Also, see the ideas near the end of this thread for more details. The minimal remedy is to solve this problem by changing the multiplicative update formula into an additive one. Please let me know if anyone is interested, and I can provide more links and resources.
In the main page of the FIP repo - it'd be nice to have a table of FIPs that lists them by number and title, with a link to the FIP itself.
Alex Fox - Neonix
Slack: @NeonixAF f019551
The Filecoin Spec describes the purpose of the sector upgrade mechanism in 2.6.1.1 Sector Lifecycle:
It is reasonable to assume that miners enter the network by adding Committed Capacity sectors, that is, sectors that do not contain user data. Once miners agree storage deals with clients, they upgrade their sectors to Regular Sectors. Alternatively, if they find Verified Clients and agree a storage deal with them, they upgrade their sector accordingly. Depending on whether or not a sector includes a (verified) deal, the miner acquires the corresponding storage power in the network.
(https://spec.filecoin.io/#section-systems.filecoin_mining.sector.lifecycle)
And in 2.6.1.6 Upgrading Sectors:
In order for a Miner to maximize storage power (and profit), they should take advantage of all available storage space immediately, even before they find enough Clients to use this space.
[...]
If the newly ProveCommitted Regular sector contains a Verified Client deal, i.e., a deal with higher Sector Quality, then the miner’s storage power will increase accordingly.
(https://spec.filecoin.io/#section-systems.filecoin_mining.sector.adding_storage)
The intention seems clear that miners should always pledge capacity when they have the resources to do so, and there is no indication that performing a sector upgrade should be discouraged in any circumstance.
There is a situation where upgrading a sector will disadvantage a miner. When upgrading a sector with a deal having a duration shorter than the lifetime of the old sector, the new sector is given the same Expiration epoch as the sector that it replaces. Due to the calculation of the Sector Quality Multiplier, the new sector has a lower storage power than a sector which had been sealed without upgrading.
For example:
Even though the upgraded sector has the same total amount of power overall, the miner receives less benefit from this power because half of it has been pushed into the future when the network will presumably have grown and the reward received for that power will be smaller.
This could result in a miner being unfairly penalised after having pledged storage space to the network when they unexpectedly receive many short deals. Or it could provide a peverse incentive for a miner to not pledge space to the network if they expect to receive significant short verified deals, as they will make the rational decision to avoid this penalty.
The calculation of Sector Quality should be changed for sectors that are upgrading CC sectors.
For the purposes of the Sector Quality calculation, the sector should be treated as if its expiration is the epoch on which the longest deal in the sector expires. As such, the miner receives the same benefit from Verified Deal Weight, at the same time, as they would have had they sealed the sector without upgrading.
After the last deal in the sector expires, it reverts back to having a Sector Quality of a CC sector, and power reduces accordingly.
Additionally, as the sector's power has reverted back to that of a CC sector, the whole sector has essentially reverted back to being a CC sector, and as such it may make sense to permit upgrading it again subject to maximum allowed sector lifetime.
The above creates an issue for the average client - if you receive an allocation of DataCap and end up with an amount that is less than enough to make a deal (e.g. 4GiB) and more than the minimum deal size, you will be unable to receive further allocations to your address. This can create unnecessary friction for users who might then be required to create a new address in order to continue making deals.
Client addresses should be able to receive additional DataCap allocations to a given address.
Cases to consider:
It is worth also discussing how a client applying for DataCap from multiple Notaries may manifest - many of these solutions focus on process and tooling for Notaries, as what may appear as a race condition could also take the form of legitimate requests:
Background
Vesting is processed via State.UnlockVestedFunds
, which is called from PreCommitSector
, ConfirmSectorProofsValid
, WithDrawBalance
and the deadline cron. The original motivation for computing vesting on sector commitment is so that any vested funds could be used for pre-commit deposit or pledge.
The vesting table is quantized to 12-hr increments. Since vesting is processed in the deadline cron every 30 minutes, the calls during sector commitment almost always achieve nothing. The most it could achieve is to accelerate a release by 30 minutes.
Processing reward vesting is quite expensive as it loads and stores a sizeable array for the vesting table. For high-scale miners, committing many sectors per deadline (or even per-epoch), this represents a sizeable portion of total gas cost and chain bandwidth consumption.
Proposal
Remove vesting from PreCommitSector
and ConfirmSectorProofsValid
, leaving it to the deadline cron. If necessary, the miner's owner can trigger vesting via WithdrawBalance
(possibly with an argument of zero).
This will reduce the gas consumption of these methods substantially, freeing up chain bandwidth for other messages, reducing gas costs, and/or improving validation speed.
First proposed in filecoin-project/specs-actors#1258
fip: <to be assigned>
title: <FIP title>
author: <a list of the author's or authors' name(s) and/or username(s), or name(s) and email(s), e.g. (use with the parentheses or triangular brackets): FirstName LastName (@GitHubUsername), FirstName LastName <[email protected]>, FirstName (@GitHubUsername) and GitHubUsername (@GitHubUsername)>
discussions-to: <URL>
status: Draft
type: <Standards Track | Informational | Meta>
category (*only required for Standards Track): <Core, Networking, Interface, or FRC>
created: <date created on, in ISO 8601 (yyyy-mm-dd) format>
requires (*optional): <FIP number(s)>
replaces (*optional): <FIP number(s)>
This is the suggested template for new FIPs.
Note that an FIP number will be assigned by an editor. When opening a pull request to submit your FIP, please use an abbreviated title in the filename, fip-draft_title_abbrev.md
.
The title should be 44 characters or less.
If you can't explain it simply, you don't understand it well enough." Provide a simplified and layman-accessible explanation of the FIP.
A short (~200 word) description of the technical issue being addressed.
The motivation is critical for FIPs that want to change the Filecoin protocol. It should clearly explain why the existing protocol specification is inadequate to address the problem that the FIP solves. FIP submissions without sufficient motivation may be rejected outright.
The technical specification should describe the syntax and semantics of any new feature. The specification should be detailed enough to allow competing, interoperable implementations for any of the current Filecoin platforms (lotus, go-filecoin, Forest, Fuhon).
The rationale fleshes out the specification by describing what motivated the design and why particular design decisions were made. It should describe alternate designs that were considered and related work, e.g. how the feature is supported in other languages. The rationale may also provide evidence of consensus within the community, and should discuss important objections or concerns raised during discussion.
All FIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The FIP must explain how the author proposes to deal with these incompatibilities. FIP submissions without a sufficient backwards compatibility treatise may be rejected outright.
Test cases for an implementation are mandatory for FIPs that are affecting consensus changes. Other FIPs can choose to include links to test cases if applicable.
The implementations must be completed before any FIP is given status "Final", but it need not be completed before the FIP is accepted. While there is merit to the approach of reaching consensus on the specification and rationale before writing code, the principle of "rough consensus and running code" is still useful when it comes to resolving many discussions of API details.
All FIPs must contain a section that discusses the security implications/considerations relevant to the proposed change. Include information that might be important for security discussions, surfaces risks and can be used throughout the life cycle of the proposal. E.g. include security-relevant design decisions, concerns, important discussions, implementation-specific guidance and pitfalls, an outline of threats and risks and how they are being addressed. FIP submissions missing the "Security Considerations" section will be rejected. An FIP cannot proceed to status "Final" without a Security Considerations discussion deemed sufficient by the reviewers.
This template was derived heavily from Ethereum’s EIP template.
Copyright and related rights waived via CC0.
This is a discussion issue for #51 (Rendered).
Summary: Exempt direct SubmitWindowedPoSt
messages that execute successfully from base-fee burn (i.e., don't burn baseFee*gasUsed
).
If a miner sends a direct on-chain message to a miner actor's "SubmitWindowedPoSt" method, and the message executes successfully, immediately "refund" all the burned gas, instead of just refunding the overestimation. However, overestimation penalties, gas premium, etc. still apply.
Motivation:
This proposal is a short-term stopgap to reduce the impact of rising base fee on continuously proving existing storage. In principle, BaseFee*GasUsage is burned to compensate the network for the resources consumed by messages and ensure incentive alignment. However, SubmitWindowPoSt is a required message to continue mining operations in Filecoin.
Unfortunately, such long-term solutions are complex and cannot be properly implemented and tested on the required timescale (before the holidays). This proposal is a short-term mitigation until those solutions are ready.
Filecoin miners are subject to pay storage fault and consensus fault fees. These are paid using locked funds (ie, unvested block rewards) and then the miner’s available balance. If these two are not enough to cover the total fee amount, the remaining part to pay is recorded in a variable (FeeDebt) in the miner’s state and the miner is declared “in fee debt”.
For a miner in debt, the call to the methods PreCommit, DeclareFaultsRecovered and WithdrawBalance fails. Moreover this miner can not mine blocks. The fact that a miner in debt can not recover faults creates an undesired loop situation, in which a miner with faulted partitions can not recover, keep increasing its debt (note that continued faults cause more fees), eventually leading to sector termination if the miner does not repay their debt after 14 days. This is especially bad for small miners with little balance. To avoid this we propose to allow recovery of faults for miners in debt.
Currently Filecoin sectors can only store deal data referenced by on-chain deals. CC sectors and non-deal areas of sectors must store null (\0
) bytes to be verifiable on-chain. This limitation is caused by how the UnsealedCID (CommD) is computed in the miner actor.
Unfortunately publishing deals is a very expensive on-chain operation, especially for smaller pieces where this cost it can be the main part of the storage fee. Off-chain deal support / alternative markets (e.g. more scalable storage market on ETH2.0 with data stored on Filecoin) seem to be good solutions to this problem, at least for non-FIL+ deals.
This proposal is only the first, but major, part of what needs to be done to support off-chain deals. It's likely that some off-chain deal protocols will require additional actor methods to be implemented (for example a method to check if a sector wasn't terminated early for settling payment channels)
Currently, when precommitting sectors, a number of DealIDs can be specified with SectorPreCommitInfo
:
// Information provided by a miner when pre-committing a sector.
type SectorPreCommitInfo struct {
SealProof abi.RegisteredSealProof
SectorNumber abi.SectorNumber
SealedCID cid.Cid `checked:"true"`
SealRandEpoch abi.ChainEpoch
DealIDs []abi.DealID // ****
Expiration abi.ChainEpoch
ReplaceCapacity bool
ReplaceSectorDeadline uint64
ReplaceSectorPartition uint64
ReplaceSectorNumber abi.SectorNumber
}
This information is then stored in miner actor state, and used to compute UnsealedCID when verifying PoRep by turning specified DealIDs into []abi.PieceInfo
s, and calling the ComputeUnsealedSectorCID
syscall through the storage market actor.
type abi.PieceInfo struct {
Size abi.PaddedPieceSize
PieceCID cid.Cid
}
ComputeUnsealedSectorCID(reg abi.RegisteredSealProof, pieces []abi.PieceInfo) (cid.Cid, error)
We can change or create a new version of the SectorPreCommitInfo
miner actor struct, changing the DealIDs
array to a new array which allows specifying non-deal PieceCIDs:
+type SectorPieceInfo struct{
+ PieceCID *cid.Cid `checked:"true"` // CommP
+ PieceSize abi.PaddedPieceSize
+
+ DealID abi.DealID
+}
// Information provided by a miner when pre-committing a sector.
type SectorPreCommitInfo struct {
SealProof abi.RegisteredSealProof
SectorNumber abi.SectorNumber
SealedCID cid.Cid `checked:"true"` // CommR
SealRandEpoch abi.ChainEpoch
- DealIDs []abi.DealID
+ Pieces []SectorPieceInfo
Expiration abi.ChainEpoch
ReplaceCapacity bool // Whether to replace a "committed capacity" no-deal sector (requires non-empty DealIDs)
// The committed capacity sector to replace, and it's deadline/partition location
ReplaceSectorDeadline uint64
ReplaceSectorPartition uint64
ReplaceSectorNumber abi.SectorNumber
}
When computing UnsealedCID for PoRep verification, []SectorPieceInfo
would be turned into []abi.PieceInfo
for the ComputeUnsealedSectorCID
actor syscall as follows:
PieceCID
is null
, assert that PieceSize==0
, then get abi.PieceInfo
for the referenced DealID
from the market actorPieceCID
is not null, assert that DealID==0
, check the multicodec/multihash the same way it's checked in the storage market actor, then create abi.PieceInfo
using specified PieceCID/PieceSize[]abi.PieceInfo
to allow interleaving non-deal and deal data(it's possible to save 1 byte per entry by changing PieceSize/DealID
to a single uint64 value (DealIdOrPieceSize
), and processing it based on whether PieceCID
is null or not)
Depending on implementation details, this proposal may involve a relatively major state migration. We should look into ways of limiting that.
Publishing storage on-chain deals already has multiple kilobytes of read/write overhead. Each abi.DealID
entry is 5B (1B cbor header, 4B for integer data). Depending on implementation, a dealID SectorPieceInfo
entry will be 7 or 8 bytes, which is a negligible difference
SectorPreCommitInfo
without breaking the old methodPersisting a slack conversation here
Hey @Kubuxu I think FIP0011 has a mistake. It claims that current consensus fault penalty is 5 * BlockReward
here but its actually just BlockReward
from here as this function both multiplies by 5 and divides by 5.
If I understand the motivation behind FIP0011 I think this means the consensus fault reporting reward should become BlockReward / 20
, the current max possible. Let me know if you're in agreement on all this. If so would you mind amending the FIP to fix this?
FYI @placer14 us resolving this blocks your implementation PR
Full smart contract capabilities will come to Filecoin, it has been in the plans since the beginning. Many people ask this, so i'm starting this issue to track the conversation. We should submit proper FIPs to add the capabilities.
We all know the Owner Address is the key of miner, even we can delete owner address after we change owner address, but when we withdraw FIL or must keep owner address stay in the lotus for some reason, it's still dangerous.
Read: FIP Draft
High-level discussion of the Principles and structure should happen in this issue.
Specific discussions on Mechanisms and Operations should happen in this repository.
cc @jnthnvctr
This proposal is an idea of mine to reduce coupling between the storage power mechanism and deal market. Coupling between these two limits design freedom around problems like supporting exponential growth in storage capacity or making deals much more scalable. This proposal is written independent of proposals like Neutron (#119). An analogous idea would apply to supersectors in order to remove linear-in-deals costs from storage maintenance. This proposal could be made either prior to Neutron (simplifying it) or subsequently.
Filecoin network storage growth will eventually be limited by the chain computation costs associated with maintaining storage if a linear amount of state is accessed or retained per sector (where sectors are a fixed size). The Neutron proposal (#119) resolves this for committed-capacity sectors, but sectors with deals still have per-sector (and per-deal) metadata. Per-sector on-chain data includes the sector->deal mapping and per-sector heterogeneous weight, pledge and penalty values. These are all required because verified deals alter a sector’s power as a means of adjusting reward distribution.
Maintaining the facility for heterogenous sectors constrains freedom for more scalable designs, even though most sectors don’t have deals. The premise that most sectors don’t have deals is not something to rely on, though. In the long term we aim for many more deals and a much greater proportion of committed storage to be in active use.
This proposal aims to reduce deal market limitations on the scale and efficiency of onboarding and maintaining exponentially larger amounts of capacity.
These goals are to be sought within existing product and crypto-economic constraints around sound storage and economic security.
The current storage power mechanism assigns a different power value to equal-size sectors based on the size and duration of Filecoin Plus (FIL+) verified deals. This is a means of incentivising useful storage, delegating the definition of “useful” to an off-chain notary. The incentive for storing verified deals is the increased block reward that may be earned from the increased power of those sectors, despite constant physical storage and infrastructure costs. In essence, the network subsidises useful storage by taxing the other power.
Storage power has two different roles, currently coupled. One is to secure the network through raising the economic cost of some party maintaining a significant fraction of the total power, and the other is to determine the distribution of rewards. The block reward is split between rewarding security and subsidising useful storage. This coupling theoretically means the verified deal power boost reduces the hardware cost to attack the network by a factor of 10, if colluding notaries would bless all of a malicious miner’s storage (this is an impractical attack today, though).
This subsidy could be made much more direct, reducing complexity and coupling between the storage market and individual sectors, and clearly separating security from reward distribution policy.
Every sector has uniform power corresponding to its raw committed storage size, regardless of the presence of deals. This removes the concepts of sector quality and quality-adjusted power, and makes all bytes equal in the eyes of the power table. This would remove the DealWeight and VerifiedDealWeight fields from SectorOnChainInfo. Network power and committed storage are now the same concept and need not be tracked separately by the power actor.
Uniform power also means that all similarly-sized sectors committed at the same epoch would require the same initial pledge. Similarly the expected day reward and storage pledge (parameters to a possible future termination fee calculation) depend only on activation epoch. The complicated values maintained to track penalties for replaced CC sectors become unnecessary. Historical pledge/reward values may be maintained just once for the network by the power and reward actors. We currently store each of these numbers in chain state some ~500 times per epoch (@ 50PiB/day growth).
With uniform sector power, the power of groups of sectors may be trivially calculated by multiplication. Window PoSt deadline and partition metadata no longer need to maintain values for live, unproven, faulty and recovering power, but the sector number bitfields remain. Processing or recovering from faults does not require loading each sector’s specific values. The complexity of code and scope for error in these various derived data is much reduced.
This ability to aggregate by multiplying becomes instrumental to Neutron-like ideas around hierarchical storage accounting with deals. Without this, supporting partial faults requires on-chain metadata about the individual sectors (with deals) that comprise a supersector, restoring a linear-in-power network cost.
Note that this proposal does leave the per-sector deal IDs on chain. After this proposal and Neutron, this would be the only piece of per-sector data retained.
A miner’s chance of winning the election at each epoch is proportional to their share of the raw byte power committed to the network and non-faulty. Winning the consensus reward remains a lottery.
The market actor maintains:
As payments are periodically processed for each deal (currently every 100 epochs), the market actor adds the deal’s size multiplied by the update period epochs to the provider’s space-time tally for the current reward period.
After a reward period completes, the ratio between verified deal space-time of each provider gives their share of the deal subsidy to be claimable. A miner can claim a corresponding share of the total deal subsidy earned at any point up until the reward period expires (e.g. 60 days). Upon claiming a deal subsidy, the reward funds are inserted into the miner’s vesting schedule, alongside block rewards.
At every epoch, the reward actor queries the power actor for the total network power (= storage) and the market actor for the total verified deal space. The total block reward to be paid is then split into the power reward and the verified deal subsidy according to the ratio of storage to 9*deal space. The block reward is paid as usual, and the deal subsidy is sent to the market actor.
The primary motivation for this proposal is to remove per-sector account-keeping metadata in order to unlock exponential scaling of storage. It changes reward behaviour in a couple of ways that we must verify as being beneficial.
Everyone, for the real and effective data verification of filecoin, our team has proposed a solution that can effectively solve the data verification problem and help the filecoin ecological application. Link: filecoin-project/community#167
by Extend Labs
Abstract:To promote the storage of meaningful data, Extend Labs suggests the Filecoin community develop a method of Proof-of-Quality and reward the storage miner who has passed the proof. The basic process of the proposed PoQ is as follows. In the training phase, first, the storage miners publish the features of the data under differential privacy; second, the model miners train the PoQ model using distributed machine learning methods on the published features. In the testing phase, the storage miners load the latest PoQ model, submit the proof of quality and get the verified clients reward.
On the existing Filecoin system, most of the stored data are randomly generated data but not meaningful data, which makes the community difficult to realize its ambition. How to promote meaningful data storage and applications has become a key issue of the Filecoin community. To this end, the Filecoin community has launched the “verified clients” reward.
The key to the success of “verified clients” reward is how to build an efficient, reliable, and adaptable way to verify the quality of the data. In this context, Extended Labs suggests the community develop the “Proof-of-Quality” (PoQ). PoQ is a proof method for the quality verification of the content of the data stored by miners, and it is also the only way for storage miners to obtain the “verified clients” reward.
The main challenges of the PoQ include: 1) Meaningful data storage puts forward higher requirements for data privacy, which means the PoQ methods usually cannot directly access the data; 2) The pattern of meaningful data changes dynamically, which means the PoQ methods should be able to adjust in conjunction with community development; 3) Due to the huge attraction of the “verified clients” reward, the storage miners have greater incentives to forge the meaningful data, which is also a core issue that needs to be considered in PoQ design.
Therefore, an effective PoQ should be able to ensure data privacy, adjust with the development of the community, and prevent malicious attacks by the storage miners. Based on the above considerations, we propose a PoQ scheme based on federated computing based on existing privacy computing and distributed machine learning.
There are two stages of the proposed PoQ method: PoQ model training and PoQ model testing. In the PoQ model training phase, federated learning is used to train the PoQ model, and the miners participating in the model training will receive model training rewards; in the PoQ model testing, each storage miner generates the PoQ certificate on its stored data and obtains the “verified clients” reward.
To train the PoQ model efficiently, we employ differential privacy to publish the features of the data and use the distributed machine learning methods to train the PoQ model, under the federated computing framework [1]. The framework is shown in Figure 1. It includes three components:
Differential privacy data publish module: In this module, the features of the data will be calculated, desensitized, and finally published under differential privacy. Combined with actual needs, the data features can be the histograms of the data’s n-Gram segments. A possible solution for this module is the "Differentially private histogram publication" method [2], which is a widely used histogram publication method with grantees.
The local model training module. In this module, a specific part of the PoQ model is trained locally on a batch of samples or a subspace of the features on the differential privacy published data. The locally trained models will be reported to the global model training miners.
The global model training model. In this module, we obtain the global PoQ model by merging and tuning the local models. The split strategy of the local and global model, and the merging strategy of the global model can refer to "Scaling distributed machine learning with the parameter server", which is a widely used method in the field of distributed machine learning [3].
In practice, the differential privacy data publish module is deployed and completed by each storage miner, the local model training module is deployed on the storage miner or the model training miner, and the global model training model is completed by several (3, 5, or more) randomly selected model training miners to avoid the Sybil attack.
The generation of PoQ is shown in Figure 2. Similar to the training process, this phase will also guarantee data privacy. It has two modules as follows:
Differential privacy data publish module: this model is the same as the training phase.
PoQ generation module: In this module, first, obtains the latest PoQ model from the model training miner; second, estimates the PoQ on the features published under differential privacy; finally, submits the PoQ score to get the “verified clients” reward.
The training samples with accurate labels are the key to the success of PoQ. Possible solutions to collect the training samples include:
In the initial stage, the samples can be collected according to the meaningful data rules. The rules are based on the community consensus.
In the update stage, the samples can be collected according to meaningful actions on the data, such as the retrieve, update and so on.
Model training reward: In order to encourage miners to participate in PoQ model training, PoQ model training rewards are given to miner nodes participating in local model training and global model training. The local model training and global model training rewards are related to the amount of data involved in the training.
Verified clients reward: storage miners that passed the PoQ will get the verified clients reward. The score returned by PoQ can be quality-adjusted power.
In summary, we propose to build a PoQ method based on the latest advanced privacy computing and machine learning technologies. The proposal is featured with:
Use both the rules provided by the community and the meaningful behaviors performed by the users, as the criteria of the meaningful data.
Devise a PoQ model training method with privacy grantee. Its characteristics include: using differential privacy to publish the features of the data; training the model using the distributed large-scale machine learning methods.
Use the trained PoQ model to verify the quality of the data. Its characteristics include that the storage miners use the latest PoQ model to verify the quality of the data and get the verified clients reward.
Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
Xu J, Zhang Z, Xiao X, et al. Differentially private histogram publication[J]. The VLDB Journal, 2013, 22(6): 797-822.
Li M, Andersen D G, Park J W, et al. Scaling distributed machine learning with the parameter server[C]//OSDI. 2014: 583-598.
As holiday seasons approaching, I would propose a new cool feature for Filecoin, Pausable Mining.
Why do we need it:
So how does it work:
Miners send message to the chain to inform about its break time. Pausable mining is a fix period (1-2 week or a number of epoch), during pausing period, miner still need to participate in the network but less strictly (1 WindowsPost in 72 hours, pelnaty after 2nd missing WindowPost). The pausing miner will not able to mine any block, accept any deals or do CC sector. However, it still allow for retrieval deals.
Some people from the Filecoin team have been working on the next iteration of scalable storage growth and capacity for the Filecoin network. The recent Hyperdrive network upgrade unlocked a big multiple of capacity, but we expect mining demand to rise over time to meet this and again be limited by blockchain throughput. In the next iteration of improvements we aim to solve this problem for the long term, enabling exponential network growth. This effort is known as project Neutron (after the density of neutron stars).
We're still fleshing out many details ahead of a full FIP, but I'm filing this issue to show where we're headed and as a reference for other efforts. We'll publish more extensive design documents once we're more confident in the approach.
The Filecoin network’s capacity to onboard new storage and to maintain proofs of committed storage are limited by blockchain transaction processing throughput. The recent Hyperdrive network upgrade raised onboarding capacity to about 500-1000 PiB/day, but we expect this capacity to become saturated.
As onboarding rates increase, the fixed amount of network throughput consumed by maintaining the proofs of already-committed storage will increase, eventually toward a significant cost for the network.
Validation of the Filecoin blockchain is subject to a fixed amount of computational work per epoch (including state access), enforced as the block gas limit. Many parts of the application logic for onboarding and maintaining storage incur a constant computational and/or state cost per sector. This results in blockchain validation costs that are linear in the rate of growth of storage, and in the total amount of storage committed.
Linearities exist in:
We wish to remove or reduce all such linear costs from the blockchain validation process in order to remove limitations on rate of growth, now and in the long future when power and growth are significantly (exponentially) higher. SNARKPack goes a long way toward addressing the linear cost of PoRep and PoSt proof verification. However, there remain linear costs associated with providing the public inputs for each sector’s proof.
Our goal is to enable arbitrary amounts of storage to be committed and maintained by miners within a fixed network transaction throughput.
This means redesigning storage onboarding and maintenance state and processes to remove linear per-sector costs, or dramatically reduce constants below practical needs. We want to do this while maintaining security, micro- and macro-economic attractiveness, discoverable and verifiable information about deals, and reasonable miner operational requirements.
This effort is seeking a solution that is in reach for implementation in the next 3-6 months (which means relying on PoRep and ZK proof technologies that already exist today), and that is good enough that we won’t have to re-solve the problem within a few years
Of course there exist other, orthogonal approaches to the general problem of scaling, but these are generally longer and harder propositions (e.g. sharding, layer 2 state).
This proposal does not attempt to solve exponential growth in deals, except by making it no harder to solve that problem later. We think this sequencing is reasonable because (a) deals are in practise rare at present, and (b) off-chain aggregation into whole-sector-size deals mitigates costs in the near term. We expect exponential deal growth to be a challenge to address in 2022.
The premise behind this proposal is that we cannot store or access a fixed-size piece of state for each 32 or 64 GiB sector of storage, either while onboarding or maintaining storage. Specifically, we cannot store or access a replica commitment (CommR) per sector, nor mutate per-partition state when accounting Window PoSt. CommR in aggregate today accounts for over half of the state tree at a single epoch, and Window PoSt partition state manipulation dominates the cost of maintenance.
The key design idea is to maintain largely the same data and processes we have today, but applied to an arbitrary number of sectors as a unit. The proposal will redesign state, proofs and algorithms to enable a miner to commit to and maintain units of storage larger than one sector, with cost that is logarithmic or better in the amount of storage. Thus, with a fixed chain processing capacity, the unit of accounting and proof can increase in size over time to support unbounded storage growth and capacity. We will assume that miners will increase their unit of commitment if blockchain transaction throughput is near capacity.
The Mission of Filecoin is to create a decentralized, efficient, and robust foundation for humanity’s information. As such, one of the core goals of Filecoin is to create strong incentives for capacity committed to the network to be utilized in storage deals for valuable data. To achieve that, Filecoin clients and storage providers should be able to propose and accept negatively priced deals - aka deals where the storage provider pays the client to store their Filecoin+ data.
Right now, demand from storage providers for verified storage deals (made by clients with FIL+ datacap) is very high, since it confers a 10x multiple in block rewards proportional to the verified deal size (which helps storage providers quickly achieve and maximize profitability). However, access to these verified deals is constrained by the number of new storage clients onboarding data onto the network. To compete for this high-value resource, miners should be able to offer not just FREE storage and retrieval for verified deals, but NEGATIVELY PRICED storage/retrieval deals.
Negatively priced deals (that share the 10x multiplier returns from FIL+ data with the storage client) will help attract new high-value clients to the Filecoin market, thereby increasing the network's utility and the overall number of FIL+ deals available. As @jbenet mentioned in his EthCC talk, this is a super exciting opportunity that is uniquely available in Filecoin. Instead of having to build these auction/incentive systems outside the deal-making structures of Filecoin, we should simply encode the ability for storage providers to set a negative ask price.
A WindowPoSt message is a “unique” kind of message because it has an expiration (i.e., it has to go on chain before a given epoch in order to be valid and effective). Because of this, we need to guarantee that enough windowPoSt messages are selected and included by a block producer, even when there is message congestion.
Allocate dedicated block space for WindowPoSt messages.
We introduce a message classification system:
Each message type now has its own gas pricing mechanism. Each mechanism is the same as in the Dynamic Network Fee mechanism, but they are independent. More precisely:
We introduce variables: BlockGasLimit_A, BlockGasTarget_A, BlockGasLimit_B, BlockGasTarget_B
Each block now has two base fees: baseFee_A and baseFee_B. Both base fees are computed using the formula “NewBaseFee = OldBaseFee + delta*OldBaseFee”, where delta depends on the difference between the actual gas limit of the block (i.e., the sum of each included message’s gas limit) and the BlockGasTarget. The differences among the two are
In the computation of the delta for baseFee_B (resp. baseFee_A) we use BlockGasTarget_B/2 (resp. BlockGasTarget_A/2) and consider msg.GasLimit taken from messages of type B (resp. Type A) only;
Currently the delta is a value in the range [-⅛, ⅛], we can keep this range for both or use two different ranges.
One problem @Kubuxu pointed out for this proposal is that if we reserve too little for WindowPoSt then the window post partition will run up the corresponding base fee very high, on the other side reserving too much decreases the chain throughput and creates problem for the other messages (especially precommit/proveCommit).
(same as #49 )
ProveCommits and PreCommits are creating network congestion leading to high base fee, leading to high cost for SubmitWindowPoSt and PublishStorageDeal.
(however, differently from #49)
ProveCommits messages and gas used scale linearly with network growth.
(similar to #49)
Processing multiple ProveCommits at the same time can drastically reduce the gas used per sector, leading to a much lower gas usage from ProveCommits. We propose to add a new method: "ProveCommitAggregated".
(differently from #49)
The ProveCommitAggregated method allows for gas used in the network to scale sub-linearly with the growth of the network. A miner can submit a single short proof for many ProveCommits and the gas used for verification is sublinear in the ProveCommits aggregated. In other words, miners could be making a single ProveCommitAggregated transaction per day, instead of one per sector or one per batch of sectors.
There are ~6M individual SNARK are published on chain per day, largest miner publishes ~600k of them. With this solution - at least in theory - we would need a single SNARK per miner ~700. If we assume that miners only batch every 1,000 proofs on average, the SNARKs published per day would be ~6,000.
There are two parts that can be amortized:
The Protocol Labs team in collaboration with external researchers and engineers has improved the performances of the rust implementation of the IPP protocol (see ripp). The inner product pairing protocol for aggregating groth16 proofs has been described in this paper.
In high level, the idea is the following: given X Groth16 proofs, we can generate a single proof that these were correctly aggregated in a single proof.
Our preliminary result show that a prover can aggregate up to ~65,000 SNARKs (6,500 ProveCommits) in a size of ~78kB and with verification time of ~150ms.
Note the this type of aggregation sits on top of the existing SNARKs that we do. In other words, there is no need for a new trusted setup.
The size of an aggregated proof grows logarithmically in the numbers of proofs to aggregate, differently with batching, where the proof size scales linearly.
In other words, proposal #49 ProveCommitBatched has a limitation in the numbers of proofs to be aggregated, while ProveCommitAggregatedIPP does not. This opens up the possibility for miners to submit a daily single proof of new storage being added.
With this mechanism, miners should always prefer to aggregate multiple proofs together since they would be substantially reduce their costs.
This issue is intended for discussion. There are a number of details to work out before drafting a FIP.
This is a test that aggregates 65536 SNARKs (~6,500 poreps) with a proof size of 78kB and verification time of 197ms
Proof aggregation finished in 41685ms
65536 proofs, each having 328 public inputs...
Verification aggregated finished in 197ms (Proof Size: 78336 bytes)
Note there is a separate line of investigation to propose ProveCommitAggregated with Halo instead of IPP, however it seems that IPP would be ready to be used faster than Halo.
Window PoSt messages are necessary for ongoing maintenance of storage power. Verifying the submitted proofs is expensive and when the gas base fee rises (due to congestion) these messages become expensive. In bad cases, for small miners with mostly-empty partitions, this cost can exceed their expected reward from maintaining power. We need to ensure that these messages are cheap for miners, even when specifying a very high gas fee cap.
Note the framing of the problem here assumes congestion. There is a separate and just-as-pressing problem of reducing congestion. But even if we make great improvements there, congestion will happen sometimes in the future for different reasons (E.g. massive popularity! Deals! DeFi!) and we'll need Window PoSt to remain cheap and effective.
This is an alternative to the Fast Track for Window PoSt and related proposals.
Don't verify most Window PoSt proofs "on chain". Assume they are valid, and implement a challenge mechanism for external parties to dispute an invalid proof, verifying on-chain only at that point.
With this mechanism, the vast majority of Window PoSt proofs would never be validated on chain. The network cost of maintaining storage would be effectively constant, rather than linear in storage as it is today (at sufficient scale it would revert to linear for the work of submitting the proofs, but over that timeframe expect to develop other aggregation techniques).
This issue is intended for discussion. There are a number of details to work out before drafting a FIP.
The proof verification syscall accounts for 88% of SubmitWindowPoSt's gas cost, and loading the thousands of SectorOnChainInfo that form public inputs for the bulk of the remainder. Both of these are avoided for valid proofs, and the cost paid by a challenger rather than the miner. So the gas cost reduction will be in the range of 10-20x (maybe more for full partitions).
Rather than every node on the network validating every proof, we instead need that at least one node verifies every proof.
Note that a similar technique could be applied to ProveCommit proof verifications too, but that's a bit trickier and out of scope for this proposal.
This proposal does not include a reward for a successful challenge, though one could be added.
Off-chain verification lends itself to continued technical improvements without the need for protocol upgrades.
Some risks to consider and satisfy ourselves about.
The miner state update required for a successful challenge will be very similar to the state change currently implemented during the deadline cron when a "missed PoSt" is detected. Snapshotting the partition state is expected to be cheap since we can content-address state that already exists in the state tree.
Since submitting a proof will no longer involve loading all the sector information, we can probably increase the number of partitions that may be batched into a single submission (currently, and somewhat arbitrarily, set to 4).
A bunch of details to be worked out include:
The tie-breaker described in the spec for EC has never been implemented. Although not critical it would help miners converge more easily towards a common chain. This means that when two forks with the same weight occur all the miners that have received both forks will mine on the same one. Without a tie-breaker they would (randomly?) mine on both and may extend the forks for longer.
For two tipsets of the same weight we choose the one with the smallest electionProof. In the case where two Tipsets of equal weight have the same min Election proof, the miner will compare the next smallest Election proof in the Tipset (and select the Tipset with the next smaller proof). This continues until one Tipset is selected.
Any solution that is both deterministic and non-biasable (i.e. an adversary may not make their chain more likely to be selected by the chain selection algorithm in the case of forks of equal weight) may work.
The ElectionProof is unbiasable and hence is a good candidate for tie-breaker. Any deterministic rule other than minimum
would work just as fine.
Draft FIP here: https://docs.google.com/document/d/1VHm_0rh9XCcjJsrnpbLtbFpLNRmOgkEoNTNlt0NHiOQ/edit?usp=sharing
Chain congestion due to sector sealing messages -- PreCommit
and ProveCommit
-- is driving up the BaseFee and making WindowPosts, Deals, and other messages very expensive. Other proposals (#24, #24-b, #42) propose ways to make WindowPost cheaper or get a dedicated control gas plane, but most of those solutions will take significant time to implement.
It seems that Miners for now have settled on a BaseFee expense between 1-5 nFIL (gas expenditure dominated by PreCommit
and ProveCommit
.
The other solutions discussed are the right thing to do. But in the meantime:
We might alleviate the pain by introducing a gas fee rate increase for PreCommit
and ProveCommit
messages. This means multiply the gas fee by a SealingFeeRate
-- a factor, say 10x
or even 20x
. This will cause parties that are sealing massive amounts currently (in this difficult congestion period) to arrive at a BaseFee about 10x cheaper.
Scales with BaseFee. because this is a multiplication on top of the gas costs of those messages, it will scale with the BaseFee, and will cause the network to find the miners' price point much faster.
Burn fees pass to miners sealing. This is unlikely to reduce burn fees, only move the proportion paid by most users (sp WindowPosts) to be paid by miners growing their storage significantly.
Downside: harms sealing new storage. This harms all sealing, not just CC sectors. This will hurt new sectors with storage, as they will be more expensive to seal. This may be an acceptable price to pay until we get a better solution in.
Network Policy Lever. This creates a network policy lever that trades off between cheaper capacity onboarding or cheaper everything else.
This should be very easy to implement. this is why I'm proposing it. This should be a relatively straight forward change to introduce in gas accounting.
The current process requires a pull request to be opened before the FIP number is assigned which requires the author to guess the next number the editor will assign with possible conflict (e.g. #13) or use a temporary filename (#9). This is going to get messier as the volume of proposals increases.
I propose that FIP-1 be amended to require a Github issue to be opened first, containing the substance of what would become the WIP version of the FIP. The FIP Editor reviews and assigns a FIP number. Then the author proceeds to open a PR for the FIL to put it in WIP status.
Due to optimizations and evolving software I propose changing Gas charged for following operations:
Call Name | Old Gas | New Gas | Change |
---|---|---|---|
OnVerifyPostBase | 374296768 * | 117680921 | -68% |
OnVerifyPostPerSector | 42819 * | 43780 | +2% |
OnIpldGet | 75242 | 114617 | +52% |
OnIpldPutBase | 84070 | 353640 | +320% |
StorageGasMultiplier | 1000 | 1300 | +30% |
* - OnVerifyPost currently has arbitrary 2x discount applied, Old Gas numbers are including this discount, New Gas do not
Above changes can be divided into four categories:
For CC sectors, allow miners to choose if they want to seal empty data or pack data pieces which could be real for CC
sector, and it can be verified accordingly.
Currently in seal process, miners pack empty pieces (all zeros) into CC sectors, after porep finished, create precommit
messages with empty dealID information. In verification of commit messages, verifier will get the DealID from
corresponding precommit info, and if DealID is empty, verifier recover the unsealed id by all zeros, which limit the CC
sector to seal only zeroed data.
We propose design to allow miners to pack real data for CC sectors, add unsealed id in precommit message and keep it on
chain. It introduce the possibility for CC upgrade without re-seal, and turn the piece into verified data afterwards (
Separated FIPs).
Extra onchain storage for “unsealed ID” in SectorOnChainInfo will be required. As this field provides the possibility
for post-verification between data pieces and unsealed ID.
Fully backwards compatibility is expected
N/A
No impact on existing incentives, but provides extra flexibility to miners to pack real data in CC sectors. As the
onchain unsealed id is available, the cheaper CC upgrade and new process to assign DataCap is possible for the follow up
FIP
WIP
Copyright and related rights waived via CC0.
This is a very long-term idea, and shouldn't be considered until after FIL+ (formerly verified clients) is deployed and working smoothly (see FIP-3 for more on that). Also, if this has been proposed elsewhere, just close the issue; I just figured I'd jot down the idea quickly.
Anyway, here's the thought:
It could be interesting to (one day) extend the Filecoin Plus program to also include verification of miners, instead of just clients (basically, enable the protocol to incorporate and compute on just a little bit of trust on the other side of the storage market as well). In this model, a verified miner would be audited occasionally (e.g. to show that they have some sort of accountable legal entity, have the right kinds of redundancies & data loss prevention). The incentive to do this would be that they don’t have to submit proofs (allowing them to save energy and decrease their operating costs). As I see it, one big reason for PoSts is that the network is permissionless - so all accountability has to be derisked with collateral and computability.
It's a nontrivial growth in the scope for the project, but it seems like a possible way to increase scalability, as there'd be less data in the chain if there are fewer proofs.
Problem
If a miner finishes sealing a sector during a challenge window, the sector gets assigned to the next deadline. This leads to sectors spread across multiple deadlines, even if a deadline is not filled with 2349 sectors.
Example:
deadline partitions sectors (faults) proven partitions
0 1 686 (0) 0
1 1 13 (0) 0
2 1 9 (0) 0
This leads to unnecessary messages on the chain, and also unnecessary fees paid by miners, especially when wPoSt-fees are as high as they are in this time of writing.
Proposed Solution
In a earlier discussion on Slack, nickle proposed two solutions:
Discussion
Nickle noted that solution 1 might be messy regarding security.
Read: FIP Draft
Given the state of the network, honest miners are sometimes disproportionately penalized for operational failures by the SectorFaultDetectionFee (SP), especially when missing a WindowPoSt. This proposal reduces fees on detected faults, without sacrificing much of the security or the incentive to provide reliable storage.
Specifically:
The miner PreCommitSector
method only supports committing a single sector at a time. It's one of the two highest frequency methods observed on the chain at present (the other being ProveCommitSector). High-growth miners commit sectors at rates exceeding 1 per epoch. It's also a relatively expensive method, with multiple internal sends and loading and storing state including:
AllocatedSectors
bitfield (read and modify)PrecommittedSectors
HAMT (read and modify)Sectors
AMT (read)PreCommittedSectorsExpiry
AMT (read and modify)A PreCommitSectorsBatch
method has potential to amortize some of these costs across multiple sectors. If miner operators implemented a relatively short batch aggregation period (a few epochs), the number of invocations could be reduced significantly, and some of the state manipulations above reduced in proportion.
The proposal is to remove the auction mechanism from rewards for reporting consensus faults
See: #60
All participant today are suffering from filecoin congestion issue. To make things worse, top miners who have more power share of the network produce more traffic to keep their leverage. Thinking about in real world, you have to pay congestion fee more if you own more vehicles today. So if we add a "trade-off" to the action of adding power simply, we might get things revert.
This is a Technical FIP, or Filecoin Technical Proposal (FTP).
As talked in the thread: https://filecoinproject.slack.com/archives/C0179RNEMU4/p1600622389349900
many people are suffer from the lotus sync problem.
Background:
Two server running lotus daemon which use same network. One is vary fast in sync. the other one is always slow in sync.
We wish to speedup the slow one using the fast one by add reward/score
to the trusted fast daemon manually.
Similar to:
INFO pubsub [email protected]/gossipsub.go:1499 peer 16Uiu2HAmRH6MMzsPXNRWzQvFyQgV8CVe3e6hsffjNBm5FavWDhJJ
didn’t follow up in 1 IWANT requests; adding penalty
expose the API of peer score will be better.
Fast sync will make the Filecoin network stronger.
Simple Summary
In Filecoin network,miners use worker and PoSt account to signature the messages. And the private keys of worker and post wallet have to be stored on the Lotus-miner server or other signature server for a long time. Once the server was attacked by hackers and the private key was stolen, The miner will suffer great loss of asserts.
Change Motivation
We proposal a different signature method of worker wallet and post wallet by classifying the messages to protect the miner’s asserts.
Specification
In filecoin , there are two main work accounts: worker account and post account.
Worker account: miners use it to submit the PreCommitSector and ProveCommitSector messages when sealing the sectors. During these processes, miners need to provide the pledge of sectors and pay gas fee.(part of gas fee will be payed to the miner which packed the messages and the rest will be burned by transferring to f099). Usually, these transactions are a kind of contract transfer.
Post account: miners use it to submit the SubWindowedPoSt messages. During these processes, During these processes, miners also need to pay gas fee.(part of gas fee will be payed to the miner which packed the messages and the rest will be burned by transferring to f099). Usually, these transactions are a kind of non-contract transfer.
So we can classify the transfer messages as follows.
Working transfer : the transaction behavior of miner in the process of sealing the sectors, submitting the Posts and so on. Including providing the pledge of sectors, paying the gas fee.
Financial transfer : the transactions that are not working transfer. Such as miner A transacts to miner B, miner A transacts to individual account, miner A transacts to f099.
Therefore, we can make a definition by smart contract that if Worker account and Post account are doing some working transferring, which are contract transfer, they just adopt the Single signature. However, if they are doing some financial transferring, which are non-contract transfer, they are suggested to adopt the Multi signature.
By this way, we can effectively protect the miner’s asserts from loss when the private key of worker/post account was stolen.
Chinese Vice Premier called for crackdown on cyrtocurrency mining in 5/24. As follow up instructions to IDCs might come up from the goverment realy soon, there could be risk that IDCs to cutoff infrastructure supplies for hosted filecoin nodes in China in a matter of days, this will be a massive hit on nodes hosted in China and will be a huge hit for filecoin network as well permanetnly. As an prepared emergency remediation for that worst case scenario, we propose an intermediate period for those nodes to migrate to oversea and their power can be allowed to come back afterward.
Define a start height and end height, during start and end height, the ongoing missing window post pernality reduced to far less than 5br, and the 14 days deadline of terminate sectors for missing post suspended, while end height pass, penaltis and sector terminations resume. This will allow miners to have some time (presumerbily much longer than 14 days considering the logistic complexity) to migrate their nodes physically to the locations that mining is allowed for long term.
I'm opening this issue as the official discussion thread for the HAMT improvement FIP I'm currently drafting. There are several popular outstanding breaking changes many of which are already implemented in the go filecoin hamt that would improve the protocol in terms of performance, simplicity and safety. Since each change is small on its own I am bundling them all into one FIP to reduce overhead. However each change can be considered separately and if there is a strong reason to exclude one of the four I plan to do this while the FIP is in draft stage.
SetIfAbsent
which only writes the key if not previously present returning a boolean indicating set/no set.Meta note: bundling changes into one FIP like this is an experiment that I don't believe has been tried before. Feel free to critique the bundling as well as the HAMT issues in this thread if you see problems.
Allow miner to send an upgraded verified deal after the piece data has been sealed and sector has been landed on chain.
Today if client/miner wants to take advantage of Filecoin Plus program, the process is client send DataCap request to notary, with supplemental in formations provided, notary approve the request and assign DataCap to the client, then client start the deal process by sending proposal(s) with VerifiedDeal field marked "true", and miner go through the rest of deal process steps, while being part of Filecoin Plus, miners enjoy the higher quality adjusted power and clients enjoy lower storage price, a de factor binding has been imposed between filecoin plus onboarding process and deal-seal process by making successful onboarding process as the pre-condition before deal-seal
This proposal is aiming to break this de factor dependency without changing or compromising the Filecoin Plus Client Onboarding process, but just wants to introduce an additional deal upgrade message to be allowed to submit to the chain to either 1, make the previous regular deal to be an "verified deal" or 2, To supplement deal information related to a CC sector which sealed piece data with meaningful content
1, Given the benefit of Filecoin Plus to both clients and miners, clients might tend to apply for DataCap before store their data in the chain, and DataCap approval process normally take days even weeks, and it's possible that eventually only partial or even no DataCap is granted, therefore it could be a barrier or a delay factor for clients to land their data in the network. Decouple Filecoin Plus and propose deal will encourage client to store their data in Filecoin first and apply DataCap in parallel or later.
2, Although the network storage power is over 6EiB, majority of it is Committed Capacity with zero data in it. Miners are not intended to use existing "CC upgrade" to make it a sector with deals because of no extra economic motivation, given the "post verification" mechanism in place, expect deals might be increasing sharply and it could be economic efficient to leverage existing committed capacity with zero piece by CC upgrade, therefore the overall piece data size parentage could be increased w.r.t overall network storage power. (For reference, in date of Jun 4, In the total 768062 precommit message, only 2 were "ReplaceCapcity" )
3, For CC sectors with non zero data piece (assume to implement in FIP "Pack arbitrary data in CC sectors and be able to upgrade them to verified data later"), no re-seal will be needed to upgrade the CC sector to a "deal sector"
Two scenario will need to be considered in design:
1, New "Upgrade deal" proposal needs to be defined with verified field to be true, and new "UpggadeStorageDeal" needs to be added to the Market Actor, to link the existing original deal to the sector already committed to the network, and update the quality adjusted power.
2, With another FIP "Pack arbitrary data in CC sectors and be able to upgrade them to verified data later" implemented, a complemented proposal need to be defined to the CC sector, and a "ComplemntStorageDeal" needs to be
added to the Market Actor to link the complement deal to the existed CC sector.
There will be no impact to the existing sectors/deals committed to the chain
Seal process will keep as it is, there will be no chance to bring sectors to the network in faster manner by this proposal.
The DataCap approval process will not be no change (or even be strengthen for notary to retrieve the piece data before grant the DataCap), therefore there will be no shortcut for clients/miners to bring more quality adjusted power nonauthentic way.
1, Encourage client to bring more business data to the network without waiting for DataCap proving process
2, Encourage miner to leverage CC upgrade more often
3, For "CC sectors with data piece", no re-seal needed for CC upgrade.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.