subspace / subspace

Subspace Network reference implementation

Subspace Network Monorepo

This is a mono repository for Subspace Network implementation, primarily containing Subspace node/client using Substrate framework and farmer app implementations.

Repository structure

The structure of this repository is the following:

• crates contains Subspace-specific Rust crates used to build node and farmer, most are following Substrate naming conventions
  • subspace-node is an implementation of the node for Subspace protocol
  • subspace-farmer is a CLI farmer app
• domains contains client and runtime code for decoupled execution and domains
• orml contains a fork of orml vesting pallet with modified dependencies

How to run

Please refer to farming.md on how to run farmer.

If you are looking to farm offline, or build from source for development purposes please refer to development.md.

subspace's Issues

Replace unsigned root block extrinsic with inherent

Otherwise root block inclusion is subject to transaction pool limits and can fail unexpectedly.

CLI commands on farmer to manage keypair

Right now there is no way to see public key or export keypair for re-export into the extension for instance

Use `u32` for the `offset` field in `BlockObject`

No need to use [u8; 3] for the offset, just use u32 for a better readability.

subspace/crates/subspace-core-primitives/src/objects.rs

Line 45 in 6af677b

offset: [u8; 3],

subspace/crates/subspace-runtime/src/lib.rs

Lines 441 to 447 in 6af677b

	let offset = (base_extrinsic_offset + call_object_location.offset).to_le_bytes();
	// Block is known to never exceed 16MiB, hence 24-bit addressing
	block_object_mapping.objects.push(BlockObject::V0 {
	hash: call_object_location.hash,
	offset: [offset[0], offset[1], offset[2]],
	});

Remove BABE and Grandpa from dev dependencies

Those are used for testing, but, ideally, we'd use our own consensus instead, which would reduce number of dependencies and maybe even CI times.

Clean weight data from the aux storage

We should probably follow or at least take inspiration from paritytech/substrate#10748

During Salt Change, the wrong Commitment Database being deleted (occasionally)

An example scenario:

1. 1st challenge [salt: 1, next_salt: 2] is sent to farming
2. farming creates commitments for these two salts
3. farming then calls find_by_range for the salt 1. The least recent one is: 2.
4. 2nd challenge [salt:1, next_salt: 2] is sent to farming
5. farming checks for if these commitments exist (this does not update LRU)
6. farming then calls find_by_range for the salt 1. The least recent one is: 2.
7. 3rd challenge [salt:2, next_salt: 3] is sent to farming
8. since next_salt:3 is a new one, commitment.create() is called
9. least recent one is popped, which is 2.
10. find_by_range cannot find the solution for salt:2, since there is no such database...

Probably modifying find_by_range in a way, that it would not change the order of the items in the LRU cache will suffice for the fix.

Support dynamic eon duration changes

Code right now can't really deal with that

Application gets killed on macOS 12

Looks something like this:

MacBook-Pro:Downloads Soleorax$ ./subspace-node-macos-x86_64-snapshot-2022-mar-09 \

      --chain testnet \
      --wasm-execution compiled \
      --execution wasm \
      --bootnodes "/dns/farm-rpc.subspace.network/tcp/30333/p2p/12D3KooWPjMZuSYj35ehced2MTJFf95upwpHKgKUrFRfHwohzJXr" \
      --rpc-cors all \
      --rpc-methods unsafe \
      --ws-external \
      --validator \
      --telemetry-url "wss://telemetry.polkadot.io/submit/ 1" \
      --name Duckets

2022-03-19 08:14:00 Subspace
2022-03-19 08:14:00 ✌️  version 0.1.0-042302e-x86_64-macos
2022-03-19 08:14:00 ❤️  by Subspace Labs https://subspace.network/, 2021-2022
2022-03-19 08:14:00 📋 Chain specification: Subspace testnet
2022-03-19 08:14:00 🏷  Node name: Duckets
2022-03-19 08:14:00 👤 Role: AUTHORITY
2022-03-19 08:14:00 💾 Database: RocksDb at /Users/Soleorax/Library/Application Support/subspace-node-macos-x86_64-snapshot-2022-mar-09/chains/subspace_test/db/full
2022-03-19 08:14:00 ⛓  Native runtime: subspace-100 (subspace-1.tx1.au1)
2022-03-19 08:14:01 🔨 Initializing Genesis block/state (state: 0x1f74…d19c, header-hash: 0x332e…4a74)
Killed: 9

I'm suspecting that it is only happening on macOS 12 or 12.1.

Syncing on Windows really slow

Syncing on Windows takes about 6 Hours, on mac with the same internet connection this sync takes about 30 min. When starting on windows you get the following output. Which may indicate that only a single thread is being used for the syncing of the node, causing it to take significantly longer than needed. Issue has also been reported by a user aswell.

2022-03-29 22:43:19 Subspace
2022-03-29 22:43:19 ✌️  version 0.1.0-042302e-x86_64-windows-msvc
2022-03-29 22:43:19 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022
2022-03-29 22:43:19 📋 Chain specification: Subspace testnet
2022-03-29 22:43:19 🏷  Node name: Void-1
2022-03-29 22:43:19 👤 Role: AUTHORITY
2022-03-29 22:43:19 💾 Database: RocksDb at C:\Users\Justin\AppData\Local\subspace-node-windows-x86_64-snapshot-2022-mar-09.exe\data\chains\subspace_test\db\full
2022-03-29 22:43:19 ⛓  Native runtime: subspace-100 (subspace-1.tx1.au1)
2022-03-29 22:43:50 Last archived block 265113
2022-03-29 22:43:50 Archiving already produced blocks 265114..=265202
2022-03-29 22:43:50 🏷  Local node identity is: 12D3KooWPJ3JoaHSG3fGBxHnkn5ASWqbUzFHAGdZk2kANp2t6bCA
2022-03-29 22:43:52 🧑‍🌾 Starting Subspace Authorship worker
2022-03-29 22:43:52 📦 Highest known block at #265302
2022-03-29 22:43:52 〽️ Prometheus exporter started at 127.0.0.1:9615
2022-03-29 22:43:52 Multi-threaded server is not available on Windows. Falling back to single thread.
2022-03-29 22:43:52 Listening for new connections on 0.0.0.0:9944.
2022-03-29 22:43:53 🔍 Discovered new external address for our node: /ip4/71.209.182.243/tcp/30333/p2p/12D3KooWPJ3JoaHSG3fGBxHnkn5ASWqbUzFHAGdZk2kANp2t6bCA
2022-03-29 22:43:57 ⚙️  Syncing, target=#336948 (24 peers), best: #265348 (0x12c1…3949), finalized #0 (0x332e…4a74), ⬇ 3.5MiB/s ⬆ 18.8kiB/s
2022-03-29 22:44:02 ⚙️  Syncing 12.9 bps, target=#336949 (25 peers), best: #265413 (0xcf82…8227), finalized #0 (0x332e…4a74), ⬇ 108.2kiB/s ⬆ 22.9kiB/s

Edit: Issue is also being tracked for users on the Forum here: https://community.subspace.network/t/multi-threaded-server-is-not-available-on-windows-falling-back-to-single-thread/46

Unify the concept of commitment and tag for a piece

Ref #85 (comment)

Implement Cost-of-Storage

For some interval (block, epoch, eon ?) re-compute and enforce the cost-of-storage which serves as the base transaction fee (mass) as opposed to the weight (computation cost).

For the size of the history, make sure we're using the post-processed size, not the original size.

• Add notion of cost-of-storage
• Correctly compute the cost-of-storage (CoS) from spec and track in the runtime
• Derive the base tx fee (light client, wallet, etc) from the CoS
• Enforce the CoS when validating txs in the mem-pool and during the block import pipeline
• Do not enforce it right away (think how?)

Pallet ObjectStore

Simple pallet to store data on Subspace without any feeds or other advanced concepts

Simplify build scripts and snapshot action

When we upgrade Substrate to version with paritytech/substrate#10845 included we can remove hacks with .git directory and use an argument environment variable in Dockerfile instead.

Error: Header Rejected, too far in the future

Error Code

💔 Verification failed for block 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d received from peer: 12D3KooWPEynXbQeuGwai76xheGav44cCs3n4cpMpCP94g7my1wT, "Header 0xf514f794c67003ed3ca47f8d3dbb9056b1aefc6de8b1a50a761048f39a5dd09d rejected: too far in the future"

Description

The user contacted me regarding their node crashing due to a power outage, when they reconnected there were no peers connected and their chain was behind the longest known chain.

As time progressed, in some cases the node would catch up; connect to 4 peers and work for about 1 min then fall back behind and drop all the peers again.

Expected Behavior

The node should catch up to the chain, and keep caught up.
furthermore, the node should not lose the peer connection.

Actual Behavior

The node would try to sync with other nodes and catch up for a moment, then fall far behind again.

Attempted Troubleshooting

We rebooted the machine, attempted to reconnect the node, and the issue persisted.
From here we purged the chain and attempted sync from Genysis. initially, it was syncing very effectively, but once it got to the longest chain the issue arose again.

General farmer usage

Steps to Reproduce

*This may be hard to replicate.

1. Run node & farmer as per normal workflow.
2. Unexpectedly power off the machine.
3. Boot machine & attempt to restart node & farmer
4. see if the issue arises.

Your Environment

• Version used: Spartan Testnet
• Operating System and version: Windows 10
• Desktop, Laptop, or Server: Lenovo Legion

Farmer doesn't archive blocks on the first start during sync unless restarted

When node is syncing and farmer is started for the first time, farmer doesn't archive anything beyond the first segment for some reason unless stopped and restarted. This is not an expected behavior and must be fixed.

Fix racy test `test_find_by_tag`

The test is probabilistic and annoyingly it errors sometimes, this must be fixed.

Split and handle storage and compute fees separately

Portion of storage fee should go into escrow, portion should go to the farmer (block producer) right away, portion of the escrow should be added to block reward. Ratio of those should be upgradable.
Compute should go to executors though.

Add Code Signing to Binaries

It would be best to add code signing to the binaries to prevent popups for users.

Investigate RPC buffering

Reproduces easily after increasing WS buffer capacity:

cargo run --release --bin subspace-node -- --chain testnet --validator --tmp --port 30335 --ws-port 1236 --node-key 0000000000000000000000000000000000000000000000000000000000000001 --ws-max-out-buffer-capacity=100

For some reason after storm of relayer transactions it breaks RPC and farmer is no longer able to connect:

[2021-11-04T06:56:03Z INFO  subspace_farmer::commands::farm] Archived segment 74 at block 114
[2021-11-04T06:56:08Z INFO  subspace_farmer::commands::farm] Archived segment 75 at block 118
[2021-11-04T06:56:09Z WARN  soketto::connection] 84a74de9: accumulated message length exceeds maximum
[2021-11-04T06:56:09Z ERROR jsonrpsee_ws_client::client] Error: Connection(MessageTooLarge { current: 10551135, maximum: 10485760 }) terminating client
[2021-11-04T06:56:09Z INFO  subspace_farmer::commands::farm] Background plotting finished successfully
[2021-11-04T06:56:09Z ERROR subspace_farmer::commands::farm] Failed to get block #119 from RPC: The background task been terminated because: Networking or low-level protocol error: WebSocket connection error: 0; restart required
nazar-pc@nazar-pc:/w/s/subspace> cargo run --release --bin subspace-farmer -- farm --custom-path /x/y --ws-server ws://127.0.0.1:1236
    Finished release [optimized] target(s) in 0.21s
     Running `target/release/subspace-farmer farm --custom-path /fast-crap/subspace3 --ws-server 'ws://127.0.0.1:1236'`
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Opening plot
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Opening commitments
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Opening object mapping
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Connecting to RPC server: ws://127.0.0.1:1236
[2021-11-04T07:03:42Z INFO  subspace_farmer::identity] Opening existing keypair
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Subscribing to slot info
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Last archived block 118
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Subscribing to new heads
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Plotting new blocks in the background
[2021-11-04T07:03:42Z WARN  soketto::connection] c55f2cb0: accumulated message length exceeds maximum
[2021-11-04T07:03:42Z ERROR jsonrpsee_ws_client::client] Error: Connection(MessageTooLarge { current: 10551135, maximum: 10485760 }) terminating client
[2021-11-04T07:03:42Z INFO  subspace_farmer::commands::farm] Background plotting finished successfully
[2021-11-04T07:03:42Z ERROR subspace_farmer::commands::farm] Failed to get block #119 from RPC: The background task been terminated because: Networking or low-level protocol error: WebSocket connection error: 0; restart required

It might be that some blocks are getting too large to fit into one RPC message, in which case we'll have to increase capacity on node, farmer and relayer or we might want to instead decrease max block size to make hex encoding always fit default 10M buffer size.

Switch to CPU encoding if GPU encoding fails

Related to #244, we should just fall back to CPU in case GPU encoding fails so we don't break farmer completely.

Block production hang (missing farmer signature)

This happened on testnet:

2022-01-03 09:13:08 💤 Idle (1 peers), best: #173230 (0x2e66…2e5f), finalized #0 (0x1aa7…2a5d), ⬇ 20.3kiB/s ⬆ 34.3kiB/s    
2022-01-03 09:13:11 🙌 Starting consensus session on top of parent 0x2e663b53bdc4c8a8c82fc6f6f12f7990cd67d73df0567dcca0903b07d3e32e5f    
2022-01-03 09:13:11 🎁 Prepared block for proposing at 173231 (10 ms) [hash: 0x460a77d4fcaf137119ef20c065002d4d9a8e0827a6c4e0ed3b8aa07d43a110aa; parent_hash: 0x2e66…2e5f; extrinsics (15): [0x0ec4…2c3f, 0xcfd0…15f5, 0x87b2…c087, 0x53de…f584, 0x4ab6…9340, 0xa72f…3d64, 0x644e…c39f, 0x8f54…24ed, 0x712f…2c24, 0xdd7e…8fa7, 0x4625…2e72, 0xadbb…5719, 0x01e5…2dd5, 0xe9d6…8a38, 0x7c81…066d]]    
2022-01-03 09:13:12 🔖 Pre-sealed block for proposal at 173231. Hash now 0xe154b49762081974ec224149e2ae121cee4e41f08826a6f2fe19d91518e1b013, previously 0x460a77d4fcaf137119ef20c065002d4d9a8e0827a6c4e0ed3b8aa07d43a110aa.    
2022-01-03 09:13:12 ✨ Imported #173231 (0xe154…b013)    
2022-01-03 09:13:13 💤 Idle (1 peers), best: #173231 (0xe154…b013), finalized #0 (0x1aa7…2a5d), ⬇ 15.9kiB/s ⬆ 33.6kiB/s    
2022-01-03 09:13:13 🙌 Starting consensus session on top of parent 0xe154b49762081974ec224149e2ae121cee4e41f08826a6f2fe19d91518e1b013    
2022-01-03 09:13:13 🎁 Prepared block for proposing at 173232 (5 ms) [hash: 0x18c0e70d066b6e9b19a2b34572fba77b47e979ac1a2d51079b7a791733fd9187; parent_hash: 0xe154…b013; extrinsics (5): [0x836c…6d31, 0x3cf3…5100, 0x12bb…b669, 0x1664…25a3, 0xc6f8…23e2]]    
2022-01-03 09:13:14 🔖 Pre-sealed block for proposal at 173232. Hash now 0x1a154a880d4f6c43f83f3afaad63a108e79d4db8bfb911b684aa12d01c3f16b2, previously 0x18c0e70d066b6e9b19a2b34572fba77b47e979ac1a2d51079b7a791733fd9187.    
2022-01-03 09:13:14 ✨ Imported #173232 (0x1a15…16b2)    
2022-01-03 09:13:18 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 13.6kiB/s ⬆ 6.0kiB/s    
2022-01-03 09:13:23 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 10.2kiB/s ⬆ 6 B/s    
2022-01-03 09:13:28 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 4.2kiB/s ⬆ 0    
2022-01-03 09:13:30 🙌 Starting consensus session on top of parent 0x1a154a880d4f6c43f83f3afaad63a108e79d4db8bfb911b684aa12d01c3f16b2    
2022-01-03 09:13:30 🎁 Prepared block for proposing at 173233 (10 ms) [hash: 0xf797621d05f02e9643e348104b32db334e321758630f5ac877c2ffd21164d5f4; parent_hash: 0x1a15…16b2; extrinsics (20): [0xb695…9128, 0xbf93…815b, 0x5b39…c50c, 0xa26d…04f1, 0xfa26…a307, 0x477f…c5b6, 0xdc04…021c, 0xd33c…7b41, 0xe494…4883, 0xcdff…84bc, 0x0eaa…b89b, 0x73ea…27f5, 0xc4ea…710d, 0x358a…63b9, 0x246c…e326, 0x27e7…2542, 0xb074…682b, 0xdf0d…3f5e, 0x5514…3e98, 0xec8c…a7bd]]    
2022-01-03 09:13:30 WS Error <Capacity>: Reached the limit of the output buffer for the connection.    
2022-01-03 09:13:33 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 24 B/s ⬆ 24 B/s    
2022-01-03 09:13:35 Failed to create block import params: CannotSign([238, 66, 253, 208, 52, 74, 0, 120, 176, 129, 155, 117, 173, 212, 130, 46, 225, 108, 80, 150, 173, 214, 109, 10, 19, 12, 79, 54, 209, 241, 186, 105], "Farmer didn't sign header")    
2022-01-03 09:13:38 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 3.5kiB/s ⬆ 0    
2022-01-03 09:13:43 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 0 ⬆ 0    
2022-01-03 09:13:48 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 24 B/s ⬆ 24 B/s    
2022-01-03 09:13:53 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 0 ⬆ 0    
2022-01-03 09:13:58 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 0 ⬆ 0    
2022-01-03 09:14:03 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 24 B/s ⬆ 24 B/s    
2022-01-03 09:14:08 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 33 B/s ⬆ 26 B/s    
2022-01-03 09:14:13 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 0 ⬆ 0    
2022-01-03 09:14:18 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 24 B/s ⬆ 30 B/s    
2022-01-03 09:14:23 💤 Idle (1 peers), best: #173232 (0x1a15…16b2), finalized #0 (0x1aa7…2a5d), ⬇ 0 ⬆ 0    

And on farmer side nothing interesting:

[2022-01-03T09:12:50Z INFO  subspace_farmer::farming] Successfully signed block 0x228c9ef478888d3770264121a934eeea86e7e2eee52649a4324cca609e214a28 and sent signature to node
[2022-01-03T09:12:50Z INFO  subspace_farmer::plotting] Archived segment 287785 at block 173129
[2022-01-03T09:13:06Z INFO  subspace_farmer::farming] Successfully signed block 0x3bd2e661fb93c38e10e3c2360489c6e8941511bcecaf3e865f8c782331cd7397 and sent signature to node
[2022-01-03T09:13:12Z INFO  subspace_farmer::farming] Successfully signed block 0x460a77d4fcaf137119ef20c065002d4d9a8e0827a6c4e0ed3b8aa07d43a110aa and sent signature to node
[2022-01-03T09:13:14Z INFO  subspace_farmer::farming] Successfully signed block 0x18c0e70d066b6e9b19a2b34572fba77b47e979ac1a2d51079b7a791733fd9187 and sent signature to node

So by the looks of it:

• either something happened on farmer side, it failed to send response with signature and stopped accepting new messages over subscriptions (which caused "Reached the limit of the output buffer for the connection" on the node side)
• or "Reached the limit of the output buffer for the connection" happened on the node side first and that in turn broke subscriptions

Interestingly, WS connection was still alive and once node stopped farmer exited too.

Make object mapping support batched extrinsics

Currently we only support standalone put calls, but need to also support when wrapped with calls from pallet utility

Cirrus inherents inclusion signal is flaky

The final list of extrinsics derived from the bundles still can include multiple same kind of inherents somehow, I haven't found a stable way to reproduce this, it occurs randomly in the local development.

2022-01-22 15:17:03.011 ERROR tokio-runtime-worker runtime: [Secondarychain] panicked at 'Timestamp must be updated only once in the block', /home/xlc/.cargo/git/checkouts/substrate-7bc20b373ca3e834/6b3ec3b/frame/timestamp/src/lib.rs:204:13
2022-01-22 15:17:03.011 DEBUG tokio-runtime-worker sync: [Secondarychain] Propagating transactions
2022-01-22 15:17:03.011 ERROR tokio-runtime-worker cirrus::executor: [Secondarychain] Error at processing bundles. relay_parent=0x010d5c76d85979efcfbfcf9198bc8ba537a797e65f2bde1d5400ec310ca78975 error=RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x1683 - <unknown>!rust_begin_unwind\n    1: 0x1443 - <unknown>!core::panicking::panic_fmt::h8f11323637b4db3b\n    2: 0x2f6d0 - <unknown>!<parachain_template_runtime::Call as sp_runtime::traits::Dispatchable>::dispatch::h1798db4b84d44dc7\n    3: 0x29827 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::hdeb02be4f0f6bbb5\n    4: 0x3fbd9 - <unknown>!BlockBuilder_apply_extrinsic\n"))))

subspace/cumulus/client/cirrus-executor/src/lib.rs

Lines 297 to 309 in c3dbabe

	// Check if we should include the inherents in this bundle.
	//
	// We don't include the inherent extrinsics in each bundle because we use
	// the block builder to import the block that contains all the extrinsics
	// converted from a batch of bundles later. That being said, if we include
	// the inherents per bundle, the final extrinsics fed into the block builder
	// will contain multiple same kind of inherent extrinsic, e.g, timestamp,
	// the block builder will panic when importing because the timestamp call
	// should be executed once and only once per block.
	let include_inherents = self
	.to_include_inherents
	.compare_exchange(true, false, Ordering::SeqCst, Ordering::Relaxed)
	.unwrap_or(false);

subspace/cumulus/client/cirrus-executor/src/lib.rs

Lines 320 to 325 in c3dbabe

	match self.process_bundles_impl(primary_hash, bundles, shuffling_seed).await {
	Ok(res) => {
	// Reset the signal to include the inherent extrinsics on next `produce_bundle`.
	self.to_include_inherents.store(true, Ordering::Relaxed);
	res
	},

Succinct Proofs of Replication

Remove encodings & witness from block header.

Depends on DSN such that we can retrieve arbitrary pieces from the farmer network.

Chain selection (fork choice rule) should heaviest chain not longest chain

Node stuck and unable to sync

Serge hit this on farmnet with M1 Macbook (should be irrelevant, but still).

Got stuck with these:

2022-02-14 14:23:46 💤 Idle (0 peers), best: #366410 (0x6c13…d667), finalized #0 (0x6ada…0d38), ⬇ 1.0kiB/s ⬆ 1.1kiB/s
2022-02-14 14:23:51 💤 Idle (0 peers), best: #366410 (0x6c13…d667), finalized #0 (0x6ada…0d38), ⬇ 0.5kiB/s ⬆ 0.4kiB/s
2022-02-14 14:23:56 💤 Idle (0 peers), best: #366410 (0x6c13…d667), finalized #0 (0x6ada…0d38), ⬇ 2.0kiB/s ⬆ 0.8kiB/s
2022-02-14 14:24:01 💤 Idle (0 peers), best: #366410 (0x6c13…d667), finalized #0 (0x6ada…0d38), ⬇ 0.7kiB/s ⬆ 0.7kiB/s

After restart:

2022-02-14 14:51:31 Subspace
2022-02-14 14:51:31 ✌️  version 0.1.0-35cf6f5-x86_64-macos
2022-02-14 14:51:31 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022
2022-02-14 14:51:31 📋 Chain specification: Subspace testnet
2022-02-14 14:51:31 🏷  Node name: undefined-is-not-a-function
2022-02-14 14:51:31 👤 Role: AUTHORITY
2022-02-14 14:51:31 💾 Database: RocksDb at /Users/apple/Library/Application Support/subspace-node-x86_64-macos-11-snapshot-2022-jan-05/chains/subspace_test/db/full
2022-02-14 14:51:31 ⛓  Native runtime: subspace-100 (subspace-1.tx1.au1)
2022-02-14 14:51:33 Last archived block 366201
2022-02-14 14:51:33 Archiving already produced blocks 366202..=366310
2022-02-14 14:51:34 🏷  Local node identity is: 12D3KooWJBPe9vHbvLzB4roWxLwHgh1rxvgnFFGpidNAgHUjiQuN
2022-02-14 14:51:44 🧑‍🌾 Starting Subspace Authorship worker
2022-02-14 14:51:44 📦 Highest known block at #366410
2022-02-14 14:51:44 〽️ Prometheus exporter started at 127.0.0.1:9615
2022-02-14 14:51:44 Listening for new connections on 0.0.0.0:9944.
2022-02-14 14:51:45 🔍 Discovered new external address for our node: /ip4/91.196.192.130/tcp/30333/p2p/12D3KooWJBPe9vHbvLzB4roWxLwHgh1rxvgnFFGpidNAgHUjiQuN
2022-02-14 14:51:45 Accepted a new tcp connection from 127.0.0.1:51646.
2022-02-14 14:51:45 ♻️  Reorg on #366410,0x6c13…d667 to #366410,0x3c51…9da2, common ancestor #366409,0x0884…2080
2022-02-14 14:51:45 ✨ Imported #366410 (0x3c51…9da2)
2022-02-14 14:51:47 Accepted a new tcp connection from 127.0.0.1:51665
2022-02-14 14:51:49 ⚙️  Syncing, target=#377924 (6 peers), best: #366457 (0xa881…cc53), finalized #0 (0x6ada…0d38), ⬇ 2.3MiB/s ⬆ 246.6kiB/s
2022-02-14 14:51:54 ⚙️  Syncing 11.1 bps, target=#377925 (6 peers), best: #366513 (0xc718…90e6), finalized #0 (0x6ada…0d38), ⬇ 6.8kiB/s ⬆ 0.2kiB/s
2022-02-14 14:51:55 💔 Error importing block 0x4d8010e5a1fce53e5b134e21070f8c6bcd53aa7034de2aee22b2cee03638e6e8: Err(Other(ClientImport("Stored root block extrinsics were not found: [V0 { segment_index: 3390, records_root: [134, 163, 6, 26, 192, 255, 51, 26, 172, 57, 231, 97, 73, 251, 122, 39, 252, 63, 229, 79, 222, 41, 173, 67, 33, 204, 17, 70, 1, 141, 192, 96], prev_root_block_hash: [208, 163, 46, 46, 110, 5, 8, 13, 60, 244, 85, 169, 170, 84, 217, 21, 72, 156, 223, 27, 127, 209, 11, 241, 192, 119, 202, 21, 193, 150, 25, 226], last_archived_block: LastArchivedBlock { number: 366418, archived_progress: Partial(405) } }]")))
2022-02-14 14:51:55 💔 Error importing block 0xabcd7bdcddeb0fd1d4450c60fe2767ed67c2270c197949d1b37cf7b72108fb16: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0xba1a278066fa5d17e2ea3adc1af1e7f8ed148e83f6d79269d1bf9491f4b9cb11: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0xd43f320decdaa99cde125c0b2137f34b194938d9cb876ffcc71041a1b58bf80d: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0x774be9b0814fcdfed013666c39e949acb8cc26218c6875b7c223a5484fbf90d6: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0x79e04ab2016c46e4aaf2b7b9ab3da84b8793e7788fe8ea39eb8c90988c6eed7b: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0xaf931428ae0ccc5753af935bb5c72ea8b982a85d44731e1f3193e5abae3e5f67: Err(UnknownParent)
2022-02-14 14:51:55 💔 Error importing block 0x4d8010e5a1fce53e5b134e21070f8c6bcd53aa7034de2aee22b2cee03638e6e8: Err(Other(ClientImport("Stored root block extrinsics were not found: [V0 { segment_index: 3390, records_root: [134, 163, 6, 26, 192, 255, 51, 26, 172, 57, 231, 97, 73, 251, 122, 39, 252, 63, 229, 79, 222, 41, 173, 67, 33, 204, 17, 70, 1, 141, 192, 96], prev_root_block_hash: [208, 163, 46, 46, 110, 5, 8, 13, 60, 244, 85, 169, 170, 84, 217, 21, 72, 156, 223, 27, 127, 209, 11, 241, 192, 119, 202, 21, 193, 150, 25, 226], last_archived_block: LastArchivedBlock { number: 366418, archived_progress: Partial(405) } }]")))
2022-02-14 14:51:55 💔 Error importing block 0x4d8010e5a1fce53e5b134e21070f8c6bcd53aa7034de2aee22b2cee03638e6e8: Err(Other(ClientImport("Stored root block extrinsics were not found: [V0 { segment_index: 3390, records_root: [134, 163, 6, 26, 192, 255, 51, 26, 172, 57, 231, 97, 73, 251, 122, 39, 252, 63, 229, 79, 222, 41, 173, 67, 33, 204, 17, 70, 1, 141, 192, 96], prev_root_block_hash: [208, 163, 46, 46, 110, 5, 8, 13, 60, 244, 85, 169, 170, 84, 217, 21, 72, 156, 223, 27, 127, 209, 11, 241, 192, 119, 202, 21, 193, 150, 25, 226], last_archived_block: LastArchivedBlock { number: 366418, archived_progress: Partial(405) } }]")))
2022-02-14 14:51:59 💤 Idle (0 peers), best: #366517 (0xed9b…8096), finalized #0 (0x6ada…0d38), ⬇ 874.6kiB/s ⬆ 1.6kiB/s

On the farmer side last lines before it got stuck:

[2022-02-13T17:13:15Z INFO  subspace_farmer::farming] Successfully signed block 0xdfdc628555c9fcec3a646053d58f1f09d26e5047dedca3187f38c1b70c52563b and sent signature to node
[2022-02-13T17:13:27Z INFO  subspace_farmer::farming] Successfully signed block 0x344ec7d97c81696d51c7ee3f0a682b9e8066c1624d52666876aa10143726ba18 and sent signature to node
[2022-02-13T17:13:40Z INFO  subspace_farmer::farming] Successfully signed block 0x33f261b92ea5a1795e32d35956097cf5559b308d3103f40124cf1585eb1df9f0 and sent signature to node
[2022-02-13T17:13:59Z INFO  subspace_farmer::farming] Successfully signed block 0x386af47216ee26bbfd718893f1dbc0735dcfbbf57f8cca01c17cdbf645d6af8e and sent signature to node
[2022-02-13T17:13:59Z INFO  subspace_farmer::plotting] Archived segment 3389 at block 366309

There must be some logic error in the code that causes this.

Build and publish container images and executables on releases

Fix reed-solomon compilation on M1

Disable simd-accel, see rust-rse/reed-solomon-erasure#86

Refine the rust documentation

I suggest we go through all the existing crates to see if there is some room to polish or add more descriptive docs, potentially with some small renamings/refactorings.

• pallet-feeds(#199)
• pallet-offences-subspace(#98)
• pallet-subspace(#118)
• sc-consensus-subspace
  • #99
  • #158
• sc-consensus-subspace-rpc(#115)
• sp-consensus-subspace(#97)
• subspace-archiving
• subspace-core-primitives(#85)
• subspace-farmer
• subspace-node(#89)
• subspace-runtime(#88)
• subspace-solving(#89)

Upgrade test runtime environment to support transactions

Current test environment provided by forks of substrate-test-runtime and substrate-test-runtime-client doesn't support transactions and makes it harder to maintain quality of the software. We need to fix that and and unlock corresponding tests.

Don't try to solve unless node is fully synced

Investigate block production stall for the duration of plot recommitment

For ~116G plot recommitment took several minutes:

[2021-11-18T22:09:28Z INFO  subspace_farmer::farming] Salt will update to 0c00000000000000 soon, recommitting in background
[2021-11-18T22:22:14Z INFO  subspace_farmer::farming] Finished recommitment in background for 0c00000000000000 in 766.2087 seconds

And coincidentally block production stalled for about the same time:

2021-11-18 22:09:27 🙌 Starting consensus session on top of parent 0x3a9921a6129501004c55bfb2289c8f144755a7a6f12e7d8f7c91c33b98209adb    
2021-11-18 22:09:27 🎁 Prepared block for proposing at 54456 [hash: 0xed3a65676e7103a74c77f9c27546c59af2d494ee6034172cdbe8f8c12a730068; parent_hash: 0x3a99…9adb; extrinsics (3): [0xaa8d…d010, 0x8b00…bc64, 0xdc0e…f635]]    
2021-11-18 22:09:27 🔖 Pre-sealed block for proposal at 54456. Hash now 0x1cedc221b843158bea21e5ee3ef68898b17b19e4a3f7f67845160bed1ab1dd4f, previously 0xed3a65676e7103a74c77f9c27546c59af2d494ee6034172cdbe8f8c12a730068.    
2021-11-18 22:09:27 ✨ Imported #54456 (0x1ced…dd4f)    
2021-11-18 22:09:28 💤 Idle (1 peers), best: #54456 (0x1ced…dd4f), finalized #0 (0xce2f…a5eb), ⬇ 685.7kiB/s ⬆ 687.5kiB/s    
...
2021-11-18 22:22:23 💤 Idle (1 peers), best: #54456 (0x1ced…dd4f), finalized #0 (0xce2f…a5eb), ⬇ 24 B/s ⬆ 24 B/s    
2021-11-18 22:22:27 🙌 Starting consensus session on top of parent 0x1cedc221b843158bea21e5ee3ef68898b17b19e4a3f7f67845160bed1ab1dd4f    
2021-11-18 22:22:27 🎁 Prepared block for proposing at 54457 [hash: 0x3d3f28901b730f430733500720ccb1ab725d4357f6e5777d1c65e1689fbe9f6c; parent_hash: 0x1ced…dd4f; extrinsics (3): [0x8674…64b0, 0xe6f9…b16d, 0x770c…351a]]    
2021-11-18 22:22:27 🔖 Pre-sealed block for proposal at 54457. Hash now 0xdb8b025129fae6b31744d05f0635f17636bcf99ac42e4f05b9d96f6d6138838e, previously 0x3d3f28901b730f430733500720ccb1ab725d4357f6e5777d1c65e1689fbe9f6c.    
2021-11-18 22:22:27 ✨ Imported #54457 (0xdb8b…838e)    

This is a problem and shouldn't happen. We can reasonably expect that recommitments will affect I/O, but it should not stall block production completely.

Broken block production

I suspect #278 broke block production, now I get this all the time:

2022-03-14 13:54:20 Invalid solution received for slot 1647258860: InvalidTag(Slot(1647258860))    

Derive salt from on-chain randomness

Unexpected Epoch Change on Windows binary

Testing the snapshot binaries, I have a Local bootstrap node running on a desktop and am testing each distribution connecting to the same bootstrap.

On Ubuntu I had no issues, converting to windows it runs but we are getting an Unexpected Epoch Change error.

no relayer running on this bootstrap node during testing

Command Used

./subspace-node.exe --chain ./chain-spec-raw.json --base-path /var/subspace --name subspace-node --pruning archive --port 30333 --rpc-port 9933 --ws-port 9944 --rpc-cors all --ws-external --rpc-methods unsafe --wasm-execution compiled --execution wasm --validator --telemetry-url "wss://telemetry.polkadot.io/submit/ 9" --bootnodes "/dns/192.168.0.30/tcp/30333/p2p/12D3KooWF8A9Cpmc5ErJreBSbvnh1iur53ynA8ShrG8zP8kjaXnq"

Output of Command

←[2m2021-12-08 00:41:52←[0m 💤 Idle (1 peers), best: #775 (0x1bbb…961b), finalized #0 (0x5b27…ed7e), ⬇ 1.9kiB/s ⬆ 2.8kiB/s
←[2m2021-12-08 00:41:53←[0m 🙌 Starting consensus session on top of parent 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b
←[2m2021-12-08 00:41:53←[0m 🎁 Prepared block for proposing at 776 (13 ms) [hash: 0x65ad437a275eba478ad96d5e9f20b694d1897c2453ffe3d808bba16546085550; parent_hash: 0x1bbb…961b; extrinsics (1): [0x0939…0c09]]
←[2m2021-12-08 00:41:53←[0m 🔖 Pre-sealed block for proposal at 776. Hash now 0xc0219427147d76ba94bd62c04ddedadc0606d56eae334a5c0a5b01c31768e980, previously 0x65ad437a275eba478ad96d5e9f20b694d1897c2453ffe3d808bba16546085550.
←[2m2021-12-08 00:41:53←[0m Error with block built on 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b: ClientImport("Unexpected epoch change")
←[2m2021-12-08 00:41:54←[0m 🙌 Starting consensus session on top of parent 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b
←[2m2021-12-08 00:41:54←[0m 🎁 Prepared block for proposing at 776 (15 ms) [hash: 0xc81ce1c6c1e94ffc95281bfdf47cc30f2cc9deedfdbfe850590214bb31a9dd45; parent_hash: 0x1bbb…961b; extrinsics (1): [0x461b…c2a5]]
←[2m2021-12-08 00:41:54←[0m 🔖 Pre-sealed block for proposal at 776. Hash now 0x35bc25671bf303cbb4fdf061deea64c2eb8d5d59dfbe39563b6510c983b28215, previously 0xc81ce1c6c1e94ffc95281bfdf47cc30f2cc9deedfdbfe850590214bb31a9dd45.
←[2m2021-12-08 00:41:54←[0m Error with block built on 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b: ClientImport("Unexpected epoch change")
←[2m2021-12-08 00:41:55←[0m 🙌 Starting consensus session on top of parent 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b
←[2m2021-12-08 00:41:55←[0m 🎁 Prepared block for proposing at 776 (13 ms) [hash: 0xa0f4b8ba087aaf04a61708d0adba3eca9c2a470f68a4db17cce63bb1a898d9e0; parent_hash: 0x1bbb…961b; extrinsics (1): [0x764d…7377]]
←[2m2021-12-08 00:41:55←[0m 🔖 Pre-sealed block for proposal at 776. Hash now 0x845285827671891821fff229e555fbd29b3d8e37b250a3a2077e72a3478073a4, previously 0xa0f4b8ba087aaf04a61708d0adba3eca9c2a470f68a4db17cce63bb1a898d9e0.
←[2m2021-12-08 00:41:55←[0m Error with block built on 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b: ClientImport("Unexpected epoch change")
←[2m2021-12-08 00:41:57←[0m 🙌 Starting consensus session on top of parent 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b
←[2m2021-12-08 00:41:57←[0m 🎁 Prepared block for proposing at 776 (14 ms) [hash: 0x3c54ec2c6ea0738c9e7cfb2264d8386c8587fc3fba99157cdbe960bd8b6386f7; parent_hash: 0x1bbb…961b; extrinsics (1): [0xccf7…09b9]]
←[2m2021-12-08 00:41:57←[0m 🔖 Pre-sealed block for proposal at 776. Hash now 0x92135e497faeecc307437c0d2187dc1566a8b237322f0a9bd078604072095257, previously 0x3c54ec2c6ea0738c9e7cfb2264d8386c8587fc3fba99157cdbe960bd8b6386f7.
←[2m2021-12-08 00:41:57←[0m Error with block built on 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b: ClientImport("Unexpected epoch change")
←[2m2021-12-08 00:41:57←[0m 💤 Idle (1 peers), best: #775 (0x1bbb…961b), finalized #0 (0x5b27…ed7e), ⬇ 0.3kiB/s ⬆ 117.5kiB/s
←[2m2021-12-08 00:41:58←[0m 🙌 Starting consensus session on top of parent 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b
←[2m2021-12-08 00:41:58←[0m 🎁 Prepared block for proposing at 776 (12 ms) [hash: 0xe382fcbb8b63cbbdd61a4b3526c6cf6ce73b91e5b9d7fe93859897c933c303d9; parent_hash: 0x1bbb…961b; extrinsics (1): [0x1c33…19b4]]
←[2m2021-12-08 00:41:58←[0m 🔖 Pre-sealed block for proposal at 776. Hash now 0x003af92a1be46492d9003d25223ed06da9dfd74c2e6f9c1dc13bcd6fa88bb738, previously 0xe382fcbb8b63cbbdd61a4b3526c6cf6ce73b91e5b9d7fe93859897c933c303d9.
←[2m2021-12-08 00:41:58←[0m Error with block built on 0x1bbb7c9e0901b24af103da861905a1953c6b9c6868d58b578b8f3537744d961b: ClientImport("Unexpected epoch change")

Farmer Output

PS C:\Users\Justin\Downloads> .\subspace-farmer.exe farm
[2021-12-08T07:40:22Z INFO  subspace_farmer::commands::farm] Opening plot
[2021-12-08T07:40:22Z INFO  subspace_farmer::commands::farm] Opening commitments
[2021-12-08T07:40:22Z INFO  subspace_farmer::commands::farm] Opening object mapping
[2021-12-08T07:40:22Z INFO  subspace_farmer::commands::farm] Connecting to node at ws://127.0.0.1:9944
[2021-12-08T07:40:22Z INFO  jsonrpsee_ws_client::transport] Connection established to target: Target { sockaddrs: [], host: "127.0.0.1", host_header: "127.0.0.1:9944", mode: Plain, path_and_query: "/" }
[2021-12-08T07:40:22Z INFO  subspace_farmer::identity] Generating new keypair
[2021-12-08T07:40:22Z INFO  subspace_farmer::commands::farm] WS RPC server listening on 127.0.0.1:9955
[2021-12-08T07:40:22Z INFO  subspace_farmer::farming] Subscribing to slot info
[2021-12-08T07:40:22Z INFO  subspace_farmer::plotting] Plotting new blocks in the background
[2021-12-08T07:40:22Z INFO  subspace_farmer::plotting] Subscribing to new heads
[2021-12-08T07:40:23Z INFO  subspace_farmer::farming] Salt updated to 0000000000000000, recommitting in background
[2021-12-08T07:40:23Z INFO  subspace_farmer::farming] Salt will update to 0100000000000000 soon, recommitting in background
[2021-12-08T07:40:23Z INFO  subspace_farmer::farming] Finished recommitment for 0000000000000000 in 0.080552496 seconds
[2021-12-08T07:40:23Z INFO  subspace_farmer::farming] Finished recommitment in background for 0100000000000000 in 0.081318304 seconds
[2021-12-08T07:40:23Z INFO  subspace_farmer::plotting] Archived segment 0 at block 0
[2021-12-08T07:41:35Z INFO  subspace_farmer::plotting] Archived segment 1 at block 110
[2021-12-08T07:41:39Z INFO  subspace_farmer::plotting] Archived segment 2 at block 219
[2021-12-08T07:41:42Z INFO  subspace_farmer::plotting] Archived segment 3 at block 328
[2021-12-08T07:41:44Z INFO  subspace_farmer::plotting] Archived segment 4 at block 437
[2021-12-08T07:41:47Z INFO  subspace_farmer::plotting] Archived segment 5 at block 546
[2021-12-08T07:41:50Z INFO  subspace_farmer::farming] Salt will update to 0200000000000000 soon, recommitting in background
[2021-12-08T07:41:50Z INFO  subspace_farmer::farming] Finished recommitment in background for 0200000000000000 in 0.0869153 seconds
[2021-12-08T07:41:50Z INFO  subspace_farmer::plotting] Archived segment 6 at block 655

Support epoch transitions with no blocks

It should be possible to jump over empty epochs, currently fails with "Unexpected epoch change"

Partial replica support

Farmer should support limiting plot size to specific size (or number of pieces), re-plot new pieces that are closer to farmer's node ID as history is archived by the network and support all current features (for the pieces that farmer actually stores).

Feed ownership

pallet-feeds right now doens't remember who created feed and allows anyone to push data to any feed.

We need to:

• track who created feed
• limit puts to feed owner
• support getting feeds associated with owner
• support transfers of feed ownership

Refactor the duplicated RPC types

subspace/crates/sc-consensus-subspace-rpc/src/lib.rs

Line 52 in d7de2cc

// TODO: Move RPC types into separate crate shared with farmer

Windows 10 Binary does not run on Windows 11

Attempted to help a user get running last night, they were on Windows 11 and were unable to run the Windows 10 binary. I do not have Windows 11 yet so I have been unable to test with this OS. There may be an option to run in "Compatibility Mode" on windows, but as I mentioned I am unable to verify this.

The best course of action would just be to build for Window 11 if possible.

Minimize predictability of eon updates

Eon salt shouldn't be known for longer than X in advance, right now it is known for 1 eons in advance effectively.

node takes too much time to start listening to the farmer

On my machine (iMac M1 8GB Ram), when I try to start a node from scratch, it takes around 15-20 sec for the node to open the port for listening to the farmer.

Is this the expected behaviour?

Sign blocks on the farmer side

Stop sending secret key to the node, it was a temporary workaround we need to get rid of

Merkle Root segment index X not found (slot Y)

Under heavy storm of transactions these messages may occur:

2021-11-04 08:53:13 ✨ Imported #113 (0xd7d6…2b8e)
2021-11-04 08:53:15 🙌 Starting consensus session on top of parent 0xd7d6ed92e055cce6613f54d47fb8771943cc34ed7407d59069b9b379e9f62b8e
2021-11-04 08:53:15 🎁 Prepared block for proposing at 114 [hash: 0x178bac4ccbf31fbdef301505e5eb4f110531a53325d11da0e60d11359dda40c2; parent_hash: 0xd7d6…2b8e; extrinsics (432): [0x7a48…124e, ......, 0xd64f…b5ad]]
2021-11-04 08:53:15 🔖 Pre-sealed block for proposal at 114. Hash now 0x74427d87e981d0943fbfba91576187cffdb119fe9231ecbd9045303c2674332b, previously 0x178bac4ccbf31fbdef301505e5eb4f110531a53325d11da0e60d11359dda40c2.
2021-11-04 08:53:15 ✨ Imported #114 (0x7442…332b)
2021-11-04 08:53:15 📦 Submitted Subspace root block with segment index 10.
2021-11-04 08:53:15 💤 Idle (0 peers), best: #114 (0x7442…332b), finalized #0 (0x1e0d…5700), ⬇ 0 ⬆ 0
2021-11-04 08:53:17 Merkle Root segment index 10 not found (slot 1636008797)
2021-11-04 08:53:18 🙌 Starting consensus session on top of parent 0x74427d87e981d0943fbfba91576187cffdb119fe9231ecbd9045303c2674332b
2021-11-04 08:53:18 🎁 Prepared block for proposing at 115 [hash: 0x4474e8f9fe7fd12c7bafb08f6aef00b8d3bfeef16c62e74506be04ddaf3c5915; parent_hash: 0x7442…332b; extrinsics (2): [0xae38…31c2, 0x4a7c…7e39]]
2021-11-04 08:53:18 🔖 Pre-sealed block for proposal at 115. Hash now 0x3703d388bdc9dd02b36e554147410a17cbdc4c7b89dfc057c7f00d5d640131bf, previously 0x4474e8f9fe7fd12c7bafb08f6aef00b8d3bfeef16c62e74506be04ddaf3c5915.
2021-11-04 08:53:18 ✨ Imported #115 (0x3703…31bf)
2021-11-04 08:53:20 💤 Idle (0 peers), best: #115 (0x3703…31bf), finalized #0 (0x1e0d…5700), ⬇ 0 ⬆ 0
2021-11-04 08:53:25 🙌 Starting consensus session on top of parent 0x3703d388bdc9dd02b36e554147410a17cbdc4c7b89dfc057c7f00d5d640131bf
2021-11-04 08:53:25 🎁 Prepared block for proposing at 116 [hash: 0xe1a5023100fc3335554aa937fb354cefee7d834c85d9d42b1afd5f7a178f57e7; parent_hash: 0x3703…31bf; extrinsics (103): [0xb61b…f262, ....., 0xbeb6…1737]]
2021-11-04 08:53:25 🔖 Pre-sealed block for proposal at 116. Hash now 0x7a281c18c6d80ecbe9e3b36655834edd8ef364a61cc2fdfa2f0bac90b59d79d0, previously 0xe1a5023100fc3335554aa937fb354cefee7d834c85d9d42b1afd5f7a178f57e7.
2021-11-04 08:53:25 ✨ Imported #116 (0x7a28…79d0)
2021-11-04 08:53:25 💤 Idle (0 peers), best: #116 (0x7a28…79d0), finalized #0 (0x1e0d…5700), ⬇ 0 ⬆ 0
2021-11-04 08:53:26 🙌 Starting consensus session on top of parent 0x7a281c18c6d80ecbe9e3b36655834edd8ef364a61cc2fdfa2f0bac90b59d79d0
2021-11-04 08:53:26 🎁 Prepared block for proposing at 117 [hash: 0x0ad0153a8562fddea0eeb34a863b76b35722658a855b63d613a8a7cebead27d5; parent_hash: 0x7a28…79d0; extrinsics (355): [0x89a2…c685, ....., 0x2b84…d67c]]
2021-11-04 08:53:26 🔖 Pre-sealed block for proposal at 117. Hash now 0xbf6410e257071211d9deba4cbf177f1cc853e0d992db8f571af8368b41bb039f, previously 0x0ad0153a8562fddea0eeb34a863b76b35722658a855b63d613a8a7cebead27d5.
2021-11-04 08:53:26 ✨ Imported #117 (0xbf64…039f)
2021-11-04 08:53:27 🙌 Starting consensus session on top of parent 0xbf6410e257071211d9deba4cbf177f1cc853e0d992db8f571af8368b41bb039f
2021-11-04 08:53:27 🎁 Prepared block for proposing at 118 [hash: 0x13c994d7163d101e97e1bd7b31df0d2ee55590e948adca4864f378dca2135a35; parent_hash: 0xbf64…039f; extrinsics (111): [0x8124…b88f, ....., 0xc226…fe10]]
2021-11-04 08:53:27 🔖 Pre-sealed block for proposal at 118. Hash now 0x61291f449cb166b2329101f84684c48bd7749e5ea0ad3933db72a1de99f7e581, previously 0x13c994d7163d101e97e1bd7b31df0d2ee55590e948adca4864f378dca2135a35.
2021-11-04 08:53:27 ✨ Imported #118 (0x6129…e581)
2021-11-04 08:53:27 📦 Submitted Subspace root block with segment index 11.
2021-11-04 08:53:27 📦 Submitted Subspace root block with segment index 12.
2021-11-04 08:53:27 📦 Submitted Subspace root block with segment index 13.
2021-11-04 08:53:28 Merkle Root segment index 11 not found (slot 1636008808)
2021-11-04 08:53:30 💤 Idle (0 peers), best: #118 (0x6129…e581), finalized #0 (0x1e0d…5700), ⬇ 0 ⬆ 0
2021-11-04 08:53:32 Merkle Root segment index 13 not found (slot 1636008812)

Need to check what happens there exactly and why.

We might need to tweak something about extrinsics priority since we expect them to always be included in the very next block no matter what.

Windows 11 - Failed to encode a piece: CUDA encoding error

System

Lenovo Legion 7

CPU - AMD Radeon Ryzen 9 5900HX
GPU - Geforce RTX 3080 Laptop GPU
RAM - 32GB
OS - Windows 11 version 10.0.22000
File - .\subspace-farmer-x86_64-windows-2022-snapshot-2022-jan-05.exe
Command - farm

Error

Failed to encode a piece: error: CUDA encoding error: Not enough memory available on GPU for pieces

Note

This error could be present due to the fact we had to pin the CUDA version for the Windows 11 build.

Increase eon duration to 1 week

Stack overflow in `tokio-runtime-worker` in fork-tree implementation

Stack overflow is happening in tokio-runtime-worker thread at large block height (450_000-900_000 blocks):

(gdb) bt
#0  0x000055555594cc39 in <sc_client_db::BlockchainDb<Block> as sp_blockchain::header_metadata::HeaderMetadata<Block>>::header_metadata ()
#1  0x0000555555fba225 in sp_blockchain::header_metadata::lowest_common_ancestor ()
#2  0x0000555555acb10d in sc_client_api::utils::is_descendent_of::{{closure}} ()
#3  0x00005555560aeb21 in fork_tree::node_implementation::Node<H,N,V>::import ()
... many more lines just like that
#2998 0x00005555560aea7c in fork_tree::node_implementation::Node<H,N,V>::import ()
#2999 0x0000555555ebdc0d in fork_tree::ForkTree<H,N,V>::import ()
#3000 0x0000555555ae3ce5 in sc_consensus_epochs::EpochChanges<Hash,Number,E>::import ()
#3001 0x00005555560dd4cd in <sc_consensus_poc::PoCBlockImport<Block,Client,Inner> as sc_consensus::block_import::BlockImport<Block>>::import_block::{{closure}} ()
#3002 0x00005555561601a3 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#3003 0x00005555560ff475 in sc_consensus_slots::SimpleSlotWorker::on_slot::{{closure}} ()
#3004 0x0000555556177be3 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#3005 0x000055555616a129 in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#3006 0x0000555556163aae in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#3007 0x0000555555cf78c8 in <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll ()
#3008 0x0000555555f845bf in <futures_util::future::join::Join<Fut1,Fut2> as core::future::future::Future>::poll ()
#3009 0x00005555559cab7a in <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll ()
#3010 0x00005555559c5a63 in <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll ()
#3011 0x0000555555debfaf in <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll ()
#3012 0x0000555555cd1aa7 in <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll ()
#3013 0x000055555619e64f in <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll ()
#3014 0x0000555555e51f42 in std::thread::local::LocalKey<T>::with ()
#3015 0x0000555555c064ca in tokio::park::thread::CachedParkThread::block_on ()
#3016 0x0000555555c8feba in tokio::runtime::handle::Handle::block_on ()
#3017 0x000055555614d53f in tokio::runtime::task::core::CoreStage<T>::poll ()
#3018 0x0000555555910650 in tokio::runtime::task::harness::Harness<T,S>::poll ()
#3019 0x0000555556b8e162 in tokio::runtime::blocking::pool::Inner::run ()
#3020 0x0000555556b91d5b in std::sys_common::backtrace::__rust_begin_short_backtrace::he5973d87642e29ac ()
#3021 0x0000555556b922b8 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#3022 0x000055555748e5e3 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/2c7bc5e33c25e29058cbafefe680da8d5e9220e9/library/alloc/src/boxed.rs:1636
#3023 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/2c7bc5e33c25e29058cbafefe680da8d5e9220e9/library/alloc/src/boxed.rs:1636
#3024 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:106
#3025 0x00007ffff7d30450 in start_thread (arg=0x7ffb6f9fc640) at pthread_create.c:473
#3026 0x00007ffff7afbd53 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Initially reported as a question here: paritytech/substrate#9990

Issue with restarts on non-archival nodes

====================

Version: 0.1.0-d6f12ab5-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:610:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:502:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/sys_common/backtrace.rs:139:18
   4: rust_begin_unwind
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:498:5
   5: core::panicking::panic_fmt
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/core/src/panicking.rs:107:14
   6: core::result::unwrap_failed
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/core/src/result.rs:1613:5
   7: sc_consensus_subspace::block_import
   8: subspace_node::service::new_partial
   9: subspace_node::service::new_full
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: std::thread::local::LocalKey<T>::with
  12: tokio::park::thread::CachedParkThread::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: tokio::runtime::Runtime::block_on
  15: sc_cli::runner::Runner<C>::run_node_until_exit
  16: subspace_node::command::run
  17: subspace_node::main
  18: std::sys_common::backtrace::__rust_begin_short_backtrace
  19: std::rt::lang_start::{{closure}}
  20: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:406:40
      std::panicking::try
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:370:19
      std::panic::catch_unwind
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panic.rs:133:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/rt.rs:128:48
      std::panicking::try::do_call
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:406:40
      std::panicking::try
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panicking.rs:370:19
      std::panic::catch_unwind
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/panic.rs:133:14
      std::rt::lang_start_internal
             at /rustc/cc946fcd326f7d85d4af096efdc73538622568e9/library/std/src/rt.rs:128:20
  21: main
  22: __libc_start_main
             at ./csu/../csu/libc-start.c:332:16
  23: _start


Thread 'main' panicked at 'Failed to get `confirmation_depth_k` from runtime API: Application(UnknownBlock("State already discarded for BlockId::Number(0)"))', /web/subspace/subspace/crates/sc-consensus-subspace/src/lib.rs:1650

Essentially we shoudn't use genesis block for these kinds of queries since its state might be pruned already.

Build fails if `CARGO_TARGET_DIR` is set

Build of wasm runtimes fails if CARGO_TARGET_DIR is set. subspace_wasm_tools assumes to have target directory relative to manifest, which is not the case when user can specify any CARGO_TARGET_DIR and change target directory location.

I tried to fix it, but problem seems to be a bit deeper, as substrate_wasm_builder in the case when it fails to find target directory uses crates output directory for building wasm.

start plotting without syncing is finished

Is it possible to start plotting, although syncing is not finished yet?

Theoretically, should be. But I assume there are many practical implications that are hindering this opportunity.