koinos / koinos-p2p

The p2p microservice orchestrates the distribution of blocks and transactions between peers.

Koinos P2P

Peer to peer microservice of the Koinos blockchain.

koinos-p2p's Issues

Import all known required go modules to koinos-p2p

Koinos types and libp2p for certain.

Story #6

P2P Automated Testing

As a blockchain developer, I want to have automated testing for p2p code, so that I can have assurance when checking in changes to the repository.

Module: go-libp2p-testing

Tasks

• libp2p toolbox research [koinos/koinos-p2p-archaic#2]
• Write unit tests for koinos-p2p [#2]

Gossip Protocol

As a koinos node operator, I want my node to have all current blocks and transactions.

• Listen to transactions and blocks via RabbitMQ broadcast (#30)
• Nodes incrementally advertise new blocks and transactions to peers from RabbitMQ broadcast (#31)
• Nodes can request inventory items that they do not have and then apply them locally (#32)
• Add time based expiration for node inventory that can be easily tuned at a later date (#33)

Note: Blocks should be the header with a vector of transaction IDs to be constructed prior to sending to the controller

Hook up CI / code coverage to koinos-p2p

Get this repo hooked up to Travis CI, with testing, linting, and code coverage.

Story #6

Update Sync Response Protocol

Update the sync response protocol to handle variant messages and responses

Note: Potentially Using https://github.com/libp2p/go-libp2p-gorpc

Instance ID does not respect yaml config

In c++ microservices, the instance id can be overwritten by the yaml config.

P2P currently only accepts an instance ID via command line arguments.

Old blocks should be considered invalid for gossip purposes

We should have some gossip cutoff threshold, say gossip_cutoff = irreversibility - 20. Any earlier block should be reported as invalid to the gossipsub code and not propagated further. That way we if there's an earlier "wave" of peers, it should disconnect from sync and start to gossip at some higher threshold, say gossip_dropout = irreversibility - 10.

Essentially for some fixed height H the following sequence of events should occur:

• (a) Block of height H is first produced somewhere on the network
• (b) Blocks of height H (on different forks) are available via gossip
• (c) Height H becomes irreversible
• (d) Clients whose height is less than H disconnect from gossip and sync
• (e) Blocks of height H are suppressed from the gossip network

Startup logic

We need to implement startup logic to ensure the cluster is ready for this particular microservice.

Dependents:

• koinos-chain
• koinos-block-store

Blocked by koinos/koinos-mq-golang#17

Nodes incrementally advertise new blocks and transactions to peers from RabbitMQ broadcast

RabbitMQ

Add RabbitMQ support to P2P

Compare Chain ID and Block Info

Compare Chain ID, Block Height, and Block ID to peer; disconnect if Chain ID differs, or if already in sync with peer.

Story #3

Listen to transactions and blocks via RabbitMQ broadcast

Drop to gossip mode automatically

The --gossip flag currently does nothing. The intent of --gossip is to join gossip when "caught up" according to some metric. Some questions to answer:

• What metric should be used?
• How do we check the metric?
• Can the metric trigger dropping out of gossip?
• If so, how do we check for going from "caught up" to "not caught up" (e.g. we're far behind due to hours-long internet outage)?

Currently, as a workaround you can use the --force-gossip flag to drop to gossip mode immediately on startup.

Check that block timestamps are not far in the future

Node operators should run NTP or other reasonable timekeeping, so we can trust that system time is reasonably indicative of the actual time. So let's forbid block timestamps more than a few seconds in the future (according to the local system clock).

I'm not 100% sure that this check belongs in the p2p microservice, but I can't immediately think of a better place for it.

Naughty Points

As a koinos node operator, I want my p2p daemon to be resilient to malicious actors.

• Naughty points. Points are added for violations and decay over time. If they grow past a threshold, the peer is disconnected from and added to a decaying black list.
  • Ill formed messages
  • Asking for the same data repeatedly
  • Repeatedly sending invalid blocks and transactions
  • Timeout
  • Advertising a block that contains transactions not in the peer's inventory should award naughty points
  • Outdated/mismatched protocol versions

Write unit tests for koinos-p2p

Possibly use libp2p toolbox to write unit tests for koinos-p2p.

Finish integrating the prototypes created last sprint

Integrate:

• Peer discovery
• Pub/sub
• RPC

P2P Prototype

Prototype P2P Gossip Protocol (See #4 for initial requirements)

The purpose of this research task is to make a best effort at #4, but understand that some aspects of the p2p protocol are currently unknown. This task will either aid in the completion of #4 or add requirements to the design of #4.

Handle apply block failure

If apply block fails, disconnect from the peer and disregard the block

Story #3

Initialize Koinos P2P repo

As a blockchain engineer, I want a solid foundation for koinos p2p microservice, so that I can be held accountable for the code I am about to produce.

• Create a barebones go module, main function and test suite (#7)
• Import all known required go modules (koinos-types, libp2p) (#8)
• Hook up CI / code coverage (#9)

Use newly refactored Koinos types

Bump the Koinos types version and cleanup RPC code.

Koinos P2P Seed Generation

When providing a seed that generates the Peer ID it seems to not be consistent. Investigate with possible code changes to allow for the Peer ID to be the same on each startup.

Default AMQP server

The default login credentials for AMQP are incorrect.

Stub communication functions

Stubs should be created for the communication functions:
get_head_block
apply_block
get_ancestor_at_height
get_chain_id

Story #3

Crash when AMQP is not connected

I inadvertently misconfigured AMQP to dial the wrong port, I got this in the logs when a peer tried to connect to me:

2021/02/17 16:46:26 Connecting to AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:26 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:26 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:46:26 Starting node at address: /ip4/10.89.0.35/tcp/8889/p2p/QmSNE2wo3hH8Yfnwn9GERrD6ziyLRZkYPK8FsZHthMjZMM
2021/02/17 16:46:27 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:27 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:46:30 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:30 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:46:35 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:35 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:46:42 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:42 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:46:51 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:46:51 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:47:02 Dialing AMQP server amqp://guest:guest@localhost:5673/
2021/02/17 16:47:02 AMQP error dialing server: dial tcp 127.0.0.1:5673: connect: connection refused
2021/02/17 16:47:14 Peer connected to us to sync: QmfVKgvmDy-1-3
2021/02/17 16:47:14 QmfVKgvmDy-1-3: checking peer's chain id
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x6608d5]

goroutine 16 [running]:
github.com/streadway/amqp.(*Channel).Publish(0x0, 0xb143a9, 0xa, 0xc000026690, 0x10, 0xc000020000, 0x0, 0xb17ecd, 0x10, 0x0, ...)
        /root/go/pkg/mod/github.com/streadway/[email protected]/channel.go:1331 +0x75
github.com/koinos/koinos-mq-golang.(*KoinosMQ).SendRPC(0xc00024a280, 0xb17ecd, 0x10, 0xb0f536, 0x5, 0xc0005b4440, 0x3d, 0x40, 0x3c, 0x8, ...)
        /opt/koinos-src/koinos-mq-golang/koinosmq.go:254 +0x245
github.com/koinos/koinos-p2p/internal/rpc.KoinosRPC.GetChainID(0xc00024a280, 0xa2c8c0, 0x1, 0x114b630)
        /opt/koinos-src/koinos-p2p/internal/rpc/koinos_rpc.go:173 +0xf3
github.com/koinos/koinos-p2p/internal/protocol.SyncProtocol.handleStream(0xcf7860, 0xc0000ac178, 0xcfc240, 0xc000100000, 0xc0001e6010, 0xcfc920, 0xc00000ed60)
        /opt/koinos-src/koinos-p2p/internal/protocol/sync_protocol.go:78 +0x313
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1(0xc0001941a0, 0x12, 0x7f9a6410b1c8, 0xc00000ed60, 0x1, 0x0)
        /root/go/pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:566 +0x9f
created by github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler
        /root/go/pkg/mod/github.com/libp2p/[email protected]/p2p/host/basic/basic_host.go:416 +0x613

Running integration-clean branch (commit 095d36b)

Update `types`, `mq`, and binary outputs in the format `koinos_*` (snake_case)

Blocked by koinos/koinos-types#152

Log messages

Improve logging when receiving blocks over the gossip/sync protocol.

We should include Block ID, Height, and Previous when logging received blocks over sync/gossip.
We should include Transaction ID when logging transactions over gossip.

Define p2p protocol using libp2p interface

Create a new protocol for sync mode, hook it up to the host / node

Story #3

Ensure that we don't gossip blocks we can't apply

We shouldn't gossip blocks we can't apply locally.

Libp2p has the ability to do an application-specific check for gossip items.

We need to be sure that block application is properly plugged into that application-specific check interface.

Sync Manager

Implement the sync manager

Sync Manager Algorithm:
Connect to N peers (protocol contains chain id check irreversible fork check)
Get head block for each peer
Target sync block is (min(peer_head_blocks) - my_head_block) / 2 (perhaps maximum of 100k)
For each peer, request block id at target sync block height, separating peers in to different forks depending on response
For each fork, make batch requests to get caught up to head block id at height for that fork
If our node is now within delta D blocks of peer, switch it to gossip mode
Repeat until all peers are in gossip mode

Blocked on #44

Separate p2p topic specifically for peer exchange

Participation in peer exchange (PEX) is on a per-topic basis. We want to get PEX peers but not transactions or blocks in some relatively common situations (e.g. spinning up a brand-new node, or restarting an outdated node after hours of downtime).

The best way to implement this is to create a new topic, parallel to blocks and transactions, specifically for PEX.

---

This PEX topic would then somehow sync its peers.

https://pkg.go.dev/github.com/libp2p/go-libp2p-pubsub#Topic.ListPeers

This function looks promising.

https://pkg.go.dev/github.com/libp2p/go-libp2p-pubsub#PubSubRouter

If this router can be used, it has events when a peer is added or removed, so this also might be of use.

https://github.com/libp2p/go-libp2p-discovery

This library also exists for the express purpose of peer discovery. This may be the more correct method.

Peer Discovery

As a Koinos Node operator, I want my p2p node to learn about new peers from existing peers.

Notes:

• Look at https://github.com/libp2p/go-libp2p-discovery
• Should try and target N peers and refuse connections over M peers (configurable)

Deliverables:

• Prototype of peer discovery (might be sufficient for prod)

Nodes can request inventory items that they do not have and then apply them locally

Reconnect to Peers

If a connection goes down, libp2p does not seem to automatically want to reconnect to peers.

P2P: RabbitMQ RPC

As a P2P node, I want to get current info from koinosd and the block store.

The required queries are: get_head_block, apply_block, get_ancestor_at_height, get_chain_id

• Add required arg and ret structs to koinos types for required queries. (koinos/koinos-types#120)
• Add RabbitMQ support to koinos-p2p (#21)
• Implement stubbed p2p methods: (#22)
  • get_head_block
  • apply_block
  • get_ancestor_at_height
  • get_chain_id

Fix automated tasks

Make the branch big-p2p-refactor tests pass in CI, which should be all Sync related code.

Stub Sync Manager

Stub the sync manager that provides the registration interface, and add message request/response definitions

Sync Protocol

As a Koinos node operator, I want to get a new Koinos node from genesis to the current head block.

• Define p2p protocol using libp2p interface (#14)
• Figure out how to test libp2p protocols (#15)
• Stub out communication functions (get_head_block, apply_block, get_ancestor_at_height, get_chain_id) (#16)
• Compare Chain ID, Block Height, and Block ID to peer; disconnect if Chain ID differs, or if in sync with peer (#17)
• If on same fork batch receive and apply blocks from peer, else hang up (#18)

Implement RPC Calls

Implement stubbed p2p methods:

• get_head_block
• apply_block
• get_ancestor_at_height
• get_chain_id

Create Dockerfile

Create a Dockerfile to build and run the microservice.

Fork heads management

As a blockchain dev, I would like to maintain fork information without using RPC polling techniques.

• Add a broadcast message to koinos-types called koinos::broadcast::fork_heads (koinos/koinos-types#180)
• Modify koinos-chain to broadcast the new message at the same time we broadcast block acceptance (koinos/koinos-chain#379)
• Modify koinos-p2p to maintain the local fork heads data based on the broadcast message (#93)

Error and cancellation handling for readMessages

Currently readMessages here has a few problems:

• Should use select to wait on both ctx and ch
• Should have a mechanism to attribute the error to the peer specified by GetFrom() if Next() returns err
• Should have a mechanism to attribute the error to the peer specified by GetFrom() if block fails to apply (as determined by channel consumer)

To implement these mechanisms, we should probably lift errPeers out of its current location in SyncManager to some point closer to top-level (e.g. KoinosP2PNode), so that way errPeers can service errors generated in both SyncManager and GossipManager.

Add time based expiration for node inventory that can be easily tuned at a later date

State Machine

As a Koinos node operator, I want the p2p microservice to be able to move between sync and gossips modes for each peer as needed to keep my and other's nodes in sync.

We may be able to replace the sync protocol with a simple RPC wrapper https://github.com/libp2p/go-libp2p-gorpc

Sync Manager Algorithm:
Connect to N peers (protocol contains chain id check irreversible fork check)
Get head block for each peer
Target sync block is (min(peer_head_blocks) - my_head_block) / 2 (perhaps maximum of 100k)
For each peer, request block id at target sync block height, separating peers in to different forks depending on response
For each fork, make batch requests to get caught up to head block id at height for that fork
If our node is now within delta D blocks of peer, switch it to gossip mode
Repeat until all peers are in gossip mode

All new connections are added as "sync mode"
New connections register themselves to a manager. The manager tracks head blocks and potential forks and delegates batch requests to peers to get all peers. (Can pipeline between peers)
When a peer has no more new blocks to provide, the manager sets the node to "gossip mode" (hang up and dial peer in gossip mode)

Future optimization: Combine both modes in to a single protocol with combined RPC model

• Stub the sync manager that provides the registration interface, and add message request/response definitions (#44)
• Implement the sync manager to adhere to the algorithm above (#45)
• Update the sync protocol to handle messages from the manager (keeping the chain id and fork checks during handshake) (#46)
• Update the sync response protocol to handle variant messages and responses (#47)

Update Sync Protocol

Update the sync protocol to handle messages from the manager (keeping the chain id and fork checks during handshake)

Note: Potentially making use of https://github.com/libp2p/go-libp2p-gorpc

Create barebones repo

Create a barebones go module, main function and test suite.

Story #6

Figure out how to test libp2p protocols

Look into testing libp2p protocols using the libp2p-toolbox

Story #3

Libp2p Pub/Sub

We stumbled across and official pub/sub written on top of libp2p which looks like it might meet our needs for gossip mode.

https://github.com/libp2p/go-libp2p-pubsub

Research how pubsub works and if it looks compatible with what we need, write a prototype to replace gossip.

Check peer fork, apply blocks

If peer is on the same fork then batch receive and apply blocks from peer, else hang up
If apply block fails, disconnect from the peer and disregard the block
Story #3

Implement unit test for BlockDownloadManager

Currently BlockDownloadManager has only been tested by somewhat ad hoc integration tests. It needs unit test(s).