datafast-network / datafast-runtime

A Subgraph-indexing-runtime that prioritises performance & cost efficiency

Datafast-Runtime

Document available at https://runtime.datafast.network

datafast-runtime's Issues

Bump deltalake to 17 to fix "Scarlar value unimplemented" for protobuf serialization

ProgressControl check for indexing continuity might not be correct

Progress Tracking / Checkpoint Control

Provide SWR the capability to resume progress and prevent duplicate block consumption

Implement Array type for store

Implement Filter Call Function for log

Critical Bug in logic of handling creating-data-source & no-address-datasource

review bug

Impl dedicated Logger for WasmInstance

Use a dedicated Slog's Logger instance for each wasm instance

revert Error types

CLI Mode

App can be run as CLI with options

Truncate data

Each datasource should be able to config maximum block history to store data

Structured cache system

Add more logs

SWR is lacking logs. Must provide more logs for it.

• Log on different wasm-hosts should have different target names
• Log should be enough, not too many that it can cause performance degradation
• Might need some macros to make it happen

Real DB for Store

Implement store.get_related & store.get_in_block for wasm host

Source from grpc

Implement GRPC server/client for ingest source input

1. Init GRPC Server
2. Init Substream client

Impl IPFS loader

Implement NATS

Subscribe block data from NATS (so SWR can run along with Pullcaster)

Implement Graphql server and querier

Logging

Add more logs, and format it to be more readable

Update Application Diagram

Update diagram on readme, blocking by #41

Source from TCP

Implement TCP socket for ingest source input

Implement RPC call

Create rpc client handle call function for wasm host

Chunk BatchStatement

Verify non null data

Graph schema define non-null data but Schemas & Database are not verifying them. Must check before insert.

Quick Transform Test CLI

Provide SWR the ability to run as a command line for testing indenpendent transform function

Parse subgraph.graphql to DLL or db Schema

Optimize subgraph's create-source

creating wasm-sources takes ~ 20ms (even if only 3 datasources) - which is quite a lot.

If query 10k-block step, and create new sources every time we process 1 single block, then:

• it will take 20ms * 10_000 = 200_000ms = 200s = 3.3' (3 minutes 20 seconds)
• if process 1M blocks, it will cost in total 3.3 * 100 = 330' = 5.5 hours
• if process 10M blocks, it will cost 5.5 * 10 = 55h = 2.3 day

Implement create new datasource for wasm host

Rpc get latest block

Memory leak in wasmer host

Prometheus

Load test NATS

If NATS cannot carry more than 100Mb block, it is useless.

Handle datasource without address

Polling Query for blocks at chain-head

Binding Block's store & Ethereum Block store