[Feat.Req] Deny senders over a certain amount of unaggregated fees

indexer-service-rs

Introduction

A Rust impl for The Graph indexer service to provide data services as an Indexer, integrated with TAP which is a fast, efficient, and trustless unidirectional micro-payments system.

Features

Receive paid or free query requests and route to graph node
Route "meta" queries on indexing statuses and deployment health
Serve indexer information such as health, indexer version, and operator address
Monitor allocations, attestation signers, and manage receipts using TAP, store receipts in the indexer database
Record performance and service metrics

Quick start

$ cargo run -p service -- --help

Usage: service --config <FILE>

Options:
      --config <FILE>  Path to the configuration file.
                       See https://github.com/graphprotocol/indexer-rs/tree/main/service for examples.
  -h, --help           Print help

All the configuration is done through a TOML file. Please see up-to-date TOML configuration templates:

Upgrading

We follow conventional semantics for package versioning. An indexer may set a minor version specification for automatic patch updates while preventing breaking changes. To safely upgrading the package, we recommend the following steps:

Review Release Notes: Before upgrading, check the release notes for the new version to understand what changes, fixes, or new features are included.
Review Documentation: Check the up-to-date documentation for an accurate reflection of the changes made during the upgrade.
Backup Configuration: Save your current configuration files and any local modifications you've made to the existing codebase.
Deploy: Replace the old executable or docker image with the new one and restart the service to apply the upgrade.
Monitor and Validate: After the upgrade, monitor system behavior and performance metrics to validate that the service is running as expected.

These steps should ensure a smooth transition to the latest version of indexer-service-rs, harnessing new capabilities while maintaining system integrity.

Contributing

Contributions guide

Supported request and response format examples

✗ curl http://localhost:7300/
Ready to roll!

✗ curl http://localhost:7300/health
{"healthy":true}

✗ curl http://localhost:7300/version
{"version":"0.1.0","dependencies":{}}

✗ curl http://localhost:7300/operator/info
{"publicKey":"0xacb05407d78129b5717bb51712d3e23a78a10929"}

# Subgraph queries
# Checks for receipts and authorization
✗ curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer token-for-graph-node-query-endpoint' --data '{"query": "{_meta{block{number}}}"}' http://localhost:7300/subgraphs/id/QmacQnSgia4iDPWHpeY6aWxesRFdb8o5DKZUx96zZqEWrB
"{\"data\":{\"_meta\":{\"block\":{\"number\":9425787}}}}"

# Takes hex representation for subgraphs deployment id aside from IPFS hash representation
✗ curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer token-for-graph-node-query-endpoint' --data '{"query": "{_meta{block{number}}}"}' http://localhost:7300/subgraphs/id/0xb655ca6f49e73728a102219726ff678d61d8fb792874792e9f0d9887dc616600
"{\"data\":{\"_meta\":{\"block\":{\"number\":9425787}}}}"

# Free query auth token check failed
✗ curl -X POST -H 'Content-Type: application/json' -H 'Authorization: blah' --data '{"query": "{_meta{block{number}}}"}' http://localhost:7300/subgraphs/id/0xb655ca6f49e73728a102219726ff678d61d8fb792874792e9f0d9887dc616600
"Invalid Tap-Receipt header provided"%

# Subgraph health check
✗ curl http://localhost:7300/subgraphs/health/QmVhiE4nax9i86UBnBmQCYDzvjWuwHShYh7aspGPQhU5Sj
"Subgraph deployment is up to date"%
## Unfound subgraph
✗ curl http://localhost:7300/subgraphs/health/QmacQnSgia4iDPWHpeY6aWxesRFdb8o5DKZUx96zZqEWrB
"Invalid indexing status"%

# Network queries
# Checks for auth and configuration to serve-network-subgraph
✗ curl -X POST -H 'Content-Type: application/json' -H 'Authorization: token-for-network-subgraph' --data '{"query": "{_meta{block{number}}}"}' http://localhost:7300/network
"Not enabled or authorized query"

# Indexing status resolver - Route supported root field queries to graph node status endpoint
✗ curl -X POST -H 'Content-Type: application/json' --data '{"query": "{blockHashFromNumber(network:\"goerli\", blockNumber: 9069120)}"}' http://localhost:7300/status
{"data":{"blockHashFromNumber":"e1e5472636db73ba5496aee098dc21310683c95eb30fc46f9ba6c36d8b28d58e"}}%

# Indexing status resolver -
✗ curl -X POST -H 'Content-Type: application/json' --data '{"query": "{indexingStatuses {subgraph health} }"}' http://localhost:7300/status
{"data":{"indexingStatuses":[{"subgraph":"QmVhiE4nax9i86UBnBmQCYDzvjWuwHShYh7aspGPQhU5Sj","health":"healthy"},{"subgraph":"QmWVtsWk8Pqn3zY3czDjyoVreshRLmoz9jko3mQ4uvxQDj","health":"healthy"},{"subgraph":"QmacQnSgia4iDPWHpeY6aWxesRFdb8o5DKZUx96zZqEWrB","health":"healthy"}]}}

# Indexing status resolver - Filter out the unsupported queries
✗ curl -X POST -H 'Content-Type: application/json' --data '{"query": "{_meta{block{number}}}"}' http://localhost:7300/status
{"errors":[{"locations":[{"line":1,"column":2}],"message":"Type `Query` has no field `_meta`"}]}%

######## Cost server - read-only graphql query
curl -X GET -H 'Content-Type: application/json' --data '{"query": "{ costModel(deployment: \"Qmb5Ysp5oCUXhLA8NmxmYKDAX2nCMnh7Vvb5uffb9n5vss\") { deployment model variables }} "}' http://localhost:7300/cost

curl -X GET -H 'Content-Type: application/json' --data '{"query": "{ costModel(deployment: \"Qmb5Ysp5oCUXhLA8NmxmYKDAX2nCMnh7Vvb5uffb9n5vss\") { deployment model variables }} "}' http://localhost:7300/cost
{"data":{"costModel":{"deployment":"0xbd499f7673ca32ef4a642207a8bebdd0fb03888cf2678b298438e3a1ae5206ea","model":"default => 0.00025;","variables":null}}}%

curl -X GET -H 'Content-Type: application/json' --data '{"query": "{ costModel(deployment: \"Qmb5Ysp5oCUXhLA8NmxmYKDAX2nCMnh7Vvb5uffb9n5vas\") { deployment model variables }} "}' http://localhost:7300/cost
{"data":{"costModel":null}}%

curl -X GET -H 'Content-Type: application/json' --data '{"query": "{ costModel(deployment: \"Qmb5Ysp5oCUXhLA8NmxmYKDAX2nCMnh7Vvb5uffb9n5vss\") { deployment odel variables }} "}' http://localhost:7300/cost
{"errors":[{"message":"Cannot query field \"odel\" on type \"CostModel\". Did you mean \"model\"?","locations":[{"line":1,"column":88}]}]}%

curl -X GET -H 'Content-Type: application/json' --data '{"query": "{ costModels(deployments: [\"Qmb5Ysp5oCUXhLA8NmxmYKDAX2nCMnh7Vvb5uffb9n5vss\"]) { deployment model variables }} "}' http://localhost:7300/cost
{"data":{"costModels":[{"deployment":"0xbd499f7673ca32ef4a642207a8bebdd0fb03888cf2678b298438e3a1ae5206ea","model":"default => 0.00025;","variables":null}]}}%

Dependency choices

switching from actix-web to axum for the service server
App profiling should utilize perf, flamegraphs or cpu profilers, and benches to track and collect performance data. The typescript implementation uses gcloud-profile
Consider replacing and adding parts from TAP manager
postgres database connection required to indexer management server database, shared with the indexer agent
No migration in indexer service as it might introduce conflicts to the database; indexer agent is solely responsible for database management.

Indexer common components

Temporarily live inside the indexer-service package under src/common.

Simple indexer management client to track NetworkSubgraph and postgres connection.

NetworkSubgraph instance track both remote API endpoint and local deployment query endpoint.
- TODO: query indexing status of local deployment, only use remote API as fallback.
Keeps cost model schema and resolvers with postgres and graphQL types: costModel(deployment) and costModels(deployments). If deployments is empty, all cost models are returned.
- Global cost model fallback used when specific deployments are queried
No database migration in indexer service as it might introduce schema conflicts; indexer agent is solely responsible for database management.

Indexer native dependency

Linked dependency could not be linked directly with git url "https://github.com/graphprotocol/indexer" and path "packages/indexer-native/native" at the same time, and could not access it on crates.io. So copid the folder to local repo with the version at https://github.com/graphprotocol/indexer/blob/972658b3ce8c512ad7b4dc575d29cd9d5377e3fe/packages/indexer-native/native.

Since indexer-service will be written in Rust and no need for typescript, indexer-native's neon build and util has been removed.

Component NativeSignatureVerifier renamed to SignatureVerifier.

Separate package in the workspace under 'native'.

common-ts components

Temporarily live inside the indexer-service package under src/types

Address
readNumber

	// TODO: Request compression and response decompression. Also a fancy user agent?
	let client = HttpClientBuilder::default()
	.request_timeout(Duration::from_secs(
	inner.config.tap.rav_request_timeout_secs,
	))
	.build(&inner.sender_aggregator_endpoint)?;

	let required_checks = vec![
	ReceiptCheck::CheckUnique,
	ReceiptCheck::CheckAllocationId,
	ReceiptCheck::CheckTimestamp,
	// ReceiptCheck::CheckValue,
	ReceiptCheck::CheckSignature,
	ReceiptCheck::CheckAndReserveEscrow,
	];

	// TODO: replace with a proper implementation once the gateway registry contract is ready
	let sender_aggregator_endpoints = aggregator_endpoints::load_aggregator_endpoints(
	config.tap.sender_aggregator_endpoints_file.clone(),
	);

	// TODO: remove SenderAllocationRelationship instances that are finished. Ideally done in
	// another async task?

	// TODO: Handle invalid receipts

graphprotocol / indexer-rs Goto Github PK

indexer-rs's Introduction

indexer-service-rs

Introduction

Features

Quick start

Upgrading

Contributing

Supported request and response format examples

Dependency choices

Indexer common components

Indexer native dependency

common-ts components

Components checklist (basic, not extensive)

indexer-rs's People

Contributors

Stargazers

Watchers

Forkers

indexer-rs's Issues

steps to reproduce

Problem statement

Proposal

Alternative considerations

Additional context

Recommend Projects

Recommend Topics

Recommend Org

Jobs