protean-labs / subgrounds

A Pythonic data access layer for applications querying data from The Graph Network.

⚠️ This repository is no longer maintained ⚠️

The official repository has been moved to https://github.com/0xPlaygrounds/subgrounds

subgrounds's Issues

Pip installation error

I am using python 3 (tried both 3.8 and 3.7) on Ubuntu 20.04 LTS.
On trying to install using pip 22.0.4,
pip install subgrounds

I get these errors
ERROR: Could not find a version that satisfies the requirement subgrounds (from versions: none)
ERROR: No matching distribution found for subgrounds

Shorthand to query all entities

Description

Create a special value ALL that can be used as the first argument in list queries to fetch all entities.

Example

swaps = uniswapV2.Query.swaps(
  orderBy=uniswapV2.Swap.timestamp,
  orderDirection='desc',
  first=ALL
)

Review integration with Python Language Server

We should start to map out the design space for the interaction between the python classes generated by a subgraph and the autocomplete/linter.

1. better discovery for fields through type previews
2. can identify possible naming discrepencies between class properties and the graphql schema
3. autocomplete for both imported fields and synthetic fields

Immediate single entity query

Description

Subgrounds should allow developers to query a single entity from an id. The query (and returned object) should only have scalar, non-list values.

For nested fields returning non-scalar (i.e.: other entities) or list values, they should be queried at read time.

Example

sg = Subgrounds()
uniswapV2 = sg.load_subgraph('https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2')

# Fetch the pair and all its non-list scalar values with the id '0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc' 
some_pair = uniswapV2.Pair('0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc')

# Since txCount is a non-list scalar, the field's value would be prefetched 
# and reading it would not send a new query 
num_tx = some_pair.txCount

# Since token0 is not a scalar value, then it would be fetched at read 
# time and reading it would imply sengin a new query (all handled in the background)
token0_symbol = some_pair.token0.symbol

Ranged pagination

Description

Subgrounds should be able to paginate based on a condition (e.g.: get all Uniswap V2 swaps between time t0 and time t1, get all Aave V2 USDC borrows larger than amount).

Example

The following subgrounds fieldpath implies the use of pagination even though no first argument is present

swaps = uniswapV2.Query.swaps(
  orderBy=uniswapV2.Swap.timestamp,
  orderDirection='desc',
  where=[
    uniswapV2.Swap.timestamp >= 1641013200
  ]
)

Better `FieldPath` argument validation

Is your feature request related to a problem? Please describe.
Yes. When constructing FieldPaths with arguments, any extra arguments (i.e.: arguments which don't exist in the subgraph schema) are silently ignored, leading to harder debugging in case of errors like typos.

Moreover, although the values of real arguments are type-checked, the exceptions are cryptic and uninformative.

Describe the solution you'd like
When constructing FieldPaths with arguments, for each argument, Subgrounds should check whether:

1. The argument exists in the schema
2. The type of the argument's value matches the type indicated in the schema
3. If using relative FieldPaths (e.g.: subgraph.Query.entities(orderBy=subgraph.Entity.field)), validate that the field type on which that argument is applied matches the type of the relative FieldPath (i.e.: the type of entities matches the Entity object).

Error messages should also be clear and informative. E.g.: "X is not a valid value for argument orderBy".

Describe alternatives you've considered
None

Additional context
None

Implementation checklist

• Validate arguments
  • Check whether the argument exists in the schema
  • Check whether the value given to the argument matches the type specified in the schema
  • Check whether relative FieldPaths match the type of the field
• Improved error messages

Explore networkx integration

Description

Networkx is a Python graph manipulation and visualization library which is supported by Dash/Plotly (see https://plotly.com/python/network-graphs/). Integrating networkx in Subgrounds could provide interesting visualization options which (correct me if I am wrong), even Dune does not support yet.

Support for ranged time-travel queries

Description

The Graph supports time-travel queries where one can optionally specify the block number at which query should be executed. This will execute the query against the state of the subgraph at that block (see time-travel queries documentation).

Currently, Subgrounds supports this out of the box. Using the Uniswap V3 subgraph as reference:

>>> pair = uniswap_v3.Query.pool(
...   id='0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8',
...   block={'number': 14673000}
... )

>>> sg.query([
...   pair.token0Price,
...   pair.token1Price
... ])
(2918.0646522066227, 0.00034269288695978894)

However, if one wishes to query the subgraph at multiple blocks (e.g.: getting historical data), one still has to create multiple queries. Subgrounds should support support providing a block range and interval to make it easier to run a query at multiple blocks. For example (WIP, not final):

>>> pair = uniswap_v3.Query.pool(
...   id='0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8',
...   block={
...     'from': 14663000,
...     'to': 14674000,
...     'interval': 1000
...   }
... )

>>> sg.query([
...   pair.token0Price,
...   pair.token1Price
... ])
[
  (14663000, 2807.466744495506, 0.0003561929992441272),
  (14664000, 2831.947023023911, 0.00035311395017983597),
  (14665000, 2866.517275937252, 0.0003488553892189729),
  ...
  (14674000, 2929.497568964089, 0.0003413554633375626),
]

If Subgrounds is performing a ranged time-travel query, then the returned data should be a list containing the query result at each block with an additional block field indicating the block at which the query was executed.

Lists of scalars not properly handled in `df_of_json`

Description

Queries that select fields of type scalar lists (e.g.: [BigInt!]!, [String!]!...) are not flattened properly in df_of_json (subgrounds.dataframe_utils).

Inline `SyntheticFields`

Is your feature request related to a problem? Please describe.
Yes. Currently, SyntheticFields must be explicitly added to a subgraph object. This works when synthetic fields do not need to change after they are defined. However, to have a SyntheticField that changes over the lifetime of a Subgrounds program, one would have to constantly rebind it. Example:

def query_with_n(n: int):
    subgraph.MyEntity.field_plus_n = subgraph.Entity.field + n

    return sg.query([
        subgraph.Query.myEntities.field_plus_n
    ])

Describe the solution you'd like
Add support for inline SyntheticFields:

def query_with_n_inline(n: int):
    sg.query([
        subgraph.Query.myEntities.field + n
    ])

This will require figuring out a naming scheme for the inline SyntheticFields since, due to the fact that they are not explicitly added to an object, they do not have a user defined name. Using our example, something like field_N could work, where N is the number of inline SyntheticField using field as dependency (would have to figure out how to extend this to inline SyntheticFields that have multiple field dependencies).

Describe alternatives you've considered
None.

Additional context
None.

Implementation checklist

• Add a new function to Subgrounds object to be applied to the fpaths argument of toplevel functions (e.g.: query_df) which will generate temporary transformation layers on the fly for the inline SyntheticFields
• Tests!

Timeseries

Description

Given a subgrounds fieldpath(s) representing a list of entities, then it should be possible to wrap those fieldpaths in a timeseries which would normalize the data according to time key, interval, aggregation method and interpolation method.

Supported intervals (to start):

• hourly
• daily
• weekly
• monthly

Supported aggregation methods:

• mean
• sum
• first
• last
• median
• min
• max
• count

Supported interpolation methods:

• backward fill (use next value to fill in missing value)
• forward fill (use previous value to fill in missing value)

Example

sg = Subgrounds()
uniswapV2 = sg.load_subgraph("https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2")

Swap.price0 = abs(Swap.amount1In - Swap.amount1Out) / abs(Swap.amount0In - Swap.amount0Out)
Swap.price1 = abs(Swap.amount0In - Swap.amount0Out) / abs(Swap.amount1In - Swap.amount1Out)

swaps = uniswapV2.Query.swaps(
  orderBy=Swap.timestamp,
  orderDirection='desc',
  first=500,
)

price0_hourly_close = Timeseries(
  x=swaps.timestamp,
  y=swaps.price0,
  interval='hour',
  aggregation='last',
  interpolation='ffill'
)

Last and first built-in synthetic fields on list fields

Description

Given a subgraph with a query field entities of type [Entity!]!, then entities.first and entities.last should produce a field path that represents the first and last of those entities ordered by timestamp (by default). If timestamp is not a valid field of entity Entity, then the field by which to order the entities can be specified using the orderBy argument.

Example

sg = Subgrounds()
uniswapV2 = sg.load_subgraph("https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2")

# These two are equivalent
last_swap_id = uniswapV2.Query.swaps.last().id

last_swap_id = uniswapV2.Query.swaps(
  orderBy=Swap.timestamp,
  orderDirection='desc',
  first=1
).id

# These two are also equivalent
last_pair_id = unsiwapV2.Query.pairs.last(orderBy='createdAtTimestamp').id

last_pair_id = uniswapV2.Query.pairs(
  orderBy=Pair.createdAtTimestamp,
  orderDirection='desc',
  first=1
).id

Conflicting field paths are overwritten

Description

Multiple field paths that overlap in terms of entity selection but differ in their arguments will be merged into one if present in the same request (one overwriting the other).

Subgrounds requests with arguments

Description

Subgrounds fieldpaths and requests creation whould allow developers to specify variables in their queries.

The requests containing variables would act as request templates which can be used to create actual requests.

Example

sg = Subgrounds()
uniswapV2 = sg.load_subgraph("https://api.thegraph.com/subgraphs/name/uniswap/uniswap-v2")

swaps = Query.swaps(
  orderBy=Swap.timestamp,
  orderDirection='desc',
  first=500,
  where=[
    Swap.pair == Variable('address')
  ]
)

req_template = sg.mk_request([
  swaps.timestamp, 
  swaps.pair.token0.symbol,
  swaps.pair.token1.symbol,
  swaps.amount0In,
  swaps.amount1In,
  swaps.amount0Out,
  swaps.amount1Out
])

actual_req = req_template(address='0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc')

data = sg.execute(actual_req)

Automatic field selection for non-leaf fieldpaths

Description

Subgrounds should allow users to query partial fieldpaths, i.e.: fieldpaths that do not end at a scalar field, in which case Subgrounds should implicitely select all fields of the object at which the fieldpath ends. In case these fields are also objects, then the id of those objects should be queried.

Example

Consider the following toy subgraph schema:

type Pair @entity {
  id: ID!
  createdAtTimestamp: BigInt!
  token0: Token!
  token1: Token!
}

type Token @entity {
  id: ID!
  symbol: String!
  name: String!
}

Then these two queries should be equivalent:

pairs = subgraph.Query.pairs(...)

# First query
sg.query_df([
  pairs
])

# Second query
sg.query_df([
  pairs.id,
  pairs.createdAtTimestamp,
  pairs.token0.id,
  pairs.token1.id
])

Likewise, the following two queries should also be equivalent:

pairs = subgraph.Query.pairs(...)

# First query
sg.query_df([
  pairs.token0
  pairs.token1
])

# Second query
sg.query_df([
  pairs.token0.id,
  pairs.token0.symbol,
  pairs.token0.name,
  pairs.token1.id,
  pairs.token1.symbol,
  pairs.token1.name,
])

Synthetic Fields over existing field paths

Is your feature request related to a problem? Please describe.

A lot of times, what I want to perform when creating a synthetic field is to post-process the queried value of a field path but retain the same name.

Describe the solution you'd like
For example

# Converting decimals for an erc20 token amount transparently 
subgraph.Entity.token_amount = subgraph.Entity.token_amount / 1e6 

This is pretty easy to do in post-processing, but I thought it might be nice to have as a capability for synthetic fields.

Describe alternatives you've considered

When I tried to do something like this for a subgraph I'm working with, I got the following error. At the very least, it would be nice to have a more descriptive error message about this behavior not being allowed.

Exception: transform_response: data for selection Selection(fmeta=TypeMeta.FieldMeta(name='order', description=None, arguments=[], type_=TypeRef.Named(name_='PodOrder')), alias=None, arguments=[], selection=[Selection(fmeta=TypeMeta.FieldMeta(name='status', description=None, arguments=[], type_=TypeRef.NonNull(inner=TypeRef.Named(name_='String'))), alias=None, arguments=[], selection=[]), Selection(fmeta=TypeMeta.FieldMeta(name='pricePerPod', description=None, arguments=[], type_=TypeRef.NonNull(inner=TypeRef.Named(name_='Int'))), alias=None, arguments=[], selection=[])]) is neither list or dict None

Additional context
N/A

Implementation checklist
N/A

Synthetic fields division by zero

Description

When creating a synthetic field that features a division of one field by another, subgrounds will crash if the denominator is zero at any time in the computation of the synthetic field.

Defining Synthetic Fields within Component Arguments

The definition of synthetic fields have to be specified outside the definition of a Dash component

Borrow.amount2 = Borrow.amount + 1

BarChart(
        Query.borrows,
        orderBy=Borrow.timestamp,
        orderDirection="desc",
        first=100,
        x=Borrow.reserve.symbol,
        y=Borrow.amount2
      )

Defining the synthetic field inside the visualization component could provide a better UX especially when considering one off mappings.

BarChart(
        Query.borrows,
        orderBy=Borrow.timestamp,
        orderDirection="desc",
        first=100,
        x=Borrow.reserve.symbol,
        y=Borrow.amount + 1
      )

Subgrounds data formatted into pandas DataFrame uses internal hashed names

Description

Due to some recent changes (#15), Subgrounds now aliases GraphQL selections with a unique id (the hash of the field name + arguments). These are visible in the DataFrame after calling to_dataframe on a Subgrounds data blob.

Incorrect Pagination leading to missing results

Describe the bug
Missing data due to Incorrect pagination.

To Reproduce
Steps to reproduce the behavior:
Query any subgraph which has e.g. a "timestamp" and should return > 1800 entities. After the first pagination, instead of using "skip: 1800" and "first: 900", in my case it uses "skip: 0", "first: 900" and "timestamp_gt: $lastOrderingValue0".
If the timestamp of result with index 1800 is the same as 1801, this will skip 1801 and any other events which happen to have the same timestamp as index 1800.

Expected behavior
Results with same timestamp as the last returned timestamp from the previous result set of the pagination are present.

Sorry I'm not experienced enough with the library to understand how it determines over which field to paginate - a briefer summary is that pagination shouldn't occur over fields which are not guaranteed to be unique.

Synthetic fields not tied to a particular Subgraph

Laying the groundwork for the eventual capacity to seemlessly mash fields from different subgraphs.

borrows = aave.Borrows + compound.Borrows