Warp Gateway - a fast and reliable way to load SmartWeave transactions. All the data indexed by our gateway can be viewed in SonAR.
- fast - load your contract interactions in seconds, not minutes!
- reliable - built-in protection against forks and corrupted transactions
SmartWeave is an Arweave-based protocol for lazy-evaluated smart contracts. Each interaction with a contract is saved as a separate Arweave transaction. In order to evaluate the contract state, all of its interactions must be loaded first.
In the case of typical content getting a corrupted or not confirmed transaction is not a severe issue, while any inconsistency could be extremely harmful to smart contracts, leading to a potential financial loss due to the corrupted state of token contracts.
The current available solution ("general-purpose" Arweave gateway) has some flaws:
-
It doesn't scale well
The interactions can be loaded using the GQL endpoint, which can return only 100 interactions in a single batch/query. At the time of writing, each query takes from ~300ms to ~5 seconds.
For our loot contract (that has 9821 interactions) it means that loading all the interactions takes around 1 minute.
There are contracts with much more interactions - the biggest one has over 280K interactions - loading all the interactions for this contract takes ~3 hours. This clearly shows that the current solution scales poorly and is a first big obstacle for a wider SmartWeave contracts adoption. -
Corrupted transactions
The Arweave gateway GQL endpoint tends to return corrupted transactions - i.e. such transactions, that are not part of any Arweave block - probably due to some caching issues/bug in the Arweave gateway (https://discord.com/channels/357957786904166400/756557551234973696/891254856638160917) The state evaluated with such corrupted transactions is obviously flawed. In case of our loot contract - 25 transactions returned by the Arweave gateway are corrupted. -
Forked blocks problem
No protection against transactions from forked blocks.
All the issues also make things like caching the contract state very risky - as you have no guarantee that the state has been evaluated for proper inputs.
We combine data from both the Arweave Gateway and the Arweave peers directly, perform transactions validation, store and
index them in a dedicated database.
The Warp Gateway consists of three main tasks:
-
The Sync Arweave Peers Task - this task is responsible for loading information about currently active peers and ranking them by the amount of synced blocks and response times.
-
The Sync Transactions Task - this task is responsible for loading and indexing SmartWeave interaction transactions from the newly mined blocks
-
Confirm Interactions Task - the most complicated task, responsible for confirming transactions.
- It takes the first PARALLEL_REQUESTS, non-confirmed transactions with block height lower than (current_network_height - MIN_CONFIRMATIONS).
- For each set of the selected 'interactionsToCheck' transactions it makes TX_CONFIRMATION_SUCCESSFUL_ROUNDS query rounds (to randomly selected at each round peers).
- Only if we get TX_CONFIRMATION_SUCCESSFUL_ROUNDS within TX_CONFIRMATION_MAX_ROUNDS AND response for the given transaction is the same for all the successful rounds - the "confirmation" info for a given transaction is updated in the database.
- Load Evolved Contract Sources Task - loads all the contracts sources which have been indicated in 'evolve' interaction and are not yet indexed in gateway.
Tested for block height range: 0 - 831901. Benchmarks source can be found here and here.
Contract | Project | Interactions | Arweave GW | Warp GW | Warp GW(cache) |
---|---|---|---|---|---|
LkfzZvdl_vfjRXZOPjnov18cGnnK3aDKj0qSQCgkCX8 | Kyve | 281129 | 3h 7min 39s | 1 min 27s | 8s |
l6S4oMyzw_rggjt4yt4LrnRmggHQ2CdM1hna2MK4o_c | Kyve | 194326 | 2h 32min 54s | 46s | 6s |
B1SRLyFzWJjeA0ywW41Qu1j7ZpBLHsXSSrWLrT3ebd8 | Kyve | 93098 | 38min 22s | 16s | 3s |
cETTyJQYxJLVQ6nC3VxzsZf1x2-6TW2LFkGZa91gUWc | Koi | 22403 | 3min 15s | 2s | 1s |
QA7AIFVx1KBBmzC7WUNhJbDsHlSJArUT0jWrhZMZPS8 | Koi | 12228 | 1min 25s | 1s | 582ms |
SJ3l7474UHh3Dw6dWVT1bzsJ-8JvOewtGoDdOecWIZo | Pianity | 10924 | 1min 13s | 1s | 670ms |
NwaSMGCdz6Yu5vNjlMtCNBmfEkjYfT-dfYkbQQDGn5s | Koi | 10137 | 1min 5s | 1s | 491ms |
Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY | RedStone - loot | 9821 | 1min 1s | 834ms | 358ms |
-8A6RexFkpfWwuyVO98wzSFZh0d6VJuI-buTJvlwOJQ | ArDrive PST | 4786 | 20s | 710ms | 190ms |
usjm4PCxUd5mtaon7zc97-dt-3qf67yPyqgzLnLqk5A | Verto PST | 1041 | 3s | 459ms | 142ms |
List of first 15 contracts with the highest number of corrupted transactions. Corrupted transactions are transactions
that are not part of any block - but they are still returned by the Arweave GQL endpoint. This creates a huge problem
when evaluating the state - especially in the case of PSTs and transfer
interactions.
You can read more about these
issues here,
here
and here.
Contract | Corrupted |
---|---|
Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY | 25 |
-8A6RexFkpfWwuyVO98wzSFZh0d6VJuI-buTJvlwOJQ | 9 |
1TFZeEewEgUpqT5i2dsZSIRKJq3h1C7ZVi-gE8G-W6U | 8 |
usjm4PCxUd5mtaon7zc97-dt-3qf67yPyqgzLnLqk5A | 8 |
wXotIq_fSPvYWR12h6IS-kfD18Y5jkr4UPPp15e0wo0 | 7 |
NwaSMGCdz6Yu5vNjlMtCNBmfEkjYfT-dfYkbQQDGn5s | 6 |
3NQQlLebRbq32Rtdxf_xaiWCJcQZNmoLvmOxLclGRcU | 5 |
LppT1p3wri4FCKzW5buohsjWxpJHC58_rgIO-rYTMB8 | 5 |
QA7AIFVx1KBBmzC7WUNhJbDsHlSJArUT0jWrhZMZPS8 | 4 |
gTT7_-8nrB1HKyJrPoUiku8-5aL3_wva5BJEn8sUCl4 | 4 |
SJ3l7474UHh3Dw6dWVT1bzsJ-8JvOewtGoDdOecWIZo | 4 |
K9Lb5WzRHxGyQqZVKL-ckBcnwtEouEBOlphKNmLhHtY | 4 |
k-3vYDcwrusBtnouFXh6QlRvwfH57lLvnG8jnf_q1EM | 4 |
hFhD2XG0LNKQTo4WCMfhFbD2ssxMn1vOyzwZt0qiJI4 | 3 |
o-qJmQ4B0d6TnyA_awjhiBdiq0O4Vt_dNWU3pTnhTu8 | 2 |
Warp Gateway is currently publicly available under https://gateway.redstone.finance/. The contracts scanner web app is available under https://scanner.redstone.tools/.
We've prepared a dedicated plugin for the Warp Contracts SDK that allows to easily connect and load interactions from the Warp Gateway. More details here.
yarn install
yarn build
To run gateway in production:
-
Create a file
.secrets/.env
with aDB_URL
property with PostgreSQL connections string, eg:DB_URL=postgresql://<user>:<password>@<db-host>:<db-port>/<database-name>
-
Run gateway with
yarn start:prod
.
You can pass theenv_path
param with path to the.env
file, eg:
yarn start:prod --env_path .secrets/.env
To run gateway locally:
-
Create a file
.secrets/local.env
with aDB_URL
property with PostgreSQL connections string, eg:DB_URL=postgresql://<user>:<password>@<db-host>:<db-port>/<database-name>
-
Run gateway with
yarn start:local
.
- build the docker image - run script
docker-build.sh
- run the docker image - run script
docker-run.sh
/gateway/contracts?page=<page>
- returns the list of all the currently registered contracts.
Result is ordered by [last_interaction_height DESC, count(interaction) DESC]
Parameters:
contractType
[optional] - filter records by contract type. If not set, all types of contracts are returned.pst
other
sourceType
[optional] - filter records by contract source type. If not set, all contracts are returned.application/wasm
application/javascript
page
[optional] - page, e.g.:gateway/contracts?page=3
. If not set, first page is returned by default.limit
[optional] - amount of interactions per single page
Response:
{
"paging": {
// paging info object
"total": 4145,
// total number of elements
"limit": 100,
// elements per page
"items": 100,
// elements on current page
"page": 1,
// page number
"pages": 42
// total amount of pages
},
"contracts": [
// array of "contract" objects
{
"confirmed": "10753",
// amount of confirmed interactions
"contract": "NwaSMGCdz6Yu5vNjlMtCNBmfEkjYfT-dfYkbQQDGn5s",
// contract tx id
"owner": "DymtKHHegWz-HWrNnOL12Rxz_7dLrY2R3wVwTFE8788",
// contract owner address
"contract_type": "pst",
// contract type (pst/other)
"pst_ticker": "KOI",
// pst contract ticker,
"pst_name": "KOI",
// pst contract name
"src_content_type": "application/javascript",
// contract source type
"src_wasm_lang": null,
// wasm contract source language
"interactions": "10763",
// total amount of interactions
"last_interaction_height": 835229,
// last synced interaction block height
"corrupted": "6",
// amount of corrupted interactions
"total": "4145"
// total amount of contracts (helper)
}
(...)
]
}
Examples:
curl https://gateway.redstone.finance/gateway/contracts
curl https://gateway.redstone.finance/gateway/contracts?page=6
/gateway/interactions?contractId=<contract_id>&confirmationStatus=<confirmation_status>&from=<block_height_from>&to=<block_height_to>&page=<page>
- returns a list of a given contract interactions, ordered by
[block_height ASC]
.
Parameters:
contractId
- tx id of the contract to load the interactions forpage
[optional] - page, e.g.:gateway/interactions?page=3
. If not set, first page is returned by default.limit
[optional] - amount of interactions per single pageconfirmatinStatus
[optional], e.g.:gateway/interactions?confirmationStatus=corrupted
. If not set, loads all the contract interactions.confimed
- loads only theconfirmed
contract interactionscorrupted
- loads only thecorrupted
contract interactionsnot_corrupted
- loads bothconfirmed
andnot processed
interactions
totalCount
[optional], e.g.:gateway/interactions?totalCount=true
. If set totrue
endpoint returns interactions' count for each of confirmation statuses (confirmed | corrupted | forked | not_processed)source
[optional] - loads contract interactions based on their origin. If not set, interactions from all sources will be loadedarweave
redstone-sequencer
Response:
{
// paging info object
"paging": {
"total": 4145,
// total number of elements
"limit": 100,
// elements per page
"items": 100,
// elements on current page
"page": 1,
// page number
"pages": 42
// total amount of pages
},
// summary of interactions count based on confirmation status of each interaction
"total": {
"confirmations": {
"confirmed: "
56
",
"corrupted": "2",
"not_processed": "1",
"forked": "0"
}
},
// array of "interaction" objects
"interactions": [
{
"confirmations": "58244,58244,58244",
// amount of confirmations returned from each confirming peer
"confirming_peers": "94.130.135.178,165.227.34.27,23.88.65.188",
// peers who confirmed this interaction
"interaction": {
// a full interactions object, compatible with SmartWeave protocol GQLNodeInterface
"id": "RaW_er2EidTFdaNXm4D3qRxBZWHN1ofGiYxta-QjDEY",
"fee": {
"winston": "1473297"
},
"tags": [
{
"name": "App-Name",
"value": "SmartWeaveAction"
},
{
"name": "App-Version",
"value": "0.3.0"
},
{
"name": "Contract",
"value": "Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY"
},
{
"name": "Input",
"value": "{\"function\":\"generate\"}"
},
{
"name": "Signing-Client",
"value": "ArConnect"
},
{
"name": "Signing-Client-Version",
"value": "0.3.5"
}
],
"block": {
"id": "0ISp2BqDA9nE5eljZw8jcbJvzJrb9a0Yw4tYBM93v3Y1TGSJIzBIqjwfEQ7-jy1H",
"height": 769067,
"timestamp": 1631571122
},
"owner": {
"address": "I-5rWUehEv-MjdK9gFw09RxfSLQX9DIHxG614Wf8qo0"
},
"parent": null,
"quantity": {
"winston": "0"
},
"recipient": ""
},
"status": "confirmed"
// confirmation status of this interaction (not_processed | confirmed | corrupted | forked)
}
]
}
Examples:
https://gateway.redstone.finance/gateway/interactions?contractId=Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY
- loads all contract interactionshttps://gateway.redstone.finance/gateway/interactions?contractId=Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY&page=2
- loads all contract interactions, shows 2nd. page
https://gateway.redstone.finance/gateway/interactions?contractId=Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY&page=2&confirmationStatus=confirmed
- loads only confirmed contract interactions, shows 2nd. page
https://gateway.redstone.finance/gateway/interactions?contractId=Daj-MNSnH55TDfxqC7v4eq0lKzVIwh98srUaWqyuZtY&confirmationStatus=confirmed&from=820000&to=831901
- loads only confirmed contract interaction from block height 820000 to block height 831901
- Support for interactions with multiple "Contract"/"Input" tags - warp-contracts#17
- Support for internal writes - warp-contracts#15
- An option to define the observed contracts - so that each project could run its own instance dedicated to its contracts.
- Scale the infrastructure, create backup instances, etc.
- A form of decentralization with disputes/voting on challenged responses.
- Custom network of Arweave nodes, that will listen on and index only SmartWeave interaction transactions (probably with the help of the customized Vartex gateway).
- Even better protection against forks - analyzing blocks history
- As the amount of data being transferred is rather huge - consider moving from json to protobuf?