trustwallet / blockatlas

Clean and lightweight cross-chain transaction API

Block Atlas by Trust Wallet

THIS REPO IS NO LONGER MAINTAINED

---

BlockAtlas is a clean explorer API and transaction observer for cryptocurrencies.

BlockAtlas connects to nodes or explorer APIs of the supported coins and maps transaction data, account transaction history into a generic, easy to work with JSON format.

The observer API watches the chain for new transactions and generates notifications by guids.

Supported Coins

Block Atlas supports more than 25 blockchains: Bitcoin, Ethereum, Binance Chain etc, The full feature matrix is here.

Architecture

Blockatlas allows to:

• Get information about transactions, tokens, staking details, collectibles for supported coins.
• Subscribe for price notifications via Rabbit MQ

Platform API is independent service and can work with the specific blockchain only (like Bitcoin, Ethereum, etc)

Notifications:

• Subscriber Producer - Create new blockatlas.SubscriptionEvent [Not implemented at Atlas, write it on your own]

• Subscriber - Get subscriptions from queue, set them to the DB

• Parser - Parse the block, convert block to the transactions batch, send to queue

• Notifier - Check each transaction for having the same address as stored at DB, if so - send tx data and id to the next queue

• Notifier Consumer - Notify the user [Not implemented at Atlas, write it on your own]

New Subscriptions --(Rabbit MQ)--> Subscriber --> DB
                                                   |
                      Parser  --(Rabbit MQ)--> Notifier --(Rabbit MQ)--> Notifier Consumer --> User

The whole flow is not available at Atlas repo. We will have integration tests with it. Also there will be examples of all instances soon.

Setup

Prerequisite

• Go Toolchain versions 1.14+

  Depends on what type of Blockatlas service you would like to run will also be needed.

• Postgres to store user subscriptions and latest parsed block number

• Rabbit MQ to pass subscriptions and send transaction notifications

Quick Start

Get source code

Download source to GOPATH

go get -u github.com/trustwallet/blockatlas
cd $(go env GOPATH)/src/github.com/trustwallet/blockatlas

Build and run

Read configuration info

# Start Platform API server at port 8420 with the path to the config.yml ./
go build -o api-bin cmd/api/main.go && ./api-bin -p 8420

# Start parser with the path to the config.yml ./ 
go build -o parser-bin cmd/parser/main.go && ./parser-bin

# Start notifier with the path to the config.yml ./ 
go build -o notifier-bin cmd/notifier/main.go && ./notifier-bin

# Start subscriber with the path to the config.yml ./ 
go build -o subscriber-bin cmd/subscriber/main.go && ./subscriber-bin

make command

Build and start all services:

make go-build
make start

Build and start individual service:

make go-build-api
make start

Docker

Build and run all services:

docker-compose build
docker-compose up

Build and run individual service:

docker-compose build api
docker-compose start api

Configuration

When any of Block Atlas services started they look up inside default configuration. Most coins offering public RPC/explorer APIs are enabled, thus Block Atlas can be started and used right away, no additional configuration needed. By default starting any of the services will enable all platforms

To run a specific service only by passing environmental variable, e.g: ATLAS_PLATFORM=ethereum :

ATLAS_PLATFORM=ethereum go run cmd/api/main.go

ATLAS_PLATFORM=ethereum binance bitcoin go run cmd/api/main.go # for multiple platforms

or change in config file

# Single
platform: [ethereum]
# Multiple 
platform: [ethereum, binance, bitcoin]

This way you can one platform per binary, for scalability and sustainability.

To enable use of private endpoint:

nimiq:
  api: http://localhost:8648

It works the same for worker - you can run all observer at 1 binary or 30 coins per 30 binaries

Environment

The rest gets loaded from environment variables. Every config option is available under the ATLAS_ prefix. Nested keys are joined via _.

Example:

ATLAS_NIMIQ_API=http://localhost:8648

Tests

Unit tests

make test

Mocked tests

End-to-end tests with calls to external APIs has great value, but they are not suitable for regular CI verification, beacuse any external reason could break the tests.

# Start API server with mocked config, at port 8437 ./ 
go build -o api-bin cmd/api/main.go && ./api-bin -p 8437 -c configmock.yml

Therefore mocked API-level tests are used, whereby external APIs are replaced by mocks.

• External mocks are implemented as a simple, own, golang mockserver. It listens locally, and returns responses to specific API paths, taken from json data files.
• There is a file where API paths and corresponding data files are listed.
• Tests invoke into blockatlas through public APIs only, and are executed using newman (Postman cli -- make newman-mocked).
• Product code, and even test code should not be aware whether it runs with mocks or the real external endpoints.
• See Makefile for targets with 'mock'; platform can be started locally with mocks using make start-platform-api-mock.
• The newman tests can be executed with unmocked external APIs as well, but verifications may fail, because some APIs return variable responses. Unmocked tests are not intended for regular CI execution, but as ad-hoc development tests.
• General steps for creating new mocked tests: replace endpoint to localhost:3347, observe incoming calls (visible in mockserver's output), obtain real response from external API (with exact same parameters), place response in a file, add path + file to data file list. Restart mock, and verify that blockatlas provides correct output. Also, add verifications of results to the tests.

Docs

Swagger API docs provided at path /swagger/index.html

or you can install go-swagger and render it locally (macOS example)

Install:

brew tap go-swagger/go-swagger
brew install go-swagger

Render:

swagger serve docs/swagger.yaml

Updating Docs

• After creating a new route, add comments to your API source code, See Declarative Comments Format.

• Run $ make go-gen-docs in root folder.

blockatlas's Issues

Remove asset from BasicTx

• Remove asset property from BasicTx
• Limit platform/binance to BNB coin

ERC-20 support

Implement generic ERC-20 swap and transfer support

Return transactions list in the wrapper

Instead of returning transactions as list, can we do, in which docs will contain transactions list and this should be used across blockatlas.

{
    "docs": [{
    }],
    "status": true
}

in this case we can extend to support pagination and error handling easier

Add nonce value for ethereum/wanchain based blockchains

let's call it sequence: Int64

Ethereum support

Azure Pipelines CI

Pipeline stages

• Compile project
• Run Unit tests
• Start server with default config
• Run integration test (go run test/main.go) on temp server

Support account creation transaction

https://stellar.expert/explorer/public/account/GBQMF67PYZWCDTMSQPJXPD6SD72IKUCK5POMDS4LK33D337JCOWJPLTT

https://blockatlas.trustwalletapp.com/v1/stellar/GBQMF67PYZWCDTMSQPJXPD6SD72IKUCK5POMDS4LK33D337JCOWJPLTT

Add block: Int value to each transaction in the response

Aion support

Integration test: Check coin type

Check coin field in integration tests

Ethereum Fee

Ethereum shows wrong gas value
Correct: Fee = gasUsed * gasPrice

Make sure transaction status is correct

• API returns status: "completed" by default
• Only Tezos + Ethereum handle errors so far
• Check for all other chains where transactions can fail

Write deploy docs

Documentation on how to deploy in README

Platform Unit Tests

• Add unit tests per platform
• Cover each possible transaction type
• Check if data was deserialized correctly from RPC/API
• assert(parse(api_response) == expected_model)

Why?

Integration Tests only test connections and the actual requests, but don't verify the response content

Progress

• Binance DEX
• Nimiq
• Ripple
• Stellar
• Tezos
• Ethereum
• Aion

GoDoc

Document entire project with GoDoc

[Tezos] value not returned

http://blockatlas-trust.herokuapp.com/v1/tezos/tz1WDujRWCYjLBDfZieXW6insg5EUbg1rCRK

metadata: {
   value: ""
}

value is empty

Response Standards

This PR intend to establish minimum requirements to HTTP response. Please feel free to add a comment or suggestion. Once we finalize requirements requirements two PR must be made:

• Update README file with finalized requirements
• Implement requirements for each endpoint

Endpoint Response Payload Format
All endpoints return data in JSON format with the results of your query under data if the call is successful. Type of data can be array or hash-map, thses two only used in current implementation. If we will have endpoint returning single bit of information we need to consider type for data as number(integer) or string, example: {"data": "Coin status is active"}, price for some token {"data": 135.4}

A Status object is always included for both successful calls and failures when possible.
Any details about errors encountered can be found under the error_message and response code status under response_code.

200 Successful

{
  "total":  Number,
  "data": [] || {},
  "status": {
    "response_code": 200,
    "error_message": "",
  }
}

400 Bad Request

{
  "status": {
    "response_code": 400,
    "error_message": "The server could not process the request, likely due to an invalid argument.",
  }
}

401 Unauthorized

{
  "status": {
    "response_code": 401,
    "error_message": "Your API Key was not supplied or is invalid",
  }
}

429 Too Many Requests

{
  "status": {
    "response_code": 429,
    "error_message": "You have exceeded your API rate limit, please provide API key to keep going",
  }
}

500 Internal Server Error

{
  "status": {
    "response_code": 500,
    "error_message": "An unexpected server issue was encountered.",
  }
}

Heroku Deploy

https://tw-blockatlas.herokuapp.com

Tron support

/tron

@kolya182 can you provide endpoints from the official docs? and trongrid endpoint? it's ok if it's not working yet

Add coin: slip44/Int to each transaction model in response

coin: 60 - ethereum example

VeChain support

Return consistent result for the transactions

/v1/ethereum/0x89e0cdf0a548ecb432d00a22dfd188f45641478d?contract=0x4b4787aace23eb70da046bcc0ccfe28d4e4fb899

/v1/ethereum/0x89e0cdf0a548ecb432d00a22dfd188f45641478d

@terorie I think it would be better to return a consistent result for the transactions endpoint. Ignore previous discussion to have different format if contract or token is passed

Add support for collectibles

List of supported blockchains

Create list of blockchains with config options + how to enable them

Tezos support

Transactions format

Base

{
   "id": "12345678"
   "from": "123",
   "to": "123",
   "fee": "1234"
   "coin": 60,
   "block": 1
}

Transfer:

{
   "type": "transfer",
   "metadata: [
       "name": "Viktor Coin",
       "symbol": "VIK",
       "decimals": 18,
       "value": "12312312"
    ]
}

Token Transfer

{
   "type": "token_transfer",
   "metadata: [
       "name": "Viktor Coin",
       "symbol": "VIK",
       "token_id" : "0x123",
       "decimals": 18,
       "value": "12312312",
       "from": "123",
       "to": "123",
    ]
}

Token Transfer

{
   "type": "contract_token_transfer",
   "metadata: [
       "token": {
           "type": "token_transfer",
           "metadata: [
               "name": "Viktor Coin",
               "symbol": "VIK",
               "token_id" : "0x123",
               "decimals": 18,
               "value": "12312312",
               "from": "123",
               "to": "123",
           ]
        },
        "coin": {
               "type": "contract_call",
                "metadata": []
         }
    ]
}

Tron Token Transfer

{
   "type": "token_transfer",
   "metadata: [
       "name": "Bittorent",
       "symbol": "BTT",
       "token_id" : "1002000",
       "decimals": 8,
       "value": "12312312"
    ]
}

Collectible Transfer

{
   "type": "collectible_transfer",
   "metadata: [
       "name": "Viktor Kittie",
       "contract" : "0x123",
       "image_url": "https://google.com/img.png"
    ]
}

Token Swap

{
   "type": "token_swap"
   "metadata: [
       "input" : [
           "token_id": "0x123" - this property is optional
           "symbol": "BNB",
           "value": "123",
           "decimals": 8
       ],
       "output": [
           "token_id": "0x123" - this property is optional
           "symbol": "BTT",
           "value": "123",
           "decimals": 8
       ]
    ]
}

Contract Call

{
    "type": "contract_call",
    "metadata": [
    ]
}

Any Action

{
    "type": "any_action",
    "metadata": [
       "coin": 60,  
       "title": "Place Order",
       "key":  "place_order",     
       "token_id": "0x123" - this property is optional

       "name": "Viktor Coin",
       "symbol": "VIK",
       "decimals": 18,
       "value": "12312312"
    ]
}

Keys
place_order - Placer Order
cancel_order - Cancel order
issue_token - Issue Token
burn_token - Burn Token
mint_token - Mint Token

will continue... Keys mostly used to provide localized version on the clients by key

types: transfer, token_transfer, collectible_transfer, token_swap, contract_call and many more

Status

enum Status {
    completed
    pending
    errror
}

Theta support

Docs: https://github.com/thetatoken/theta-mainnet-integration-guide/blob/master/docs/api.md#api-reference

Looking for the indexer

https://explorer.thetatoken.org:9000/api/accounttx/{address}?type=2&pageNumber=1&limitNumber=50&isEqualType=true

Support token transactions for Binance Chain

/v1/binance/tbnb1xwalxpaes9r0z0fqdy70j3kz6aayetegjkcrgw?contract=NC1-279

Transfer asset example: https://testnet-explorer.binance.org/tx/39278E2E87D24932796A14D8CBD9C0C82A6391B8380911A813059DA8075FD07A

[Ripple] Transactions endpoint should return newest to oldest

http://blockatlas.trustwalletapp.com/v1/ripple/rEb8TK3gBgk5auZkwc6sHnwrGVJH8DuaLh

ThunderCore support

@vikmeup Can you provide a test address for ThunderCore?

Binance Integration Test: No transactions

time="2019-03-23T13:03:13+01:00" level=info msg="Testing endpoint" @platform=binance
time="2019-03-23T13:03:14+01:00" level=warning msg="No transactions" @platform=binance
time="2019-03-23T13:03:14+01:00" level=info msg="Endpoint works" @platform=binance
time="2019-03-23T13:03:14+01:00" level=info msg="Endpoint tested" @platform=binance fields.time=592.104107ms

Binance DEX integration

Related: #14 (Generic DEX model)

Describe trades on the Binance DEX chain

Lint

Run project through lint

Integration Tests

Test that runs the app and calls every platform endpoint

Request Standards to BlockAtlas

This PR intend to establish minimum requirements to HTTP request. Please feel free to add a comment or suggestion. Once we finalize requirements requirements two PR must be made:

• Make appropriate changes to the code
• Update README file with finalized requirements

Each HTTP request to BlockAtlas (BA) and response from BA must contain next headers
Accept: application/json
Accept-Encoding: gzip

[Ripple] returned wrong value

http://blockatlas.trustwalletapp.com/v1/ripple/rEb8TK3gBgk5auZkwc6sHnwrGVJH8DuaLh

{
id: "E89CFA4C75F02A456880CC98BCF8762E69AB285452A27B8F143EEF70E73D9626",
coin: 144,
from: "rP9GDNw8je4xez1vsmVtrYohHGscfMCT3P",
to: "rEb8TK3gBgk5auZkwc6sHnwrGVJH8DuaLh",
fee: "7646000000",
date: 1509614141,
type: "transfer",
block: 33933674,
metadata: {
value: "100000000000000"
}
}

https://xrpscan.com/tx/E89CFA4C75F02A456880CC98BCF8762E69AB285452A27B8F143EEF70E73D9626

I beelive value and fee contains extra zeros

Wrong fee on nimiq network

v1/nimiq/NQ86%202H8F%20YGU5%20RM77%20QSN9%20LYLH%20C56A%20CYYR%200MLA

id: "e1a911583a1678a2338922efadbdf6710136f0c558f9f2d4a8b1e87aa05c05db",
coin: 242,
from: "NQ95 QLQF E5T9 0VR8 GGAQ D1TK 4CG1 C77K NVR2",
to: "NQ86 2H8F YGU5 RM77 QSN9 LYLH C56A CYYR 0MLA",
fee: "0",
date: 1554573613,
type: "transfer",
block: 512316,
status: "completed",
sequence: 0,
metadata: {
value: "4284350514"
}

fee should not be zero

Cannot unmarshal

level=error msg="Tron: Failed to decode API response" error="json: cannot unmarshal object into Go struct field Page.error of type string"
2019-04-22T09:41:03.031492+00:00 app[web.1]: time="2019-04-22T09:41:03Z" level=error msg="Tron: Failed to get transactions for TKLnCNY5EsLNCvCXQTCn1dtqvc6vHhJUyJ" error="connection to servers failed"

Code Review and 1.0 release

Placeholder for 1.0 release and code/API review

X-Forwarded-For trust settings

• Don't trust forwarded headers by default
• Setting to let Blockatlas use them
• Ignore them in Heroku

Fetch only 25 first transactions

Only newest tx are important

Switch from legacy to new API

Placeholder issue for the future

Add go-import page

I'm using the custom import path trustwallet.com/blockatlas.

Please deploy the following HTML page so that go get works.
It will tell Go where to find the documentation and source code.
Regular browsers will get redirected to GitHub when visiting.

URLs:

• https://trustwallet.com/blockatlas?go-get=1
• https://trustwallet.com/blockatlas

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
    <meta name="go-import" content="trustwallet.com/blockatlas git https://github.com/trustwallet/blockatlas">
    <meta name="go-source" content="trustwallet.com/blockatlas https://github.com/trustwallet/blockatlas https://github.com/trustwallet/blockatlas/tree/master{/dir} https://github.com/trustwallet/blockatlas/blob/master{/dir}/{file}#L{line}">
    <meta http-equiv="refresh" content="0; url=https://github.com/trustwallet/blockatlas">
</head>
<body>
    Checkout the <a href="https://github.com/trustwallet/blockatlas">GitHub repo</a>!
</body>

Add DEX support

• Add support for Binance
• Add support for Stellar
• Add support for WAVES

Related #21

Integrate ICON

@kolya182 can you provide details for the indexer endpoint

Integration: Test only endpoints that are enabled (/v1/)

Simplify project structure

• Remove platform/x/source packages
• Enforce consistent naming scheme (Tx vs Transaction)

Docker Hub image

• Dockerfile
• trustwallet/blockatlas
• Continuous Deployment
• Gravatar image for TrustWallet Docker organization

Set GitHub description and tags

The repository description is empty right now, and I don't have permissions to modify it.

How about Clean and lightweight cross-chain transaction API. Backend of TrustWallet.?

Tags: crypto, blockchain, api, binance, trustwallet

Add Empty array response for endpoints

/tron, /icon, cosmos, theta, vechain

make empty response until implemented:

{
   total: 0,
   docs: [ ],
   status: "success"
}