cryptonomic / conseil

Query API and indexer for Tezos and other decentralized platforms.

Conseil

Conseil is an indexer and API for blockchains, currently focused on the Tezos, which allows you to seamlessly run complex queries against blockchain data. It is written in the functional style in the Scala programming language. This project forms an essential part of the Cryptonomic stack.

Conseil consists of two processes:

• Lorre, which can index the blockchain by downloading data from a blockchain node and storing it in a database.
• Conseil, which provides an API into the blockchain data indexed by Lorre.

Arronax provides a convenient user interface for querying Conseil data.

Information on using, running, configuring and developing Conseil can be found on the Conseil Wiki.

conseil's Issues

Conseil should allow filtering operations by kind

Lorre should be able to run as a service

Lorre is currently run periodically using either cron or some other scheduler. It should instead run in a persistent loop so it can be set up as a service.

Figure out how to run an alternate network

Reexamine the Tezos RPC API

Lorre should periodically purge old account data

As accounts are block contextual, account data can overwhelm the database. There should be a process to purge old account data from the database in a parametrized manner.

Create script for automatically deploying Conseil

Conseil should support contract origination

Add support in Lorre for additional 'op1' and 'op2' types for inner operations

Conseil should include unit tests for database operations

Conseil should support transaction operations

Conseil should support basic parameters for its GET routes

One should be able to get operations by block ID, blocks by level, accounts by delegate, etc.

Conseil should include logic for creating and injecting operations

Conseil should accept OPTIONS requests for CORS pre-flight checks

Arronax is not able to run Conseil API queries when Conseil is running remotely.

Conseil functionality should be available as a JavaScript library

Use Scala.js to create Conseil.js.

Expand Swagger spec for Conseil

Lorre should be resilient to forks

Lorre currently fails when a blockchain fork is encountered. While inserting the latest blocks into its database, Lorre checks for referential integrity through a foreign key which ensures a new record's predecessor hash already occurs in the blocks table. During a fork, the referential integrity does not hold and should therefore not be enforced. Furthermore, the predecessor blocks of the current head should be inserted until a known block is found.

Conseil should expose a route listing all available blockchain protocols

The Conseil REST API should expose a route called '/info' which returns metadata about the running Conseil instance. As a start, it should return all blockchains and platform supported by the given Conseil instance. This route can be used by client applications, e.g. Arronax, to deduce all available platforms.

This is a rough example of a return value of the route:

{
	"platforms": [

	{
	"platform": "tezos",
	"network": "zeronet"
	}

	]
}

The initial version of Nautilus should be up and running

Make code compatible with zeronet

All code in this repo was written against the Tezos alphanet. The zeronet rollout introduces breaking change in the Tezos RPC interface. Therefore, the code must be updated to work with the latest API spec.

Delegation functionality should be added to the Conseil spec

Manually deploy Conseil to 'cryptonomic1' and note steps down

Install Jenkins on 'nautilus' and create job for running Conseil deployment script

Create program for transferring data between Tezos nodes and Conseil database

Run 'beluga1' in sandbox mode

Conseil should allow searching for all operations associated with a specific account

Conseil should not hard code the location of the key store for SSL functionality

Lorre should parameterize the blockchain and network it runs for

At the moment, both Lorre and Conseil are capable of connecting to a single database to write and read data. As a consequence, they effectively only support one network at a time. This should be rectified such that Tezos zeronet and betanet can both be supported by Lorre and Conseil and multiple instances of the programs can run on the same codebase.

As part of this ticket, we should also validate the network parameter when the client makes an API call.

Install, use and examine EzTz

Conseil should support delegation operations

Conseil should support the parsing of contract scripts into its database

Currently, 'script' and 'program' fields from Tezos RPC results are being ignored. These should be parsed properly and included in the Conseil database.

Conseil unit tests for injecting operations should run against a mock node

Conseil should be integrated into TravisCI

Conseil should display associated accounts when displaying transactions

There should be a continously running instance of Conseil

Developers should be able to connect to https://conseil.cryptonomic.tech/.

Switch 'beluga1' to run an alternate network.

Conseil should support origination and transactions for smart contracts

Currently, Conseil only supports the origination of accounts without code or storage and transfers of funds between them. The logic should be extended to also support smart contracts.

Lorre should invalidate discarded blocks in the database when it detects forks

Every time Lorre detects a fork in the Tezos chain, it should invalidate discarded blocks using an 'invalidated_blocks' table.

Ensure Conseil CORS settings allow local access from Firefox

The existing CORS configuration in the Conseil codebase does not work locally with Firefox. The akka-http-cors library should be used additionally as a fix.

Conseil should support key generation

Conseil should record balances of all accounts

Due to an oversight, account balances are not being stored in the database nor returned by the Conseil API.

Understand how the Tezos internally stores its data

The Conseil API spec should be fully specified

Research Tezos node RPC API and eztz to create full Swagger spec of Conseil API

Conseil should run against a database

Instead of running against a Tezos node, Conseil should instead point to a database. Such a database must be created a continuously populated.

Increase default verbosity of Conseil and Lorre logging to stabilize production instances

Change Conseil logic to run against database

Lorre should run continously to refresh the Conseil database

Conseil should support write operations for zeronet

Capture all error reported by Tezos RPC API

Lorre currently ignores all reported errors. It should pick up and process them instead.

Tune down Lorre logging verbosity so its logs don't fill up host disks

Blocks without operations should still show up in Conseil queries