trueblocks / trueblocks-core

The main repository for the TrueBlocks system

If you have an existing installation, complete applicable Migrations before proceeding.

---

TrueBlocks / Unchained Index

Table of Contents

• Introduction
• Installing
• Command line
• Chifra Serve
• Troubleshooting
• The unchained index
• Docker version
• Documentation
• Linting
• Contributing
• Contact
• List of Contributors

Introduction

TrueBlocks improves access to blockchain data for any EVM-compatible chain (particularly Ethereum mainnet) while remaining entirely local.

Features:

• chifra init and chifra scrape which builds the Unchained Index, an index of address appearances that provides lightning-fast access to transactional histories,

• An optional binary cache, which speeds up queries to the RPC by orders of magnitude,

• Enhanced command-line options enabling much better access to chain data for data scientists and analysts. For example, easily extract all logs produced by a smart contract or view all ERC-20 holdings for an account, etc.,

• Advanced tools for producing reconciled "bank statements" and accounting export for any token including ETH.

• An infinite number of other things restricted only by your imagination.

Installing

Please see the installation instructions on our website.

Searching account histories

While optional, you most likely want to use the Unchained Index to search account histories. To do so, get the index.

Account explorer

You may use the command line, of course, to access data, but you may also wish to run an API server:

chifra daemon

Use curl to pull data or use it to drive our "pre-beta" Account Explorer. See installing the explorer. The API provides the identical tools and options as the command line and it documented here.

Generate gRPC files (developers only)

To regenerate gRPC files, you have to install protobuf tools:

brew install protobuf # Mac
go install google.golang.org/protobuf/cmd/[email protected]
go install google.golang.org/grpc/cmd/[email protected]

Command line

The TrueBlocks command-line tool is called chifra. This gives you access to all the other tools:

chifra --help

Get more help on any sub-command with chifra <cmd> --help. Full documentation is available on our website.

Chifra Serve

One of chifra's command line tools is called serve. This tool provides a simple API mimicing the command line. It is intended to be used only for single user enviroments.

To start the server, run:

chifra server

Chifra was built for the command line, a fact we purposefully take advantage of to ensure continued operation on small machines. As such, this tool is not intended to serve multiple end users in a cloud-based server environment. This is by design. Be forewarned.

Get more help on any sub-command with chifra <cmd> --help. Full documentation is available on our website.

Getting data

Let's look at the first subcommand, called status.

chifra status

If you get a bunch of data, congratulations, your installation is working. Try this command which shows every 10th block between the first and the 100,000th:

chifra blocks 0-100000:10

You should see a long stream of data. Kill the display with Control+C.

See the entire list of chifra commands with chifra --help.

Troubleshooting

Depending on your setup, you may get the following error message when you run some chifra commands:

  Warning: A request to your Ethereum node (http://localhost:8545)
  resulted in the following error [Could not connect to server].
  Specify a valid rpcProvider by editing $rootPath/trueblocks.toml.

Edit the file as instructed. You may find helpful answers on our FAQ.

See our blog for a lot of useful articles on getting started and using TrueBlocks.

If you continue to have trouble, join our discord discussion

The unchained index

The primary data structure produced by TrueBlocks is an index of address appearances called the Unchained Index. This index provides very quick access to transaction histories for any address.

You may either build the entire index from scratch (requires an EVM-compatible tracing/archive node) or you may download a snapshot of the index build from there.

This process is described in the article Indexing Addresses.

Docker version

Our official docker version is in a separate repo. Please see that repo for more information on running with Docker.

Documentation

The TrueBlocks documentation repo builds the TrueBlocks website. See our website for the best available documentation.

Linting

Our build process requires the code you submit to be linted.

In order to that, you must install the GoLang linters. See this page for more information.

To install the primary linter (called golangci-lint), run this command:

curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin v1.53.3

Verify the installation with golangci-lint --version. You should see something like this:

golangci-lint has version 1.50.1 built from <commit> on <date>

Next, run golangci-lint linters. Your system should have at least the default list to properly lint your submission.

> golangci-lint linters

You are encouraged to use additional linters. If you do, and you think they're useful, please suggest that we add it to our build process.

Contributing

A chart showing the number of stars on our repo over time.

We love contributors. Please see information about our workflow before proceeding.

1. Fork this repository into your own repo.
2. Create a branch: git checkout -b <branch_name>.
3. Make changes to your local branch and commit them to your forked repo: git commit -m '<commit_message>'
4. Push back to the original branch: git push origin TrueBlocks/trueblocks-core
5. Create the pull request.

Contact

If you have questions, comments, or complaints, please join the discussion on our discord server which is linked from our website.

List of Contributors

Thanks to the following people who have contributed to this project:

• @tjayrush
• @dszlachta
• @wildmolasses
• @MysticRyuujin
• @MattDodsonEnglish
• @crodnun

trueblocks-core's Issues

getBlock --latest should pull from etherscan as well as the local node

Or better yet, --latest should be an option on getBlock. Further, it should report how many blocks it's behind. It should also report latest block from the hard drive cache and the miniBlock cache

From https://github.com/Great-Hill-Corporation/ethslurp/issues/148

Chain reorgs

I’m not sure where to ask this. I’m running a node, but I’m not mining. Periodically, I retreive block and transaction data using RPC. It then save that data locally. I hear all this talk about re-orgs, so I would expect that once in a while, some of the data I pull would be “re-orged"

I should save data for X blocks. And then spin and re grab data fresh to check for re-orgs. This will be a big deal if the blocks are being re-orged, but I am not reflecting that.

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/146

miniBlock database does not lock the file

If the freshener is running and I try to run speedTest, speedTest hangs because it cannot find the end of file. Freshener should fail if the miniBlock database is in memory, which it is in teh case of loading it.

Should handle default function and constructor better

I don't really handle the constructor or the default function correctly in the monitor code. Look for this code:

        if (!func.name.empty())
            functions[functions.getCount()] = func;

If you remove this, it will show a new line in the grabABI tests, but they are not correct.

abis / sources

I should, given an address, pull its abi file, its source code (if found) and convert them both to something more easily readable as soon as I touch them as opposed to current which sort of just does this on an as-needed basis.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/142

Starting fullBlock scan from non-zero should use binary search

Look for this comment: // TODO: This should be a binary search not a scan. This is why it appears to wait

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/136

makeClass, should be intereactive

makeClass, when it encounters a new class, should ask for a list of fields and a directory and create the file itself

whenBlock does not work with block number zero (0)

I use '0' to determine if the input is okay, but that makes it not work with block '0'.

It would be interesting to have block '0' because we want to know what happened then.

miniBlock read is not concurrency protected

utillib: All assignments of "" to a string should be changed to EMPTY (but EMPTY should be renamed)

The EMPTY #define is just ugly. I was going to remove it, but I don't think I should as if I do and I use "" instead of EMPTY there will be that many more empty strings laying around. Probably better to have a single system-wide empty string.

Maybe change the name to emptyStr or something

From https://github.com/Great-Hill-Corporation/ethslurp/issues/131

Generated switch statements from makeClass

If a smart contract uses leading underbars (_), then all items in the switch statement check that first character. This is slower than it needs to be.

ethNames: should allow me to do this: ethslurp `ethname whatever -a`

-a should mean return address only

Upper case / lower case in ABI files names

In ~/.ethslurp/abis there are .abi files and .json files. Some of the .json are uppercase/lowercase, all of the .abis are lowercase only filenames. Weird. Should fix it.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/141

miniBlock database, if restarted, gets messed up with wrong indexes.

blockScrape and miniBlocks can both start twice. Bad idea. If either is running, neither should run

logBloom should show 0x0 if empty

not '0' which is what it does now for getBlock 50004. Plus it's both int he block and the receipt.t

From https://github.com/Great-Hill-Corporation/ethslurp/issues/123

account cache for parsed data

The local cache of an account gets duplicate events if the program is stopped and started (and/or) the 'startDate' is forcing a duplicate record.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/139

blockScrape --deep --check

Currently deep check only checks that each block has the correct list of transactions hashes. It should actually decend into the transactions and then into the receipts and then into the logs to check all data fields from QuickBlocks to the node.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/140

grabABI: class files should be stored in a folder called classDefinitions

Plus the path in the config file should not be there. If the program is run in a folder, it should look for classDefinitions and if it can't find it, don't run, and then use the pwd for the folder in the config file

This is for makeClass

From https://github.com/Great-Hill-Corporation/ethslurp/issues/135

grabABI: should accept multiple addresses

and merge the resulting ABI together to create a single parseLib

Serialize in derived classes does not check parent's return value

For example, in CFund, which is derived from CTransaciton, the Serialize function ignores the return value (it's a void return anyway). If the parent does not serialize (because of writeDeleted == false, for example) then we do serialize the child. This may already be fixed.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/145 in old ethslurp repo

BLOCK_CACHE should not be hard coded

makeClass config files

Storing the configuration files in a shared folder (~/.makeClass) doesn't work because classes conflict with each other with different implementations. Store the config files in the folders where the code arrives. Two reasons. Won't have to store the path in the file, and two generating source will be next to the code it generates. Right now, generating source is invisible.

getBalance should run from --file to pull various accounts

Plus, it should default to latest block if none is presented

getBalance should allow for -e to report in ether and -d to report in dollars

Separate monitored accounts into individual binary files and merge them on the fly

These data files must be hidden from the user. Also--if the user changes the list of accounts that goes into the hash cache, then the hash cache must either be re-written entirely (if there are new accounts) or cleaned up, if there are accounts that have been removed.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/143

Bloom filters should report better

1. All bloom filters are 256 bytes long, big num do not report leading zeros
2. Start with '0x'

from https://github.com/Great-Hill-Corporation/ethslurp/issues/132

Contract creation is ignored in monitors

Contract creation is not noticed or recorded yet, but it should be pretty easy.

I simply have to look for contract address in the transaction's receipt in ofInterest function

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/144

ethslurp: Non-field display field tokens

Like [{NOW}], [{DATE:MONTH}] [{ADDR}] [{HEADER}][{RECORDS}] etc.

Also--these should be able to accept arbitrary internal text such as [Generated on {NOW}.] Perhaps this requires a new class called something like 'Generic'

Now that Slurp is a full fledged class it can take many additional fields for summary:

name
nTransactions
toValue
avgValue
largestValue
smallestValue
totalGasUsed
averageGasPrice
etc.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/23

utillib: JSON parser is slow

Search for // TODO: THIS PER DISPLAY LOOKUP IS SLOW

From https://github.com/Great-Hill-Corporation/ethslurp/issues/125

On first time parsing an ABI

Store the parsed variable in the input data field. It's easy to generate the input data from the data itself and an ABI, it's hard to generate the parsed data because for every transaction, we have to search through all signatures. Also-- we should add counter to function and sort the function in the ABI by most frequent first thereby lessening search time.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/127

whenBlock - show nBlocks / second in most recent X blocks

The reason whenBlocks exsits is so people can see how close they are to being synced. It would be interesting to show the speed of the most recent X blocks synced per second. When the hack blocks are occuring, it's super slow. When the regular sync is happening, it's much faster.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/149

makeClass CToml1 file should be a CToml

This is the old 'pairFile' problem

utillib: toJSON doesn't work at all for non-transaction blocks

Don't really know what this means.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/121

utillib: --h (as opposed to -h) does not fail as it should

grabABI: Autogenerated code with the same function name will overwrite other class definition files

In monitors especially if two smart contracts have similar function names.

https://github.com/Great-Hill-Corporation/ethslurp/issues/124

lastBlock is incorrect

If lastBlock is first picked up from config.toml file do not add one. If lastBlock comes from the lastBlock file or the cacheDisplay function do add a one. Simple, but annoying. Maybe better to always just write lastBlock - 1 to the file or return it from the function

ethslurp: folder is the wrong name

It should be $(HOME)/.quickBlocks

Why is gasLimit part of the block data structure, but it's not written to binary?

From https://github.com/Great-Hill-Corporation/ethslurp/issues/133

Transactions may fail without sending "out of gas". Internal calls does not use gas in parent (maybe)

An internal call in this transaction (0xd20a95a9f051d6961ca766a9b2d0f18b6afdf6d8465931a3f70f2c4b1c023688) ran out of gas, but this transaction did not run out of gas, so I don't identify them as inError. The gas!=gasUsed, therefore, I don't trace it.

ethslurp: In transactions.cpp parseParams needs to be removed

From https://github.com/Great-Hill-Corporation/ethslurp/issues/130

This code was a very early way to parse functions but it is totally wrong. It has all been subsumed by the monitors, but does have the benefit of not requiring a separate piece of software. In other words remove it, but don't lose the ability to parse a totally arbitrary input data the way ethslurp does. But also don't require a full-fledged monitor program.

In other words--input parsing should happen fully through the ABI if that is the only thing present, but through the parse library if that is what is needed for speed or flexibility. Note that monitors use this code to fill in the [{FUNCTION}] tag.

grabABI: Locally stored ABI is invalid JSON

When I write the ABI file to ~/.ethslurp it isn't stored correctly. I think it's simply a matter of surrounding with '[' and ']', but make sure it's correct.

miniBlocks is not concurrency protected

If you run monitors watch.sh freshen x and then run testall, it will mess up the miniblocks mostly because ti takes so long to read miniblocks, and if it gets rewritten in between, it doesn't know where the end of the file is. Should protect against allowing write if miniblock is in the process of reading.

utillib: JSON parser is not recursive

The code for fromJSON is no recursive. It works for Ethereum because none of the classes are recursive, but it doesn't work in general.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/126

transaction class in non etherscan mode does not have blockNum or cum

Or any other duplicated stuff. It also does not have pointers to pBlokc, nor does receipt have pointers to pTrans

This shows up as empty timestamp, empty cumultageGas, etc in toJSON 50004

See getBlock tests

From https://github.com/Great-Hill-Corporation/ethslurp/issues/122

makeClass should use a toml file for configuration.

utillib: Remove all SFString code and depend only on the C++ string class plus utilities for ReverseReplace, etc.

Document minimum required setup for the node

At the very least, you cannot use warp, and you must have tracing = on. For 100% accuracy, you must also have pruning = archive.

Some code (sha3 and getLatestBlock) depends on running node, but doesn't fail gracefully

There are many others probably, but do I want to check on every request?

From https://github.com/Great-Hill-Corporation/ethslurp/issues/129

Accumultating caches is too slow

I need a much faster, and yet still decentrlaized way to accumulate per-contract caches