inx-chronicle's Introduction

inx-chronicle

Chronicle is the permanode (sometimes also called indexer or scanner) for the IOTA-based networks. It connects to a Hornet via the IOTA Node Extension (INX) interface. Through the INX interface, Chronicle listens to all blocks in the Tangle that are referenced by a milestone and stores them in a MongoDB database.

Documentation

The documentation for Chronicle can be found in our wiki, the supported APIs are documented here.

Development

This project uses conventional commits to automatically generate a CHANGELOG.md which signals breaking changes. The changelog can be created using the following command (requires the conventional-changelog-cli package on NPM):

conventional-changelog -p conventionalcommits -i CHANGELOG.md -s

Docker deployment configuration of credentials through environment variables

Docker compose will automatically load credentials for different services from a .env file that must either be located in the same directory as the docker-compose.yml file, or specified using the --env-file flag. You therefore must create such a file before you do a docker compose up. An example .env file could look like this:

MONGODB_USERNAME=root
MONGODB_PASSWORD=root
INFLUXDB_USERNAME=root
INFLUXDB_PASSWORD=password
JWT_PASSWORD=password
JWT_SALT=saltines
HORNET_CONFIG_PATH=config_testnet.json

inx-chronicle's People

Contributors

Stargazers

Watchers

inx-chronicle's Issues

Create MongoDB indices for basic queries

Define subset of indexer plugin that we want to implement

Configure API address using config and CLI flag

Right now everything is hardcoded, which we should change after #41 lands.

Provide `health` and `info` endpoints

Expose Prometheus API over INX

Similar to what inx-mqtt is doing: https://github.com/gohornet/inx-mqtt

Create a working `docker-compose.yml` file

Improve `delay` to not use 100% of the CPU

Currently, delay keeps on pushing new events onto the queue.

Write test cases for `u64` to `U64` conversion

Return actual types instead of `bson::Document`s from database queries

Write our own DTOs.

Then we can use the following things:

serde_bytes
Internal tagging for variants

Tracking Issue - `Analytics`

There are several statistics that can help us better understand how a network is used. The following analytic endpoints should be supported by Chronicle:

The activity analytics should be implemented for set time intervals of 24h, 7d, 30d, and 90d.

In addition to these required statistics @Alexandcoats has some more brainstorming ideas below, which we could tackle afterwards.

Here is what the endpoints will look like:

Activity

Ledger Status

/ledger ?ledger_index (total count unspent)
- /storage-deposit (byte cost at ledger index)
- /nfts (count of the number of unspent nft ouputs)
- /native-tokens (count of the number of unspent foundry outputs)

Properly document the `dto` module

Right now we just set the complete module to #![allow(missing_docs)]. Maybe we can copy over the comments from bee-message.

Write test cases for sync record logic

We also need to find a better way to properly clear the database after (potentially unsuccessful) test runs.

Remove `DocExt` trait

Needs DTOs for analytics and transaction history.

Provide additional Prometheus metrics

We can carry over the ones from this PR: #61

Automatically upload releases as GitHub packages to GHCR

Find a better way to configure Hornet correctly

Currently, we manually have to keep docker/config.hornet.json up to date.

Setup proper proxying of routes through INX

Store message blob for redundancy

Clean up exports of `runtime` module

Decouple `collector` from `inx` via a trait

Use an actor system like Actix or Backstage?

Separate the API from the database logic

Currently, the API endpoints access the MongoDb database directly.

Consider supporting multiple networks at once

Move Chrysalis DTOs from `bee-rest-api` to `bee-message`

Manually set actor names

We should probably set the actor names manually so that they will appear less verbose in the tokio console

Get rid of the `bin`/`lib` divide

Potential problem updating `MessageMetadata`?

https://github.com/Alexandcoats/inx-chronicle/blob/ff39e54886ad148f35353b77bd3ef88021a662b5/bin/inx-chronicle/src/collector/mod.rs#L207-L209

What happens when we want to update the MessageMetadata for a Message that is not in the database yet?

Store additional information in archive files

File format version (if we ever want to change something in the future)
Network identifier (which is not part of Message)

In a separate file: Checksum?

Investigate performance of the `Syncer`

Provide JWT authentication for the API

Store and retrieve/data from MongoDB

Set `_id` correctly for data models

Decouple INX from API via features

Decoupling the API from INX would allow us to better scale Chronicle horizontally.

Configure CORS in config file

Store raw messages in separate collections

Now that we have our own types and don't rely on deserializing raw bytes anymore, we should store the raw messages in a separate collection to increase performance.

Consider using conventional commit messages

This offers a way to do automatic changelogs. Here is more information:

https://www.conventionalcommits.org/

Sync missed milestones

Description

Chronicle should request and store milestones from the past that are available.

Requirements

On startup, an actor should be started which will send message requests to the InxWorker.
The range of milestones to request should be configurable.
If messages are not available because they have been pruned, that milestone can be skipped.
We should spawn multiple syncers and request in chunks, to avoid the situation where large swaths are unavailable and block requesting newer milestones.

Open questions

Should we request in forward or backward order?
This actor may need to be work-limited, so that it does not slow down the system.

Are you planning to do it yourself in a pull request?

Don't block the event loop when waiting to reconnect to INX

This is currently happening because of the call to tokio::time::sleep when handling the InxListenerError::InxConnection in main.rs.

Find a way to work around `u64` limitations of MongoDB

This could maybe be achieved by just storing the DTOs of the objects.

Provide proper deserialization for own types

Check if `NodeConfiguration` matches `Status` in the database

When you connect INX you can query information about the node. We should store that in the database and check that it matches on each new connection to INX.