thenewboston-blockchain / thenewboston-node Goto Github PK

Resolves this TODO:
# TODO(dmu) CRITICAL: Validate account balances

Prioritized list of features:

• [DONE] Multitype block support (related tasks: #130 )
• [IN PROGRESS] Register new node on the network (related tasks: #162 , ...)
• Nodes discovery
• [IN PROGRESS] Node start up -> #226
• [IN PROGRESS] Blockchain download (including downloading latest changes) on node start
• [IN PROGRESS] Node REST API
• Node Websockets API
• [IN PROGRESS] PV schedule
• CV block confirmation process
• Disregarding cheating PV
• Coins lock
• Points support
• Boosts support
• Username registration
• Governor Application
• Governance
• Budget
• Vote purchase
• Voting

TODO:

• For writes: write to a temporary file, close it, move to where it should be
• For appends: copy file to a temporary file, append it, move to where it should be
• FileSystemStorage can perfectly incapsulate this logic

You are testing thenewboston_node.business_logic.blockchain.base.BlockchainBase by testing thenewboston_node.business_logic.blockchain.memory_blockchain.MemoryBlockchain (we already have some tests implemented this way)

There are some backup methods that are overridden in thenewboston_node.business_logic.blockchain.memory_blockchain.MemoryBlockchain . These methods should have separate tests.

DEPENDS ON:

• #272

Documentation:
https://docs.google.com/document/d/1kU0qxuJQC8PybSEZ8OHOUldx8b7TFJg9c79TjZrsVHQ/edit?usp=sharing

See how it is implemented for account root file. Consider all nested keys

Expose online API documentation: https://www.django-rest-framework.org/topics/documenting-your-api/

Deploy Node somewhere so frontend developers can is it as a source of online documentation and for testing

DEPENDS ON:

• #223

TODO:

• Add POST /api/v1/signed-change-requests/ end-point
• If the receiving node is PV:
  • add block generated from signed change request to the blockchain
  • validate that signed change request has correct fee transaction (at least the published amount)
    • [MOVED OUT] #472
    • [MOVED OUT] #473
  • validate the signed change request itself
  • validate the signed change request against current state of the blockchain
  • if any of the validations do not pass respond to the calling node with the an error
  • [MOVED OUT] notify confirmation validators about the new block -> #113
• If the receiving node is a regular node or CV: validate the signed change request and send it to current PV

In PR https://github.com/thenewboston-developers/thenewboston-node/pull/154/files#diff-21b702ae2b51eab764bcb0017f28589b8e8eabd55f4653818d711c4eb6839922 you can see new validators are used. Refactor the rest of the models to use them. Add new validators as needed

At least take all tests from thenewboston_node/business_logic/tests/test_memory_blockchain and add them for file blockchain as well. This minimum, but I guess we need more tests for file blockchain especially related to various optimizations.

Partial blockchains - blockchains starting from block number > 0

Resolves this TODO:
# TODO(dmu) CRITICAL: Support partial blockchains

As per discussion https://docs.google.com/document/d/1_6OCmqL0OK5UiWJROe-EbRYAQa9s-iuIIQOdv_Vd5Bs/edit?disco=AAAAMUjT3hY rename "account balance" to "account state" (containing "account balance" as amount of money left on the account and "account lock" as special value to verify the validity of the account balance)

Such transfer requests should not pass validation

Instead of FileBlockchain managing directories where blocks and account root file are stored let's delegate it to storages. So we pass directory to storage on creation and forget about it on FileBlockchain level. Like this:

block_directory = '...'
block_storage = PathOptimizedFileSystemStorage(path=block_directory)

account_root_files_directory = '...'
account_root_files_storage = PathOptimizedFileSystemStorage(path=account_root_files_directory)

This will also enhance security. Right now we pass absolute file paths to storages so you can overwrite something by a mistake. After we pass base directory to the storage it will operate only on files within this directory.

TODO:

• End-point: GET /api/v1/nodes/
• Pagination (see /api/v1/blockchain-states-meta/ implementation)

Blockchain structures have various hexadecimal values: account numbers, node identifiers, signatures, hashes. All of them are represented in memory and JSON as hexadecimal strings. We probably can live with it, but when it come to persisting to disk we would like store them as true binary.

Implement this feature by overriding from_messagepack() and to_messagepack() methods. This should reduce size of the persisted data (before compression about 2 times, implement a unittest proving that).

Please, be aware that messagepack has some automatic utf-8 decoding with string. Make sure you do not mess up with it.

Make generate_blockchain.py a management command since we decided to stay with Django
Put it to core application for now.

Similar to how it is implemented in alpha version

Resolves this TODO:
# TODO(dmu) CRITICAL: Validate balance values

Resolves this TODO:
# TODO(dmu) CRITICAL: Validate last_block_number and last_block_identifiers point to correct blocks

TODO:

• Use a single file.lock for account root file and add block placed in the blockchain root (base_directory kwarg)
• Lock on enter to add_block() and unlock on exit -> raise BlockchainLockedError if trying to operate on locked blockchain
• Lock on enter to add_account_root_file() and unlock on exit -> raise BlockchainLockedError if trying to operate on locked blockchain
• All other submethods must just check that the blockchain is locked -> raise BlockchainUnlockedError if the blockchain is not locked as expected
• Implement as method decorators

DEPENDS ON:

• #252
• #106

Operations to be implemented:

• Retrieve account balance -> #102
• Retrieve account lock -> #102
• Add transfer request -> #106
• Retrieve blockchain download URL -> #107
• List known nodes -> #108
• List account transactions -> #109
• Create block (called by Primary Validator) -> #110
• Retrieve node identifier (we use it to check online-status as well) -> #111
• Register new known node -> #112

• [DONE] README.rst
• [DONE] INSTALL.rst and Poetry
• [DONE] Django and docker-compose.yml
• [DONE] pre-commit
• [DONE] yapf
• [DONE] flake8
• [DONE] unittests
• [DONE] Layer (dynaconf-like) settings
• [DONE] Dockerfile
• [DONE] Github workflows

TODO:

• Django Model to store a node in the database: node_identifier, addresses (known addresses including port number - IPs, hostnames - usually one, but let's keep it possible to have multiple), current role (choices: regular node - rn, confirmation validator cv, primary validator - pv), current_role_updated_at, online_at (last known online status datetime), created_at, updated_at
• End-point: PUT /api/v1/nodes/ {"node_identifier": "...", "addresses": ["IP or hostname:port"], "signature": "..."}
• Validation must be implemented in thenewboston_node/business_logic/network/base.py in API framework agnostic way (and let's have thenewboston_node/business_logic/network/orm_network.py)
• Signature must be validated against normalized POST body after removal of signature key
• If Node already known then it is updated (do not forget about last known online status datetime) otherwise it is created

The aim it to avoid parallel support of documentation, so as soon as you make the change you can regenerate the documentation.

Requirements:

• It must be a CLI tool
• Output is ReStructured mark up

TODO:

• [DONE] Support multiple block types -> done here: #188
• [DONE] Properly read docstrings for attribute (also allow empty docstrings)
• [DONE] Generate example values with factories (or memory blockchain instead of hardcoding)
• [DONE] Add documentation generation to gitflow actions
• [DONE] Expand object to actual types
• [DONE] Rename account root files to blockchain state
• [DONE] Properly describe datetime type
• [DONE] Review comments of this task for missing TODOs
• [DONE] Add doc generation to linter
• [DONE] Implicit docstrings for dataclasses
• [DONE] Make sure documentation update produces same hashes / signatures, etc (take care to constantify timestamps)
• [DONE] Fix having "node_type": "PRIMARY_VALIDATOR"
• [DONE] Add example value for each field
• [DONE] Correctly represent optionality for serialized fields
• [MOVED OUT] Reread the docs and if there are improvement required -> #218
• [MOVED OUT] Review and update docstrings for dataclass attributes -> #218

DEPENDS ON:

• #110

TODO:

• After adding a new block to its blockchain PV must notify each CV about the new block
• Must be a celery task (implement similar to send_block_confirmation pattern)
• Task instance is created per confirmation validator (so we can do independent retries)
• POST /api/v1/blocks/ to for notification -> #110
• Rework: #434 (what to be reworked?)

Bug Description
While analyzing the GitHub repos, I found that there are some secret keys being left open in the wild. It might lead to security threat for thenewboston.

Steps to Reproduce

1. Go to the links mentioned below
2. See the secret keys

Links and Secrets

• https://github.com/thenewboston-developers/thenewboston-node/blob/c6a812343bc42ad02614def469d8bad46ea3982d/thenewboston_node/project/settings/templates/.env

THENEWBOSTON_NODE_SECRET_KEY=90oqi[f[ohfipuuapugrp83yq09=40=oasdngkj

• thenewboston-blockchain/Website-API@89129f8#diff-f7a16228131ae92457c22657bb33458207a7578b8cdb6ff796153916fa67a22d

SECRET_KEY = 'g#$0(*8%8af27k7-e!ll^!-4yxomcx8ljv_o&_z*zhvi)f8&e7'

• https://github.com/thenewboston-developers/thenewboston-node/blob/854abbf8814729eeca9973396f1ef6a61fed7a3f/thenewboston_node/project/settings/templates/settings.unittests.py

SECRET_KEY = '90oqi[f[ohfipuuapugrp83yq09=40=oasdngkj'

Expected behavior
Secrets should not have been left publicly

Actual behavior
Secret keys are being leaked

Account Number
362019f00e72fbe002497c667ab0d4fe4f4ac1ff8785724d0a69d39fe957fead

TODO:

• [DONE] Adopt already implemented API to the current beta naming conventions -> #206
• [DONE] Implement a CLI tool to convert alpha format account root file into blockchain genesis state -> #207
• [DONE] Include blockchain genesis state into built container
• [MOVED OUT] Remove exposed 80 and defaut nginx welcome page from image -> substitute with own index page that adds link to downloadable blockchain -> #221
• [MOVED OUT] Take care nginx does not bloat log files -> #216
• [MOVED OUT] Serve blockchain format documentation from node in RST and HTML formats -> #208 , #217

Cover at least these cases if they are not already covered:

1. Each method coverage
2. All edge cases
3. All compression type choice on save
4. Decompression for each type of compression

TODO:

• GET /api/v1/accounts/{id}/transactions/
• Pagination
• Ordering asc/desc chronologically
• If node does not have complete blockchain (all blocks till number 0) and transaction beyond known blocks are requested raise NotImplemented error (later we will have to implement sync initialization or otherwise redirection of client to get the entire list of transaction from somewhere else, for example from current PV)
• Use serializer based on transaction dataclass
• Add a TODO about a need of caching implementation

Resolve issues related to the conversation below:

Dmitry Mugtasimov  6 hours ago
def test_missing_transaction():
    """Block with missing transaction should be invalid"""
    block = {
        'account_number': '0cdd4ba04456ca169baca3d66eace869520c62fe84421329086e03d91a68acdb',
        'message': {
            'balance_key': '0cdd4ba04456ca169baca3d66eace869520c62fe84421329086e03d91a68acdb',
            'txs': [
                {
                    'amount': 1,
                    'recipient': '5e12967707909e62b2bb2036c209085a784fabbc3deccefee70052b6181c8ed8'
                },
                {
                    'amount': 4,
                    'fee': BANK,
                    'recipient': 'ad1f8845c6a1abb6011a2a434a079a087c460657aad54329a84b406dce8bf314'
                }
            ]
        },
        'signature': 'cfcba759125dfbaaefa627c4a41db4e5052705875d01228e5e280e13d403483fff495157b53192a6c1032ee815429f4979c0b4beb10d259fae3692123cf01f0d'
    }
    is_valid, account_balance = is_block_valid(block=block)
    assert not is_valid
this test passes because of signature validation fails, not because PV fee transaction is missing. Do we need to validate completeness of transactions: existence of 2 fee transaction and the main money transfer transaction?

Bucky:computer:  5 hours ago
Yep we do. Blocks should contain both fees in addition to the main money Tx. We can rename 'fee': BANK to 'fee': NODE as well since we no longer have “Banks”. So:
fee: PV
fee: NODE
We also have the edge case where the node itself is currently the PV for the network. In this case they are still able to receive blocks like any other node, however instead of forwarding that block to another PV it will just process it itself. Maybe still including both fees in the block with the same recipient is the easiest solution? It would make it clear from a data point of view, however we do also have logic in place to ensure that users can not send multiple Txs to the same recipient within the same block. Any ideas on how we could structure that?

Review the code and unittests and suggest unittest code coverage improvements. Add the suggestions as comments to this task. Once you do it will we will create substasks and prioritized them for implementation.

DEPENDS ON:

• #415

TODO:

• [DONE] This task is required just to implement PV side of PV to CV integration (so we have an API end-point on CV side to call to)
• [DONE] Implement POST /api/v1/blocks/ . Respond HTTP202 on success
• [DONE] Create model PendingBlock: uuid (primary key), block (JSONField - JSONB), block_number (indexed), block_hash (indexed)
• [DONE] If block with the same block_number and block_hash has already been added do nothing otherwise add the block to the database
• [MOVED OUT] Start celery task to process the block: temporary implementation that just sends confirmation to all known nodes on the network (see #432 ) -> #434
• [MOVED OUT] #461

self.storage = PathOptimizedFileSystemStorage()

->

self.account_root_files_storage = PathOptimizedFileSystemStorage()
self.blocks_storage = PathOptimizedFileSystemStorage()

This way we can apply different depth and caching policy