owlprotocol / merkledb

Bridge web2.0 databases with decentralized smart contracts

merkledb

Projects

• cborlib-contracts: Solidity library to encode/decode CBOR data.
• firebase-to-orbitdb: Firebase Cloud Function used a plugin to reflect a Firestore database to OrbitDB document store by listening to write updates.
• merkledb-contracts: Solidity smart contract to sotre merkledb root and IPFS hash.
• orbitdb-to-merkledb: Listens to OrbitDB changes to generate a merkle tree, publish it to IPFS, and publish its root on-chain.

Local Development

Install

git clone [email protected]:owlprotocol/merkledb.git
pnpm install

merkledb's Issues

OrbitDB Parse as CBOR

Description

Convert document entries into CBOR encoding in preparation to be inserted into a Merkle tree as leaves.

Links

https://github.com/paroga/cbor-js

IPFS Trees MerkleMap

Description

Combination of merkle tree and map.

POC Dummy Data (`examples`)

Put together scripts to load dummy data into Firebase.

Firebase Cloud Function

Description

Deploy a Firebase Cloud Function to handle write updates.

Tasks

• Setup Firebase Cloud Function project
• Write function to log updates to database

Links

https://www.notion.so/owlprotocol/Firebase-to-OrbitDB-e18eb79613fa413a9fb6fb3baca3767c

Scalable Merkle Tree Library

Look into fixing the Merk and Node-Merk library/bindings.

CBOR Search Algorithm

Implement linear search algorithm

Initial Firebase Project

Description

Setup a Firebase Firestore project and populate it with mock data.

Tasks

• Generate API Key
• Create Firestore Table
• Populate Table with Mock Data (eg. name, address)

Links

https://www.notion.so/owlprotocol/Firebase-to-OrbitDB-e18eb79613fa413a9fb6fb3baca3767c

CBOR Testing Suite

Test with dummy data, and all supported types

OrbitDB CBOR leaves snapshot / listener update

Description

Convert the OrbitDB Docstore to Merkle Tree.

Tasks

• snapshot(): Snapshot the full docstore to a Merkle Tree. Used on initial setup
• Listener: Update Merkle tree with insert/delete on orbitdb updates
• Compute merkle root
• Unit Test: OrbitDB Docstore Snapshot
• Unit Test: OrbitDB Docstore onUpdate mutate merkle tree

Links

https://github.com/nomic-io/node-merk

Contract Changes

Ownable -> Access Control
Seperate DatabaseIPFS into its own setter

OrbitDB Setup & Listen for Changes

Description

Configure project environment variables to listen to an OrbitDB Docstore and listen for changes.

Taks

• Setup envvars
• Create Docstore with mock data
• Listen for database changes

Links

https://github.com/orbitdb/orbit-db/blob/main/GUIDE.md#replicating-a-database

CBOR Optimizations / Gas Efficiency Updates

Optimize gas usage in CBOR decoding

This calls for several gas optimizations in CBOR decoding:

• Implement more efficient byte copying: axic/density#1

• Implement decodeItem(uint itemNumber): scanning items up until itemNumber, then returning the bytes at that index.

  • This will require adding a limit parameter to scanIndefiniteItems. The function will stop scanning once it reaches a specific number of items and return that item's start and ending bytes.

• Implement shortCircuitDecode(bytes encodedMapping, bytes string key): Linear search to begin decoding a mapping object, but at each iteration checks if it's key matches key. If they do, short-circuit and return.

MerkleDB Unit Test

Description

Unit test the merkleProofData function that generates a proof.

Tasks

• Add https://github.com/nomic-io/node-merk as a dev dependency
• Generate a simple merkle tree of height 8 that encodes [0x00...0xFF] (0-255) using node-merk
• Publish the root to MerkleDB
• Generate a valid proof that a number is in the tree
• Generate an invalid proof that a number is in the tree

`MerkleDB` & `IMerkleDB` Solidity

Description

Create IMerkleDB interface and the implementation contract MerkleDB.

Smart Contract

From Notion

State

• bytes32 merkleRoot: Root of the merkle tree. Used to generate proofs that certain rows are present in the database.
• bytes32 merkleTreeIPFS : IPFS Hash pointing to Merkle Tree used to generate proofs
• bytes32 databaseIPFS : IPFS Hash pointing to OrbitDB document store for a more indexable table formatted version of the data.

Events

• DatabaseUpdate(bytes32 merkleRoot, bytes32 merkleTreeIPFS, bytes32 databaseIPFS): Event emitted whenever MerkleDB smart contract is updated.

Interface

Read

• public view merkleProofData(bytes memory bytes, bytes32[] path) returns (bool) : Prove that some CBOR data is in the merkle tree. Use keccak256(bytes memory) to compute the leaf node’s hash and then recursively hash keccak(a+b) of sibling nodes to get the root hash. Return true if this is indeed the stored merkleRoot.

Write

• public updateMerkleDB(bytes32 _merkleRoot, bytes32 _merkleTreeIPFS, bytes32 _databaseIPFS) onlyOwner : Update the MerkleDB data. Limited privileged access to contract owner. Emits a DatabaseUpdate event. Updated data comes from the following sources
  • merkleRoot : db.rootHash()
    • Checkpoints: https://rocksdb.org/blog/2015/11/10/use-checkpoints-for-efficient-snapshots.html
  • merkleTreeIPFS : Computed CID of state.db file data https://github.com/nomic-io/node-merk#usage after https://github.com/nomic-io/node-merk#dbflushsync
  • databaseIPFS: Mostly static according to https://github.com/orbitdb/orbit-db/blob/main/GUIDE.md#manifest, can be updated if the OrbitDB is reset (eg. changed access control)

Links

https://docs.openzeppelin.com/contracts/4.x/api/utils#MerkleProof

ipfs-trees project

IPFS Trees

Create an immutable data-structure library using IPFS as a storage layer and enabling efficient querying. This can avoid the need for 3rd party indexing, effectively solving the data availability problem for the merkledb's off-chain data bridge solution.

Tasks

• Use dag-json IPLD codec for simple huma-friendly encoding
• Tree: A simple binary tree abstraction build on top of IPFS.
• TreeReverse: A binary tree with where leaf nodes have parent references (used for MerkleTree parent traversal).
• RBTree: A red-black black tree for sorting by key.
• MerkleTree: A Merkle tree for compacting a set as a merkle root.

Links

• https://en.wikipedia.org/wiki/Red%E2%80%93black_tree
• https://en.wikipedia.org/wiki/Merkle_tree
• https://github.com/ipld/js-dag-json

OrbitDB REST Server

https://www.notion.so/owlprotocol/OrbitDB-REST-460f2aa39ee944dab99f6600508737f7

Fix IPFS types

## IPFSMerkleTree

IPFSMerkleTree

Using the existing Tree & Search Tree primites, create a decentralized lazy-loading immutable Merkle tree meant to be stored on IPFS.

Requirements

The data structure has the following requirements:

• O(log(n)) traversal insert/delete
• O(log(n)) node changes on insert/delete
• O(log(n)) merkle proof parent traversal
• O(log(n)) leaf access for proof generation
• Efficiently Immutable: inserts/deletes are done by generating copies of changed nodes only
• Lazy loading: data is fetched only when necessary and cached (but released if node garbage collected)

Architecture

The IPFSMerkleTree relies on several primitives to efficiently store the tree:

• IPFSMerkleTreeIndex: Node for encode/decode logic, hash logic.
• BalancedTree: A balanced tree to store nodes. Must NOT have any circular reference to parent. Use breath-first search with a null node for insertions.
• BinarySearchTree: hash => CID<ParentCID> A binary search tree to index leaf nodes and enable parent tree recursion.

Tasks

• Map<K,V> interface
• IPFSMap<string, CID> abstraction built on top of binary tree
• Review IPFSTree insertion/deletion (especially duplicate handling)
• MerkleTreeIndex data model
• MerkleTree insertion
• MerkleTree proof

Dockerize & Heroku Deploy

Description

Dockerize listener process for deployment on Heroku.

Recursive CBOR decoding

Allow CBOR to decode any data structures, including nested. This will require scanning recursively to mark structure start/end points.

`CBORByteUtils.sol` Utility library

Library to convert from bytes into Solidity.

Supported conversions:

• bytes : uint64 (MajorType: 0)
• bytes : int64 (MajorType: 1)
• bytes : string (MajorType: 2) (this may or may not be possible)
• bytes : string (MajorType: 3)
• bytes: uint256 (MajorType: 6 - specifically BigNum)
• bytes: true/false (MajorType: 7)

MerkleDB Publish

Description

Final step of the Merkle Tree generation consists of publishing the data onto decentralized networks.

Tasks

• Publish state.db binary to IPFS
• Publish merkleRoot and merkleIPFS to MerkleDB smart contract #1

Delete Varibles from MerkleDB contract

IPFS Trees Promise Optimization

Promise Optimization

Optimize promise calls to avoid blocking thread on concurrent calls.
Code logic should strive to await promises lazily, meaning only when data is needed.

This issue is NOT priority.

Draw.io Technical Architecture Diagram

Description

Create a technical diagram of the architecture of MerkleDB.

Tasks

• Firebase => OrbitDB bridge
• OrbitDB Listener => Merkle Tree + On-chain Root publish
• MerkleProof + CBORLib => Trigger smart contract with proof

Hacky Merkle Tree

Create hacky merkle tree using 2 Binary search trees for getParent getLeft getRight.

CBOR Decoding Library

Implement functions:

enum MajorType {
    UnsignedInteger,
    NegativeInteger,
    ByteString,
    TextString,
    Array,
    Map,
    TagOfNumber,
    SimpleFloat
}

struct CBORValue {
    MajorType majorType;
    bytes value;
}

decodeCBOR(bytes memory buf) returns (CBORValue[] memory)

Decodes a CBOR-encoded buffer into an array of decoded major types and values.