GithubHelp home page GithubHelp logo

pixelspark / catena Goto Github PK

View Code? Open in Web Editor NEW
302.0 20.0 23.0 20.45 MB

Catena is a distributed database based on a blockchain, accessible using SQL.

License: MIT License

Swift 80.90% JavaScript 3.45% CSS 2.16% HTML 0.11% Vue 13.23% Dockerfile 0.15%
blockchain sql fintech sqlite distributed-database ledger swift

catena's Introduction

Logo

Catena - SQL on a blockchain

Catena is a distributed database based on a blockchain, accessible using SQL. Catena timestamps database transactions (SQL) in a decentralized way between nodes that do not or cannot trust each other, while enforcing modification permissions ('grants') that were agreed upon earlier.

A Catena blockchain contains SQL transactions that, when executed in order, lead to the agreed-upon state of the database. The transactions are automatically replicated to, validated by, and replayed on participating clients. A Catena database can be connected to by client applications using the PostgreSQL wire protocol (pq).

Only SQL statements that modify data or structure are included in the blockchain. This is very similar to replication logs used by e.g. MySQL ('binlog').

Building

macOS

Catena builds on macOS. You need a recent version of XCode (>=8.3.2) on your system. Use the following commands to clone the Catena repository and build in debug configuration:

git clone https://github.com/pixelspark/catena.git catena
cd catena
swift build

It is also possible to generate an XCode project and build Catena from it:

swift package generate-xcodeproj

Linux

Building on Linux is fully supported. To build, first ensure Swift 4 is installed and is in the PATH. Then ensure clang and required libraries are present:

apt install clang build-essential libicu-dev libcurl4-openssl-dev openssl libssl-dev
git clone https://github.com/pixelspark/catena.git catena
cd catena
swift build

The above was tested on Ubuntu 16.04 as well as Debian Stretch with Swift 4.

Note: Due to the fact that there is no cross-platform WebSocket client implementation in Swift (the current implementation uses Starscream), outgoing peer connections are not supported for the Linux client. Incoming peer connections are possible and can be used by the client to 'talk back', so the client is functional regardless.

Docker

A docker image is available at the Docker Hub. To build the Docker image from source (useful if you don't have / can't install Swift locally, for instance):

git clone https://github.com/pixelspark/catena ./catena
cd ./catena
docker build -t pixelspark/catena .

Building the web client

Catena includes a web client, resources from the Resources/ folder. A built version of the web client is included. To build from scratch, first install the required build tools, then use gulp to compile and bundle files:

cd Resources/
npm install
gulp

Use gulp watch to have gulp recompile files on change.

Running

Natively

The following command starts Catena and initializes a new chain (replace 'debug' with 'release' after building a release version):

./.build/debug/Catena -p 8338 -m -i -s 'my seed string'

The -i switch tells Catena to initialize a chain (this deletes any persisted data, which is stored by default in catena.sqlite in the current directory). The -s switch provides Catena with a string that tells it which genesis block to accept. To enable block mining, add the '-m' command line switch.

To start another peer locally, use the following:

./.build/debug/Catena -p 8340 -s 'my seed string' -j ws://[email protected]:8338 -d peer2.sqlite

Note, the node URL in the command above can be copied from the output of the first node. Only one node is required in order to bootstrap (nodes will perform peer exchange).

Running with Docker

docker pull pixelspark/catena
docker run -p 8338:8338 -p 8339:8339 pixelspark/catena [ARGS]

Note: the port number on which Catena listens inside the container must be equal to the port number used outside the container (as Catena advertises it to peers).

Using Catena

Web client

Catena provides a web interface on port 8338 (default), which can be used for introspecting the blockchain and performing queries. It also provides a WebSocket service (on the same port) which is used for communication between peers. Provide the --no-web-client command-line option to disable the web client and associated API (note that the currently exposed API does not allow for any mutating actions).

Web client showing blocks Web client showing data

SQL interface

The (private) SQL interface is available on port 8339 (by default). If you set a different HTTP port (using the '-p' command line switch), the SQL interface will assume that port+1. You can connect to the SQL interface using the PostgreSQL command line client:

psql -h localhost -p 8334 -U random

Your username should be the public key (generated by Catena) and your password the private key. Catena will print the public and private key for the 'root' user when initializing a new chain (with the -i option) and will also print a psql command line for you to use to connect as root.

Permissions

By default, any user can execute a CREATE DATABASE statement; this creates a database, of which the invoker becomes the owner. The invoker can subsequently use GRANT and REVOKE statements to grant other users rights on the database.

More granular permissions can be granted by using template grants. These grant a user the permission to execute a certain parametrized query with self-chosen parameter values. The template query is hashed and stored in the grants table. Any query whose hash (once parameter values removed) matches that of a template granted to the user can be executed. Template grants can be most easily created through the web client.

The template grant can further restrict the set of allowable parameter values using an IF-statement:

IF ?amount > 0 THEN UPDATE balance SET balance = balance + ?amount WHERE iban = ?iban ELSE FAIL END;

FAQ

Is Catena a drop-in replacement for a regular SQL database?

No. The goal of Catena is to make it as easy as possible for developers and administrators that are used to working with SQL to adopt blockchain technology. Catena supports the PostgreSQL (pq) wire protocol to submit queries, which allows Catena to be used from many different languages (such as PHP, Go, C/C++). However, there are fundamental differences between Catena and 'regular' database systems:

  • Catena currently does not support many SQL features.
  • Catena's consistency model is very different from other databases. In particular, any changes you make are not immediately visible nor confirmed. Transactions may roll back at any time depending on which transactions are included in the 'winning' blockchain.
  • Catena will (in the future) check user privileges when changing or adding data, but can never prevent users from seeing all data (all users that are connected to a Catena blockchain can 'see' all transactions). Of course it is possible to set up a private chain.

Which SQL features are supported by Catena?

Catena supports a limited subset of SQL (Catena implements its own SQL parser to sanitize and canonicalize SQL queries). Currently, the following types of statements are supported:

  • CREATE TABLE foo (bar TEXT, baz INT);
  • INSERT INTO table (x, y, z) VALUES ('text', 1337);
  • SELECT DISTINCT *, x, y, 'value', 123 FROM table LEFT JOIN other ON x=y WHERE x=10;
  • DELETE FROM foo WHERE x=10;
  • UPDATE foo SET bar=baz WHERE bar=1;
  • DROP TABLE foo;

What kind of blockchain is implemented by Catena?

Catena uses a Blockchain based on SHA-256 hashes for proof of work, with configurable difficulty. Blocks contain transactions which contain SQL statements. Catena is written from scratch and is therefore completely different from Bitcoin, Ethereum etc.

How does a Catena node talk to other nodes?

Catena nodes expose an HTTP/WebSocket interface. A node connects to the WebSocket interface of all other nodes it knows about (initially specified from the command line) to fetch block information and exchange peers. In order for two nodes to be able to communicate, at least one must be able to accept incoming connections (i.e. not be behind NAT or firewall).

What is the consistency model for Catena?

SQL statements are grouped in transactions, which become part of a block. Once a block as been accepted in the blockchain and is succeeded by a sufficient number of newer blocks, the block has become an immutable part of the blockchain ledger.

As new blocks still run the risk of being 'replaced' by competing blocks that have been mined (which may or may not include a recent transaction), the most recent transactions run the risk of being rolled back.

How are changes to a Catena blockchain authenticated?

Transactions are required to be signed with a private key.

A transaction that modifies any table or row needs to be signed with a private key that has the privilege to modify that specific table or row. Privilege grants are stored in a special 'grants' table (which, in turn, can be modified by those that have a grant to modify that table).

To prevent replay of signed transactions, Catena stores a transaction counter for each public key, which is atomically incremented for every transaction that is executed. A transaction will not execute (again) if it has a lower transaction number than the latest number recorded in the blockchain. To successfully submit a transaction, the invoker must increment the counter by one for each transaction (or use a different key each time).

Where does the name come from?

Catena is Italian for 'chain'.

Can I run a private Catena chain?

Chains are identified by their genesis (first) block's hash. To create a private chain, use the '-s' option to specify a different starting seed.

MIT license

Copyright (c) 2017 Pixelspark, Tommy van der Vorst

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contributing

We welcome contributions of all kinds - from typo fixes to complete refactors and new features. Just be sure to contact us if you want to work on something big, to prevent double effort. You can help in the following ways:

  • Open an issue with suggestions for improvements
  • Submit a pull request (bug fix, new feature, improved documentation)

Note that before we can accept any new code to the repository, we need you to confirm in writing that your contribution is made available to us under the terms of the MIT license.

catena's People

Contributors

pixelspark avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

catena's Issues

Add optional condition to grant

Add a column to the grant table that (optionally) contains an expression that needs to evaluate to true in order for the grant to be available. The user column also still exists, but may be set to NULL to always match:

Examples:

  • kind=insert, user=NULL, condition=$invoker=X'somehash',table=sometable: equivalent to user=someHash
  • kind=create, user=NULL, condition=$invoker=$miner, table=NULL: allows miners to create a table
  • kind=insert, user=x, condition=$timestamp < y, table=z: allows inserts to a table only before a certain date/time (approximate)

Add a miner reward

Some ideas: have a $miner variable that contains the public address of a miner, then use that to send extra quota (through grants) using a query stored in 'flags' executed as first query for each block.

Compile on Linux again

The reason Catena currently doesn't compile on Linux is the WebSocket client library used (Starscream) which relies on CFNetwork. This should be replaced with something that uses (for instance) BlueSocket.

Implement proof-of-authority

Instead of PoW, allow a select set of keys to sign blocks. The genesis block decides on the mechanism used for signing blocks.

The list of keys that are allowed to sign blocks should be stored in a special table.

See also ethereum/EIPs#225

Add a quota system

Currency system for database privileges, e.g. grants with a quotum on the maximum number of times they are used, and/or the number of bytes stored, etc.

Crash on Linux upon receiving a transaction

* thread #1, name = 'catena', stop reason = signal SIGABRT
  * frame #0: 0x00007ffff4859428 libc.so.6`gsignal + 56
    frame #1: 0x00007ffff485b02a libc.so.6`abort + 362
    frame #2: 0x00007ffff489b7ea libc.so.6`___lldb_unnamed_symbol220$$libc.so.6 + 730
    frame #3: 0x00007ffff48a437a libc.so.6`___lldb_unnamed_symbol277$$libc.so.6 + 1578
    frame #4: 0x00007ffff48a853c libc.so.6`cfree + 76
    frame #5: 0x00007ffff6ae0cf8 libswiftCore.so`destroyArray value witness for Swift.String + 40
    frame #6: 0x00007ffff6969fb9 libswiftCore.so`Swift._ContiguousArrayStorage.__deallocating_deinit + 89
    frame #7: 0x00007ffff696dac4 libswiftCore.so`Swift.String.init (stringInterpolation : Swift.Array<Swift.String>...) -> Swift.String + 36
    frame #8: 0x00000000005fb78e catena`HeliumLogger.formatEntry(type=info, msg="[Node] Re-broadcasting transaction apRxbg3QQkQo+CIYtDw+gQwk/mNwSmxmBrtIbwKmEs=@1 to peers as it is new", functionName="receive(transaction:from:)", lineNum=153, fileName="/root/Sources/CatenaCore/Node.swift", self=<unavailable>) -> String at HeliumLogger.swift:278
    frame #9: 0x00000000005fab7f catena`HeliumLogger.log(type=<unavailable>, msg=Swift.String @ 0x00007fffffffb2c0, functionName=Swift.String @ 0x00007fffffffb2a8, lineNum=<unavailable>, fileName=Swift.String @ 0x00007fffffffb288, self=<unavailable>) -> () at HeliumLogger.swift:239
    frame #10: 0x00000000005fbfa6 catena`protocol witness for Logger.log(LoggerMessageType, msg : String, functionName : String, lineNum : Int, fileName : String) -> () in conformance HeliumLogger at HeliumLogger.swift:0
    frame #11: 0x0000000000433771 catena`static Log.info(msg=<unavailable>, functionName=Swift.String @ 0x00007fffffffb570, lineNum=<unavailable>, fileName=Swift.String @ 0x00007fffffffb550, self=<unavailable>) -> String, functionName : String, lineNum : Int, fileName : String) -> () at Logger.swift:130
    frame #12: 0x0000000000697adf catena`Node.receive(transaction=<unavailable>, peer=<unavailable>, self=<unavailable>, $error=<unavailable>) throws -> () at Node.swift:153
    frame #13: 0x000000000042d1be catena`main at main.swift:224
    frame #14: 0x00007ffff4844830 libc.so.6`__libc_start_main + 240
    frame #15: 0x00000000004254c9 catena`_start + 41

Conditional grants

In the grants table, add an expression column that optionally contains an expression that restricts any action performed using the grant, so you can:

  • Grant a user to only DELETE/UPDATE rows that adhere to a condition (when the grant is used to perform a DELETE/UPDATE query, the restricting expression should be ANDed to the original WHERE statement)
  • Grant a user the right to only INSERT rows with certain values
  • Grant a user to only CREATE tables with a certain name

The expression should be able to contain special variables to check certain things, e.g.:

  • $invoker: the public key of the user invoking the transaction. A grant with expression "$invoker=owner" would only allow modifications to rows that have the public key of the invoker in the 'owner' column. This could be made more complex, e.g. (CASE WHEN $owner=x THEN ... WHEN $owner=y THEN ... ELSE 0 END).
  • $index: the index of the block this transaction is part of (useful for allowing transactions only after a certain amount of time)

Add feature flags to database

Database should contain a table that holds feature flags (e.g. which signing/key algorithms are used). The table should be created in block 1.

Triggers

Perhaps with a triggers table that has on, condition (see also #11) and statement columns (and perhaps also a way to run the trigger as the invoker or as the definer).

Syntax would be 'CREATE TRIGGER' still.

Add credits system

Some ideas:

  • Per-table (or per-row?) expiry date that can be 'topped up' using credits (for which the coins are generated by mining)

  • Evicting expired CREATE/INSERT/UPDATE queries from the chain completely (requires merkle trees)

Add seed nodes

Make seed.catena.pixelspark.nl an A record that points to (non-replaying) catena nodes.

Do not advertise peers discovered through DNS seeds to other peers.

Sidechains

Currently, a node can choose whether to 'replay' SQL transactions on top of the database engine, and select only those tables that it is interested in. The query history however needs to be stored by all nodes for all tables (nodes could throw away blocks after a certain amount of time has passed, but that would make it impossible for new nodes to download the full blockchain from block 0).

It would be preferable if SQL transactions could be split into 'side chains', where each node would only 'follow' the side chains of its interest (and discard the rest).

  • A sidechain is an independent block chain with their own numbering and sidechain genesis (which identifies a particular sidechain)
  • Root chain blocks contain only the hashes of newly mined blocks (if they are accepted). For a miner to determine which sideblocks are acceptable, it only needs to check the last block in the sidechain (is index successive) and maintain state (what is the sidechain difficulty?).
  • The root chain should contain an incentive for mining a side chain block ('credits'?).

Starting a sidechain would be akin to creating a new database. The root chain may also handle payments and dropping of sidechains (either upon request, or when a certain time has exceeded).

Nodes mining a sidechain block should perhaps be required to 'prove' they also still have all history blocks (perhaps by requiring a deterministically randomly chosen history block to be included in the signature hash?).

Things to figure out:

  • How to incentivize miners to both mine the root chain as well as the side chains?
    • Make root blocks only mineable if a miner has provably done N sidechain work (sum of sidechain difficulty * blocks mined since a certain root block)?
  • Difficulty levels of the side/root chain?
  • Should there be a single difficulty level for all sidechains, or individual difficulty levels?
  • Difficulty level for a new sidechain? (genesis requirement)
  • How to ensure that nodes actually keep the blocks in side chains available for download? Require at least the miners to do so?

Add a grants system

Keep a 'grants' table in the database that validates mutations.

Add a command line switch to disable grant validation for databases that do not need it.

Crash on Linux related to NetService

precondition failed: host name 43ff16893f97. should have suffix .local: file /root/.build/checkouts/NetService.git--4183857386241951162/Sources/NetService/Responder.swift, line 6       2
Current stack trace:
0    libswiftCore.so                    0x00007f54507c66b0 swift_reportError + 120
1    libswiftCore.so                    0x00007f54507e0fc0 _swift_stdlib_reportFatalErrorInFile + 100
2    libswiftCore.so                    0x00007f54505db4dc <unavailable> + 1189084
3    libswiftCore.so                    0x00007f545077386d <unavailable> + 2861165
4    libswiftCore.so                    0x00007f54505dacb6 <unavailable> + 1186998
5    libswiftCore.so                    0x00007f5450779780 <unavailable> + 2885504
6    libswiftCore.so                    0x00007f54505db0ef <unavailable> + 1188079
7    libswiftCore.so                    0x00007f545073a499 <unavailable> + 2626713
8    libswiftCore.so                    0x00007f54505dacb6 <unavailable> + 1186998
9    libswiftCore.so                    0x00007f54506f7610 specialized _assertionFailure(StaticString, String, file : StaticString, line : UInt, flags : UInt32) -> Never + 144
10   Catena                             0x0000000000641a66 <unavailable> + 2366054
11   Catena                             0x0000000000641087 <unavailable> + 2363527
12   Catena                             0x0000000000640f85 <unavailable> + 2363269
13   Catena                             0x00000000006589e0 <unavailable> + 2460128
14   Catena                             0x00000000006ab204 <unavailable> + 2798084
15   Catena                             0x0000000000696db8 <unavailable> + 2715064
16   Catena                             0x0000000000696d0c <unavailable> + 2714892
17   Catena                             0x0000000000696e69 <unavailable> + 2715241
18   Catena                             0x000000000042b022 <unavailable> + 176162
19   libc.so.6                          0x00007f544e4ce740 __libc_start_main + 240
20   Catena                             0x00000000004254c9 <unavailable> + 152777

Allow passive peers to connect

Add reply type to 'query' request indicating the the peer does not maintain an index ("passive" action type). Change peer state to .passive and do not query anymore (but do send updates for blocks/txes).

Exchange transactions between peers

Post to other peers so they don't have to be mined at the peer that originated them. Needs a check to find out whether a transaction is already on the chain or not.

NAT Traversal

Options to investigate:

  • STUN
  • TURN
  • ICE
  • uPnP
  • NAT-PMP
  • TCP/UDP hole punching?

Add a mining difficulty control mechanism

Currently, difficulty is fixed at a low value. This could be managed using a 'flags' table that holds the difficulty and/or contains an expression to calculate the difficulty (based on $index, $averageBlockTime, or something like that).

See also #32

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.