espeoblockchain / gardener-server Goto Github PK

Node.js server for Gardener open source oracle project

License: MIT License

Smarty 0.35% Dockerfile 0.24% Makefile 0.45% Shell 4.27% TypeScript 94.69%

gardener-server's Introduction

Oracle server

This repository is a part of open source oracle project. Initially developed by Espeo Software. Contains off-chain server for processing requests from the Oracle contract.

Detailed documentation here: https://gardener.readthedocs.io

Table of contents:

Oracle theory
Architecture
Installation

Oracle theory

Oracle is a concept of getting information from outside of the blockchain to the smart contracts. Out of the box smart contracts cannot access anything outside of the blockchain network. That's were the oracle idea fits. The information exchange begins with the smart contract emitting an event describing the necessary information. A trusted off-chain server listening for such events parses it, gets data from a data source and passes it back to the smart contract.

Architecture

Oracle architecture sketch

Installation

As a first step please clone .env.tpl file into .env and fill with values:

ADDRESS - address of the server's account, from which it sends the results of request to the smart contracts
PRIVATE_KEY - private key of the server's account
ORACLE_ADDRESS - address of the Oracle smart contract
DATABASE_URL - URL with port for MongoDB connection
DATABASE_NAME - MongoDB database name
NODE_URL - URL for the WebSocket provider to blockchain network

Bare-metal

Requirements:

Node.js >= 7.6 - async/await support

Dev environment:

Run npm install to install dependencies.
Run npm run ganache to start a test blockchain.
Deploy necessary smart contracts from oracle-sm repository: cd oracle-sm/ && truffle migrate --network development.
Update the .env file if necessary.
Start the server by running npm start.

Using docker

Local environment with test blockchain (ganache)

Run: make local (single command build)
Build: make build-local
Clean: make clean-local

Environment using public blockchain

Run: make (single command build)
Build: make build
Clean: make clean

Attention! Check if variables in .env file are correct.

gardener-server's People

Contributors

Stargazers

Watchers

Forkers

hakanatabas kss-espeo julianmrodri coinfirm elohffa robertrongen

gardener-server's Issues

Trusted Computation

As a request from the Oracle execute a script in the server and return the result (with capped length).

One possibility for creating the computation:

Create an archive in IPFS and upload the binary or script, dependencies and dockerfile in it.
First argument can be the IPFS multihash that specifies the archive location. The rest of the arguments can be passed to the execution environment as environment variables.

Alternative solutions always welcome :)

Create a business-level SGX intro

It is hard to understand SGX capabilities by business people. Solve that issue by creating a high level presentation about SGX that shows what are the benefits of it from use case point of view - and to allow a differentiation from ones where SGX cannot help.

After it's done. run it with @imacoindev and Sylwia.

Fix npm vulnerabilities

Currently, running npm audit shows multiple security vulnerabilities, due to unsafe package version.

Fix it, according to best npm standards. This task is done when we have 0 security vulnerabilities after running npm audit

Investigate integration to Oracle Blockchain Platform

So far it seems that Oracle does not have a SGX integration in their Blockchain Platform. We should look into what are the benefits that we can offer to them by integrating Gardener to their platform.

Think of this in terms of:

Oracle's core business is selling database licenses and hosting to enterprises.
They have a PaaS Blockchain Platform that uses Fabric to allow their clients to have business logic on top of their stored data. This allows to automate business processes and create innovative solutions.
Oracle's clients come from all possible industries.
For example, their client uses MySQL to store their supply chain data. They want to build an automation to their workflow by implementing chain code that automates a large chunk of their process and removes risk for errors when people make entries manually across the supply chain.
How can Gardener improve the security of their Blockchain Platform and benefit their clients. For example, if they don't have remote attestation and sealing implemented, how would that benefit them and their clients?

More details of their platform can be found in:
https://www.oracle.com/application-development/cloud-services/blockchain-platform/

Create estimations for backend issues

IntelSGX proof for URL and random data sources

TLS Notary proof for URL and random data sources

Computation data source without proof

IntelSGX and TLSNotary proofs for Computation

Response to blockchain failing to send when using Mongo

Configure Gardener with PERSISTENCE=MONGODB
Send any query to Oracle
Query is processed by gardener-server
Expected: query response is sent to blockchain and no error messages are shown
Actual: sending query response fails with following error message:

Sending response failed [requestId=0xa2b8357552a4ae3ae94e99949a5ea45ff35f7e9e74657207cdcc2d5b2ec9c955] Error: No "from" address specified in neither the given options, nor the default options.
    at Object._executeMethod (/usr/src/app/node_modules/web3-eth-contract/src/index.js:832:45)
    at EthereumOracleAdapter._sendPendingResponse (/usr/src/app/src/infrastructure/blockchain/ethereum/EthereumOracleAdapter.js:73:20)
    at process._tickCallback (internal/process/next_tick.js:68:7)

XPath not handled correctly for node attributes

Example of good behavior:

xml(http://samples.openweathermap.org/data/2.5/weather?q=London&mode=xml&appid=b6907d289e10d714a6e88b30761fae22)/current/city/country

Returns:

// correct
value: "GB"
error: 0

Example of bad behavior:

xml(http://samples.openweathermap.org/data/2.5/weather?q=London&mode=xml&appid=b6907d289e10d714a6e88b30761fae22)/current/temperature/@value

Returns nothing and throw error on server side but should:

value: "280.15"
error: 0

Divide source code and test code

Currently our test files live next to source files. This leads to some awkwardness like:

inability to configure app and test environment distinctly
in some cases, app source, test source and test data living in same package (yikes!)
reducing code readibility

Objectives of this tasks are:

Move all test sources - *.spec.ts and *.integration.ts to a location separate from source code. Look up what are TS standards for this, but my idea is a "Java-like" approach with src/ root for source code and test/ root for test stuff.
Provide a distinct configuration (tsconfig.json) for app and tests. Test config and app config will likely derive from some root config (use "extends" keyword)

Sending response to blockchain fails on remote server

Go to https://monitor.gardeneroracle.io/
Execute any query
Investigate server logs (ssh to oracle-demo.espeo.pl in order to get to those)

Expected: fetched data is sent to blockchain
Actual: Data is fetched but sending it to blockchain fails with a following error:

Sending response failed [requestId=0x38b681e93ed93ca529c65bd81c9ff50d0ba0cee250846bc0a9eb95bbca99cb6c] Error: Returned error: gas required exceeds allowance (8000000) or always failing transaction
    at Object.ErrorResponse (/usr/src/app/node_modules/web3-core-helpers/src/errors.js:29:16)
    at /usr/src/app/node_modules/web3-core-requestmanager/src/index.js:140:36
    at XMLHttpRequest.request.onreadystatechange (/usr/src/app/node_modules/web3-providers-http/src/index.js:91:13)
    at XMLHttpRequestEventTarget.dispatchEvent (/usr/src/app/node_modules/xhr2-cookies/xml-http-request-event-target.ts:44:13)
    at XMLHttpRequest._setReadyState (/usr/src/app/node_modules/xhr2-cookies/xml-http-request.ts:219:8)
    at XMLHttpRequest._onHttpResponseEnd (/usr/src/app/node_modules/xhr2-cookies/xml-http-request.ts:345:8)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/xhr2-cookies/xml-http-request.ts:311:39)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1129:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Before investigating this, resolve #60 - it might help.

Adding persistance storage [MongoDB]

Currently only in memory storage is supported so data disappears during server recreation. More persistant storage needed. MongoDB would be a good shot. Even if someone want something different hexagonal architecture allows easily create new adapters for the same interfaces.

Closed API access

Create access for APIs that need authentication.

Passing error codes when request cannot be fulfilled

Currently only responses for valid requests are generated. We should support also failed ones with proper codes in order to know what exactly happens from the smart contracts perspective.

IPFS integration

Integrate IPFS as a data source.

Hyperledger Fabric integration

Integrate Gardener with Hyperledger Fabric. It should be seamless - all the smart contracts that currently work on Ethereum should also on Fabric. Server-side part of Gardener should remain unchanged.

Intel SGX

Integrate Intel SGX to prove data validity.

Random Data Provider

Gardener-server should deliver random data when requested.

Custom errors are not correctly handled

Call oracle with a malformed URL, for example UsingOracle.deployed().then(instance => instance.request("json(badUrl"))
Wait for a transaction to be mined

Expected: A stacktrace of InvalidUrlError is found in the logs
Expected: 1000 error code is being logged
Actual: No error logging at all
Actual: No error code is logged: Created response [response={"requestId":"0x3b7602292b26ffa4b1bea411d4594bcbd24a9b288169cd3f739c88a19c604cf4","state":{"name":"Processed"},"errorCode":0}]

This is just an example, such a case is also true for other custom errors, namely HttpError and InvalidContentTypeError.

Migrate npm to yarn

I suggest to migrate npm to yarn and stop freezing packages manually.

Yarn has more intuitive commands than npm, and doesn't allow user to e.g. install lib without adding it to package.json or install node_modules with update lock file.

Gardener demo

We could create a web app that demonstrates what Gardener can do to new users and developers. This is what I suggest as a high level overview of the app:

Persistent: The app is online all the time and can be accessed via browser.
Allows the user to request data by sending a request to a deployed Gardener smart contract and returns the result of the request to the browser to be seen by the user. It is a live demo of what a Gardener instance can do.
It demonstrates different data sources. For example, allows to select what type of data the user wants (random, url, etc). The amount of data sources can be increased as we integrate more to the server.
It demonstrates different proofs that can be used to verify the validity of data. For example, user can choose to use IntelSGX or NotaryTLS to verify a data source.
The on-chain contents are deployed in Ethereum testnet. Our requests are visible in the testnet for anyone to verify that they are actually executed on-chain.

Similar type of demo has been done by Provable: http://app.provable.xyz/home/test_query

Some user stories(all features are not ready, but we should be able to add them to the app later):

As a user I can have a simple demonstration of obtaining data using Gardener.
As a user I can select different data sources
As a user I can select different proofs to validate the data that is delivered is not tampered with.
As a user I can see that the style and design of the site matches that of https://gardeneroracle.io/
The demo can be accessed through demo.gardeneroracle.io

Update web3.js version

Currently we're using beta versions of web3.js:

    "web3": "1.0.0-beta.36",
    "web3-eth-contract": "1.0.0-beta.55",
    "web3-utils": "1.0.0-beta.55",

In scope of this task, please update all web3 packages to a stable version as per https://www.npmjs.com/package/web3 . Test application thoroughly and adjust code if needed.

Doing this might fix #57 or at least shed some light on it.

Verified computation

Verify the that the computation has been performed correctly.

Move entire contentType-specific logic into RequestStrategy

After #32 and #42 , some logic that is specific to contentType is still being done outside of RequestStrategy. This leads to code duplication, worse testability and maintainability.

In scope of this task, refactor exisiting solution to move contentType-speciic logic into RequestStrategy. This will probably mean that contentType enum will be interoduced, Request class will be made abstract and some extensions to it will be created and RequestUrlParser logic will be moved to RequestStrategy. Refer to Code Review comments in #32 and #42 .

While you're at it, it would be nice to add more tests around RequestExecutor logic.

0 errorCode returns when data part not exist in existing response

When data part doesn't exists in existing response oracle should return 404 not 0 errorCode.

Currently for:

//misspell, correctly should be .chartName
json(https://api.coindesk.com/v1/bpi/currentprice.json).chartname

is:

value: ""
error: 0

should be:

value: ""
error: 4004

Refactor tests to use proper mocks

Currently, many of the Gardener tests use custom, promitive mocking mechanism. An example of that can be found in ExecuteReadyRequestsUseCase.spec.ts . They look like this:
const fetchDataUseCase = () => ({ fetchData: () => Promise.resolve('fetchedData'), });

In scope of this task, refactor entire gardener-server codebase to use some standard mock mechanism instead. A specific technnique to be used is left to be researched but this looks promising: https://jestjs.io/docs/en/es6-class-mocks#automatic-mock .

Add logo to readme.md

Design of the decentralized oracle

Design the architecture of the off chain oracle network based on Schnorr signatures.

Each node signs the data with their portion of the ring signature. After ring is completed the agreed value is broadcast to the blockchain layer.
Consider de incentivizing freeloading by inputting same value as previous node in the ring.
Consider security in the p2p connections between nodes.

Rewrite using Typescript

As our whole server is written in OOP, it would be very useful from developer perspective to have more restrictions on object or parameter type. It would speed up development and be less error prone.

Deploy SGX version of gardener-server on AWS

Investigate how to hire an AWS instance that supports SGX. It needs to have a SGX-ready processor which is any modern Intel processor as well as SGX enabled in BIOS.

After we have an AWS instance, deploy a gardener stack on it.

This needs to be done as a prerequisite of SGX-powered Random Number Generator demo : #39
This task requires #22 in master to be finished.

Adding JSDocs comments to Port classes

Javascript doesn't allow to specify input/output types. As this is the essence of ports which we are using in hexagonal architecture it would be nice to get clean view of these parameters. It can be achieved by adding JSDocs comments.

Use SGX Sealing to encrypt random values generated by Enclave

Currently, a process of generating random value in Enclave is vulnerable to man-in-the-middle attack - should attacker take control of a machine that hosts gardener-server, he could replace generated value with one of his choice.

To remedy that, we should use SGX Sealing mechanism to hardcode user's public key (assymetric cryptography, does not necessarily have anything to do with blockchain keys - even though it could) . This public key would be used to encrypt generated data in enclave and send it to the user - decrypting value with his private key would be user's responsibility.

Potential security risk: attacker could take control of user's public key, intercept gardener-server machine and generate encrypted value. This means that IF such attack is feasible, then instead of just hardcoding user's public key, we could go for Diffie-Hellman Key Exchange.

Finishing this task is the last milestone for SGX implementation in the context of RNG for gambling, since having it guarantees that the only Third Trusted Party in entire RNG process is Intel - user does not even have to trust a party that hosts gardener-server.