bitfinexcom / bfx-hf-data-server

Bitfinex Honey Framework Data Server for Node.JS

The HF data server runs backtests for the HF UI Electron App. It syncs candle and trade data and then executes the strategy on it for backtesting.

The DB backend is implemented by a plugin:

• bfx-hf-models-adapter-lowdb

Regardless of the backend, a schema must be specified (providing exchange-specific API methods). The official Bitfinex schema is bfx-hf-ext-plugin-bitfinex.

Installation

For standalone usage:

git clone https://github.com/bitfinexcom/bfx-hf-data-server
cd bfx-hf-data-server
npm i

cp .env.example .env

npm run start-lowdb

For usage/extension within an existing project:

npm i --save bfx-hf-data-server

Quickstart

Follow the installation instructions, and run npm run start-lowdb.

Docs

For executable examples, refer to examples/

Example

const DataServer = require('bfx-hf-data-server')
const HFDB = require('bfx-hf-models')
const HFDBLowDBAdapter = require('bfx-hf-models-adapter-lowdb')
const { schema: HFDBBitfinexSchema } = require('bfx-hf-ext-plugin-bitfinex')

const db = new HFDB({
  schema: HFDBBitfinexSchema,
  adapter: HFDBLowDBAdapter({
    dbPath: './SOME_DB_PATH.json',
    schema: HFDBBitfinexSchema
  })
})

const ds = new DataServer({
  port: 8899,
  db
})

ds.open()

// data server ready to receive commands

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Note

This package will be maintained only via github, please use latest relases from github instead of npm.

Example on how to install specific version from github:

npm i --save-prod https://github.com/bitfinexcom/bfx-hf-data-server.git#v3.0.4

Example on how to install it latest version from github:

npm i --save-prod https://github.com/bitfinexcom/bfx-hf-data-server.git

bfx-hf-data-server's Issues

Refactor: Optimize DB trade query in exec.bt

Currently all trades are fetched at once from the DB for a backtest, whereas in the past they were fetched in chunks. This should be refactored to replicate the legacy behavior and speed up backtests.

lib/cmds/exec_bt.js

  const trades = await Trade.getInRange([
    ['exchange', '=', exchange],
    ['symbol', '=', symbol]
  ], {
    key: 'mts',
    start,
    end
  }, {
    orderBy: 'mts',
    orderDirection: 'asc'
  })

Fix: Include trade count on bt.start message

Currently we send 0 as the backtest trade count on the bt.start message, as the true count is only known later once the trades are fetched from the DB.

Fix this by fetching the trades first and passing the true trade count along with the rest of the bt.start data.

lib/cmds/exec_bt.js

send(ws, ['bt.start', '', '', start, end,, 0, candleData.length])

Refactor: Remove BFX Proxy

The bfx proxy functionality was added to allow bfx-hf-ui to communicate with the bfx API (not possible in the browser due to CORS)

This is no longer needed as it is handled by the new bfx-hf-server

Candle sync feature doesn't work

Issue type

• [x ] bug
• missing functionality
• performance
• feature request

Brief description

Anytime we send event exec.str to websocket, bfx-hf-data-server crawl candle again, nothing store in datatabase

Steps to reproduce

Additional Notes:

MongoError: could not start mongo process

PS E:\bfx\node_modules\bfx-hf-data-server\examples> npm start

[email protected] start E:\bfx\node_modules\bfx-hf-data-server
node examples/server.js

info nems Starting download of mongodb 3.6.6 to a mongodb-download directory under C:\Users\sav17\AppData\Local\Temp
bfx:hf:util:catch-uncaught unhandled rejection: MongoError: could not start mongo process: ���⥬� �� 㤠���� ���� 㪠����� ����.
(+0ms)
Promise {
{ [MongoError: could not start mongo process: ���⥬� �� 㤠���� ���� 㪠����� ����.
]
message:
'could not start mongo process: ���⥬� �� 㤠���� ���� 㪠����� ����.\r\n' } }

Websocket documentation

Issue type

• bug
• missing functionality
• performance
• feature request

Brief description

Why did you removed the websocket server documentation ?

Steps to reproduce

Additional Notes:

Syncing data is broken

Hi. Syncing data does not work. Syncing is detecting time gaps only with mts criterion. So if you backtest for example EOSUSD it will fetch the data correctly, but try to backtest XRPUSD after that for the same timeframe. It will result in 0 candles.

Also if the filtration based on symbol will work the gap detection is somehow broken, it returns very little candles... for whole year just 1500 candles

Feature: Add Tests

The test folder currently contains a stub so npm test passes; this needs to be filled in/expanded.

nodejs error when using server

node examples/server-lowdb.js

bfx:hf:models:adapter-lowdb loading from /home/quenouille/bfx-hf-data-server/db/dev.json +0ms
bfx:hf:data-server websocket API open on port 8899 +0ms
bfx:hf:data-server ws client connected +3s
bfx:hf:data-server error processing message: TypeError: Cannot destructure property exchange of 'undefined' or 'null'.
at module.exports (/home/quenouille/bfx-hf-data-server/lib/bt/sync_data.js:9:7)
at module.exports (/home/quenouille/bfx-hf-data-server/lib/cmds/exec_bt.js:27:11)
at DataServer.onWSMessage (/home/quenouille/bfx-hf-data-server/lib/server.js:345:5)
at WebSocket.emit (events.js:198:13)
at Receiver.receiverOnMessage (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/websocket.js:825:20)
at Receiver.emit (events.js:198:13)
at Receiver.dataMessage (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/receiver.js:437:14)
at Receiver.getData (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/receiver.js:367:17)
at Receiver.startLoop (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/receiver.js:143:22)
at Receiver._write (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/receiver.js:78:10)
at doWrite (_stream_writable.js:415:12)
at writeOrBuffer (_stream_writable.js:399:5)
at Receiver.Writable.write (_stream_writable.js:299:11)
at Socket.socketOnData (/home/quenouille/bfx-hf-data-server/node_modules/ws/lib/websocket.js:900:35)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17) +5ms
bfx:hf:data-server:wss:send-error internal error (103) +0ms

Getting this error when running the data_backtest server part of ema_cross.py from bfx-hf-strategy-py and also the online_bt example of bfx-hf-backtest.

Ubuntu 20.04 LTS
npm 6.14.4
Python 3.8.5

Websocket API fails with no such table candles

Issue type

• bug

  bfx:hf:data-server` error processing message: Error: ER_NO_SUCH_TABLE: Table 'mydb.candles' doesn't exist

Brief description

I am trying to use this module bfx-hf-strategy-py with the dataserver running on another port. I want to use the sql backend for it. I properly set up the connection (it succeeds) but at the first invocation of the WS API it complains that the table doesn't exist.

How do I create such table? Am I missing something?

Steps to reproduce

run the example in bfx-hf-strategy-py using the dataserver as backend