dema-trading-ai / engine

Hi, we are @dema-trading (https://dematrading.ai)! We are developing Algorithmic Trading Strategies and Trading Bots for multiple platforms like BOTS (https://bots.io).

Would you like to get in touch? Feel free to send us a message using the contactform at https://dematrading.ai/.

We are currently preparing our rocket to mars... Be sure to board quickly!

engine's Issues

Create engine wiki

We need to create some sort of wiki / documentation for the configuration and usage of our backtesting engine. If we start creating one now, keeping it updated with only take a small amount of time.

Does anyone have experience with wiki/documentations or suggestions for achieving this feature?

Options:

• Use GitHub pages to create a wiki
• Use any sort of opensource wiki software

ROI for past time

Create ROI based on passed time for a trade. Just like freq implemented it, we want to be able to set ROI values for passed time. For example:

after 1h, ROI = 10%
after 2h, ROI = 8.9%

Implementation in freq:

Todo:

• Make this easy configurable in config
• Check ROI based on passed time in open trade tick method

Change backtesting strategy

Our current implementation for backtesting takes a lot of time to process all the ticks. This time issue is the result of our backtesting engine being designed as if it is 'live trading'. I think we can change this strategy to actual backtesting since all data is already available at the beginning of the backtest. To change the current implementation to an actual backtest, the engine needs to populate indicators, buy and sell signals only once (using all data).

To do:

• Change backtesting strategy from 'live trading' to actual backtesting
  • Populate the indicators only once
  • Populate buy signals only once
  • Populate sell signals only once (incl. ROI and stoploss)
• Evaluate and improve that way that the 'Trade' class is currently used

It's hard for me to estimate how much time this is going to take to implement, but what do you guys think of the idea?

Sample strategy optimization

We need to update the sample strategy so;

• Users can see how to use the current candle's data
• Users can see how to use the previous candles's data

Attention: this has changed due to #131

Write first Unit Tests

After completing #19 , Unit tests need to be added. In consultation with Tijs, you are going to write automated tests for our engine! In the end, we strive for a code coverage of 75%.

OHLCV data / indicators to DataFrame

(Together with Tijs)

Currently, all our OHLCV data and indicators are stored in either an object array or dictionary. This is not really useful for TA-Lib / calculating purposes (since TA-Lib requires numpy array as input). We need to put all data inside a dataframe, and make this work application-wide. This is necessary for the strategies to work.

[Bug] fix bug when downloading data

Have a look at the screenshot.

Implement RapidJson

Have a look at https://pypi.org/project/python-rapidjson/ and what investigate what this could bring us in terms of speed.

Bug - progress bar on above other notifications

For most people this seems to work, but on my m1 mac, using MacOS Bug Sur (in a zsh shell) there is a small bug in the progress bar. The progress bar should go above the info tags, but for me the progress bar is shown first, and then the other prints come. We are supposed to see:

But I get:

Check profit calculations properly

I have some doubts about all my calculations and therefore want it to be checked by somebody else.

Things to check:

• Overall profit calculations (both %/$)
• Sell Reason count methods (ROI/SL/Signal)
• Trade open/close method calculations
• Per coin statistics

Things to do:

• Make all the calculations more readable
• Make separate methods for calculations (if calculation is redundant)
• Optimize calculations

Research for Hyper Optimization possibilities

In the upcoming months, we really want to implement HyperOpting in our backtesting engine. For this, we need to know what needs to be changed concerning the current implementation.

Things to check out:

• What of the current implementation should be changed to match requirements for hyperopting
• How long would it take to implement hyperopting
• What separate tasks could be made for hyperopt implementation? + task time estimation

Create deployment pipeline for stable releases

In consultation with Tijs, you can create a deployment pipeline for stable releases.

Requirements of pipeline:

• Create branche for stable release
• Only runs on push to /release/stable branche
• Builds dockerimage with :stable tag
• Pushes dockerimage to github packages

Information: https://github.com/marketplace/actions/build-and-push-docker-images

Implement sample strategy with ONLY necessary lines

Feature request:

• Create a sample strategy that acts like a template; only leave the lines that are necessary to make the strategy work. This file could be named 'strategy_template' or smth.

Add a proper CLI

Currently, to run the engine you either have to do the docker commands docker build .... and docker run ...., or use make which isn't really made for this. We still need to configure docker-compose but the commands would still be long does not look really authentic. What do you think about making an executable that handles this?

$dema run and $dema build

Dynamic stoploss for every trade

We have to have to ability to configure stoploss (SL) on a per trade basis. This allows for configuring stoplosses on the go, based on indicators for example. This should be an option for more advanced users.
The config should have the standard 'static' stoploss, which gets overridden in case a user specifies it when opening a trade.

Things to do:

• Create method in strategy base class, method should return stoploss (float) for every new trade
• Baseclass method always returns config default SL
• Create SL calculation method in strategy, so it's configurable for every trade that opens + for every strategy
• Call SL calculation method on opening trade. Store SL in trade model
• Add property to trade model for storing trade specific SL
• Adjust methods in trading module for working with trade specific SL when this is turned on

[Docs] Strategy example / getting started additions

We need a step-by-step guide of how to create your first own algorithm. This need to be done in docs part strategy examples

Poetry as virtual environment instead of Pip

Reasons:

• Poetry performs its locking significantly faster than pipenv. Because both tools perform file locking after nearly every command, poetry feels snappier in everyday use.

• When removing a dependency, Poetry also uninstalls its sub-dependencies if no other package uses them. Pipenv does not. In this benchmark, for instance, Pipenv orphans 5 dependencies.

• Pipenv has broad support. It is an official project of the Python Packaging Authority, alongside pip. It's also supported by the Heroku Python buildpack, which is useful for anyone with Heroku or Dokku-based deployment strategies (like me).
  Poetry is a one-stop shop for dependency management and package management. It simplifies creating a package, managing its dependencies, and publishing it. Compared to Pipenv, Poetry's separate add and install commands are more explicit, and it's faster for everything except for a full dependency install.

• Poetry's version 1.0 is set to address the two areas where I preferred Pipenv: being able to export a requirements.txt file, and being able to avoid installing a package as a dependency of itself.

Review / Update backtest result overview

On the backtest-overview, a couple of things needs to be changed.

Things to be revised:

• General 'generate backtest result' method. This method is way to large
• Tabs / indents in docker result

Things to be added to the backtest overview:

• Total fee paid
• Configured stoploss
• Put 'seen' in front of 'drawdown from' / 'drawdown to'
• From - To on 1 line
• Ended with $ / % on 1 line

Create configurable base currency

In the current state, we can add any coin pair we want. Even if the base currency differs. For example, we can add both BTC/USDT and ETH/BTC to our pairlist. From this point, calculated profit won't be accurate at all.

We need next functionality:

• Base currency configurable through config file
• Check pairlist for validity (if base currency is USDT, only pairs ending with "***/USDT" would be valid
• Change currency signs in backtest result based on specified currency (currently everything is displayed with $ sign)

Make strategyName configurable

Make the strategy configurable. F.e. I want to be able to create a strategy called CoolStrategy and simply put this in the config file.

• Make general strategy interface / base class
• Adjust configuration
• Adjust configuration checks
• Loading strategy based on config (TradingModule)

Implement multiple strategy examples

Feature request:

• Implement multiple strategy examples so users can clearly see how they could customize the strategy files.

Reuse existing candle data

For backtesting purposes, historical data (candles) are needed. These are fetched through an API and stored locally.
However, this data must match the data needed for a backtest exactly in terms of pair, timeframe and date.

Currently, when the data does not match your backtest needs (lets say that you are backtesting 1 day longer then you have data), the engine will erase all data it has, and download everything again.

This costs time and resources, so a check needs to be implemented that complements the existing data (if possible) so that it does matches the data needed for the backtest.

In other words, if the stored data is the correct pair and timeframe, but it is missing a few days/weeks/month, download only that period, instead of the full period

This functionality should be implemented somewhere here:

engine/data/datamodule.py

Lines 88 to 102 in 3378ecf

	if not self.check_for_datafile_existence(pair, self.config['timeframe']):
	# datafile doesn't exist. Start downloading data, create datafile
	print("[INFO] Did not find datafile for %s" % pair)
	self.history_data[pair] = []
	self.download_data_for_pair(pair, True)
	elif self.does_datafile_cover_backtesting_period(pair, self.config['timeframe']):
	# load data from datafile instead of exchange
	self.history_data[pair] = []
	self.read_data_from_datafile(pair, self.config['timeframe'])
	elif not self.does_datafile_cover_backtesting_period(pair, self.config['timeframe']):
	# remove file, download data from exchange, create new datafile
	self.history_data[pair] = []
	if self.check_for_datafile_existence(pair, self.config['timeframe']):
	self.delete_file(pair, self.config['timeframe'])
	self.download_data_for_pair(pair, True)

Roughly the following checks should be followed: is there data -> is it for the right pair -> is it for the right timeframe -> does it covers the backtesting period -> download missing candles only

Open trades table wont show when there is no profit yet

When there is no profit yet the value is still None, which you cant round. Therefore, errors and table no-show

Optimize checking presence of required config params

The current approach is not scalable and will soon prove unmanageable if more and more (required) config params are added.

engine/main.py

Lines 26 to 29 in 0bd655d

	# Check whether config contains all necessary properties
	check_config_max_open_trades(config)
	check_config_starting_capital(config)
	check_config_stoploss(config)

Make one function that takes the required param as argument and check whether it is present in the config

Furthermore, the following check doesn't work as expected, and will throw a keyerror if not present (instead of the desired handling)

engine/main.py

Line 42 in 0bd655d

if config_json['stoploss'] is None:

Something like this should do the trick

if 'stoploss' not in config_json: 

Have a look at pythons 3.6 @dataclasses package

Some guy suggested we need to have a look at the dataclasses package in Python, since this will decrease the amount of code written. Moreover, it will increase readability of the code, so double win :).

Things to be done:

• Read https://docs.python.org/3/library/dataclasses.html carefully
• Pitch to either Tijs, Marijn or Demian what you think of it / how long would it take to implement
• Implementing this package

Review seen drawdown calculating method

In the current implementation, somewhere within all the calculating or within tracking data something is missing / going wrong. The current results show wrong 'max_seen_drawdown'. I do not exactly know where things go wrong.

We can help you out completing this task!

Add fixed stake amount

We want to have the ability to set a fixed stake amount. In the current version, only compound trading is configured (% of total budget), but we also want the ability to set a fixed stake.

Things to consider:

• If stake size is set, we might want to see other backtest results (at least see the set stake size)
• Showing whether compound mode is enabled in backtest results

Things to change:

• Add config 'enable_fixed_stake' prop
• Add config 'fixed_stake' prop
• Checks for presence of both props
• Method for calculating stake size, in which you can include check for fixed / compound stake

Move fee validation to validations.py

Currently, the config of fee is checked in tradingmodule.py instead of validations.py. Move this to validations.py and make sure it still works properly

Wrong total price calculation

Total price is added twice if time key is already present in the dict.

Create progress bar

Since the adoptation of dataframes backtesting takes significantly more time. Therefore, it is nice to add a loading bar to see the the progress.

Things to be done:

• Implement progress bar in backtesting

Multi timeframe support

After discussions with @noudcorten we found out that the current implementation on multi timeframe support is not really how we want it. We need to revise this (in communication with @noudcorten )

EDIT: multitimeframe is removed and needs to be re-implemented. We want to implement this using informative pairs.

Create docker-compose file

Create a proper working docker-compose file so backtesting can be done without necessarily building the image.

• Volume mounts for /data/backtesting-data
• Volume mounts for /backtesting/strategy.py

Any suggestions regarding this issue?

[Docs] Write content for 'Making Money'

For the BETA version, it is necessary that the Making money part is filled with appropriate information. We need proper calculations and a step-by-step guide of how the process will go when you want your strategy published.

Revise Makefile so volume mounting works

Currently, in the makefile we have a section for 'runv', which should mount the current directory to the container of the engine. This is done so downloaded data could be re-used. However, this doesn't seem to work.

This is no priority, only an addition. We should be better off by using a docker-compose file. Docker-compose HAS priority.

Enhancement - add fee support

Currently, fees are excluded from all calculations. Things that must be done include:

• Determine whether the fee in config.json must be a percentage or fraction? Or can be both, and percentage is specified by typing for instance fee: "0.1%"
• Come up with suitable validations for the provided fee value (e.g., must be positive, less than 1 or 100%)
• Fees must be included in budget changes in Tradingmodule.open_trade() and .close_trade().
• Save fee statistics like total_fee_paid in TradingModule, and print them in the final results (BackTesting).

Proper error handling application wide

In the current implementation, alot of SystemExits are raised when an error occured. This is done on purpose, but will not serve our future goal of making everything maintainable.

For example:

  print("[ERROR] Backtesting datafile was not found.")
  raise SystemExit

Should be replaced with:

    raise FileNotFoundError("[ERROR] Backtesting datafile was not found.")

Plotting possibilities after running backtest

A nice addition would be some kind of plots to visualize the following things after/during backtesting:

• Budget over time
• Buy/Sell moments on price graph of certain coin pair (OHLCV)
• Average profit of trades over time
• ...

What do you guys think?
What should be changed on current implementation to match requirements for plotting?

Proper startup time

Because some indicators are calculated based on x amount of passed candles (f.e. SMA), we need a startup time to be able to generate the indicators properly.

Things to be done:

• Make startup time
• Check for presence in config file
• During the first x candles, send no signals / generate no indicators

Addition: if there's any possibility to find the appropriate startup time automatically, this would be perfect!

Incorrect backtesting period

The current way of fetching the ohlcv data can be observed in the following screenshot:

But this implementation is wrong. When looking at the documentation for binance.fetch_ohlcv() (line 1364: https://github.com/ccxt/ccxt/blob/b59bb1ce93bde9832cdb22d4a85cc7a77abc9672/python/ccxt/binance.py#L29) there is a parameter called 'limit' which is currently not used. A quick intuition behind this paramter is that 'limit' defines how many candles you want to fetch. So if timeframe='5m' and limit=12 it will fetch 12 candles from the specified date using 5m timeframe (covering 12*5m=60m=1h).

The problem with our current implementation is that the 'limit' parameter is not defined, meaning the function will choose this parameter itself. After some testing I found that not specyfying the limit paramater results in 'limit=500'. Furthermore, the maximum amount of candles you are able to fetch is 'limit=1000'.

Because the fetch_ohlcv() will always fetch 500 candles from the given date, it can greatly surpass our defined 'testing_to'. This means that our fetched data could be far beyond what we want. This needs to be fixed.

Check for missing ticks / empty tick data

We want a hard check on possible missing ticks in downloaded data. This means that we have to check if all downloaded pairs have data for every needed tick. It could be that f.e. BTC/USDT has 1 missing tick on 13.00, while the other downloaded pairs do actually have this tick. In that case, we want to be notified using console error output.

If you have any questions, hit us up on Discord.

Command line argument support

If we want to change a variable we currently have to edit the config.json, which is a bit devious. Therefore it is nice to add command line support to easily change the variables or select a different strategy.

Things to do:

• Make every argument in the config.json configurable using the command line.
• Add --help argument to see which variables can be changed

Create strategy examples

Create examples of a strategy so new users have an idea where to start.

ToDo:

• Edit strategy.py, add indicator + buy/sell signal example
• Adjust documentation

Get in touch with Tijs / Demian for good examples, if you need to.

Add marketchange (avg + bitcoin)

We want to know the average marketchange for whitelisted coins in set backtesting period.

Things to do:

• Make calculations for average marketchange for all whitelisted coins (printed %)
• Make calculations for average marketchange for bitcoin (as benchmark) (printed %)
• Include results in backtesting overview

Implement QtpyLib

We need to implement QTPyLib for our engine so we are able to use some of their functions. The link to QTPyLib could be found here:
https://pypi.org/project/QTPyLib/

Dockerize engine to prevent dependency errors

We got some messages that people had issues with dependencies as CCXT and TA-Lib. For this reason, it is nice to dockerize the engine.

Things needed:

• TA-Lib binaries install script
• Dockerfile
• Docker-compose file

Typos in README.md

There are some small typos in the README file which are nice to have changed.

To do:

• Update README.md

Write CSS for docs > make our logo visible in the topbar

Create CI/CD + add testing packages

Create a CI/CD for this engine. Including following tasks:

• Install testing framework for the engine
• Create test pipeline (on every push)
• Create build pipeline (docker-image, registry push) (on PR to main)
• Connect test-pipeline code analyzing service
• Add PR / Branche protections based on Test pipeline results
• Add badges to README

Questions:

What testing framework could best be used for Python?
What analyzing services could best be used for code analysis? I only have experience with SonarCloud

Possible coding analyzing services:

• SonarCloud
• Codacy (comes with GitHub Apps)

Add PR (review) templates

Templates should be added for Pull Requests and PR reviews. If they are added correctly, github will automatically add them to the description of PR (reviews)

Add multi timeframe support

Create a way to be able to use multiple timeframes for every pair.

Things to be done:

• Make it easy configurable
• Adjust downloading / checking for multiple timeframes
• Make data easy accessible in generate_indicators methods