draichi / t-1000

:zap: :zap: 𝘋𝘦𝘦𝘱 𝘙𝘓 𝘈𝘭𝘨𝘰𝘵𝘳𝘢𝘥𝘪𝘯𝘨 𝘸𝘪𝘵𝘩 𝘙𝘢𝘺 𝘈𝘗𝘐

T-1000 Advanced Prototype

Deep reinforcement learning multi-agent algorithmic trading framework that learns to trade from experience and then evaluate with brand new data

---

Prerequisites

• Miniconda or Anaconda

An API Key on CryptoCompare

---

Setup

Ubuntu

# paste your API Key on .env
cp .env.example .env
# make sure you have these installed
sudo apt-get install gcc g++ build-essential python-dev python3-dev -y
# create env
conda env create -f t-1000.yml
# activate it
conda activate t-1000

---

Usage

On command line

# to see all arguments available
# $ python main.py --help

# to train
python main.py -a btc eth bnb -c usd

# to test
python main.py /
    --checkpoint_path results/t-1000/model-hash/checkpoint_750/checkpoint-750

On your own file

# instatiate the environment
T_1000 = CreateEnv(assets=['OMG','BTC','ETH'],
                  currency='USDT',
                  granularity='day',
                  datapoints=600)

# define the hyperparams to train
T_1000.train(timesteps=5e4,
              checkpoint_freq=10,
              lr_schedule=[
                  [
                      [0, 7e-5],  # [timestep, lr]
                      [100, 7e-6],
                  ],
                  [
                      [0, 6e-5],
                      [100, 6e-6],
                  ]
              ],
              algo='PPO')

Once you have a sattisfatory reward_mean benchmark you can see how it performs with never seen data

# same environment
T_1000 = CreateEnv(assets=['OMG','BTC','ETH'],
                  currency='USDT',
                  granularity='day',
                  datapoints=600)

# checkpoint are saved in /results
# it will automatically use a different time period from trainnig to backtest
T_1000.backtest(checkpoint_path='path/to/checkpoint_file/checkpoint-400')

---

Features

• state of the art agents
• hyperparam grid search
• multi agent parallelization
• learning rate schedule
• result analysis

"It just needs to touch something to mimic it." - Sarah Connor, about the T-1000

---

Monitoring

Some nice tools to keep an eye while your agent train are (of course) tensorboard, gpustat and htop

# from the project home folder
$ tensorboard --logdir=models

# show how your gpu is going
$ gpustat -i

# show how your cpu and ram are going
$ htop

---

Credits

• Papers
• Analyzing cryptocurrency markets using python
• Q-trader
• Trading-Gym
• Tensor Trade

---

To do

• Bind the agent's output with an exchange place order API

t-1000's Issues

init_data

Might think of leaving df as pandas dataframe until end of step to speed up iterations through the list.

possible limitation

I possibly have two problems that is happening with me when training my model

1. Training with much more data than the default on this project is allocating much RAM and returning the reward mean to nan.
2. The training is limited to 250 iterations.

There is anything I can do to resolve those problems?

PC Specs:
12x CPU
120 GB RAM
Ubuntu 16.04

How Do I Push Into Production?

I was looking into these algorithms and was wondering how to bring RL into production. Google doesn't have anything on the subject. How do I put this algorithm into production?

TensorboardX failed to log hparams

Hi again @Draichi. I've moved on from https://github.com/Draichi/Stock-Trading-Visualization (although I must admit, it's always frustrating to leave behind a code base without getting it running).

This new code is throwing errors:

1. from ray.rllib.evaluation.episode import _flatten_action (cannot import _flatten_action)
2. when running main.py I get the following error and then my 3 trials terminate. I'm unclear if the below error is even related to the trials terminating to be honest.

ERROR logger.py:281 -- TensorboardX failed to log hparams. This may be due to an unsupported type in the hyperparameter values. Traceback (most recent call last): File "/Users/xxx/xxx/xxx/xxx/env/lib/python3.6/site-packages/ray/tune/logger.py", line 276, in _try_log_hparams hparam_dict=scrubbed_params, metric_dict=result) File "/Users/xxx/xxx/xxx/xxx/env/lib/python3.6/site-packages/tensorboardX/summary.py", line 103, in hparams ssi.hparams[k].number_value = v TypeError: array([0.e+00, 5.e-05]) has type numpy.ndarray, but expected one of: int, long, float 2020-05-26 23:42:29,556 WARNING util.py:137 -- The process_trial operation took 1.1992979049682617 seconds to complete, which may be a performance bottleneck.

About action_type

Hello. Thank you for sharing such an interesting project. After reading your program, I am a little confused about the approach in the function:_buy_or_sell() in core_env.py. Here's my question: why the decision to buy or sell is determined by inequality (like: if action_type <index / 2 + 1 ...) rather than directly using action_type?

What is the mechanism here? I think this will result in some assts that will not be used to buy or sell.

What I also want to know is the action_type defined here. If my assets list is ['BTC', 'ETH', 'LTC', 'BNB'], what will be in the action_type? Is it hold, buy BTC, sell BTC, buy ETH, sell ETH, buy LTC, sell LTC, buy BNB, sell BNB, 9 in total? Is there an actual order?

Thank you so much and look forward to your reply!

work in progress?

thank you for sharing your project. I've been testing several projects that use PPO as well as doing mine and so far could not get results, however when training yours I see steadily increasing rewards and the code makes sense. So I'm ready to contribute to it because as far as I see PPO algorithm tend not to overfit as others.
When I try to evaluate the checkpoints it generates I see gym's "not implemented" error. Is this because there is Work in Progress you are doing or am I doing something wrong?

(gym_trading) canermac-3:cryptocurrency_prediction apple$ rllib rollout /Users/apple/ray_results/default/PPO_Trading-v0_0_2019-04-16_12-38-51strcz86z/checkpoint_40/checkpoint-40 --run PPO --env Trading-v0 --steps 1000
 
lz4 not available, disabling sample compression. This will significantly impact RLlib performance. To install lz4, run `pip install lz4`.
2019-04-16 19:28:22,976	WARNING worker.py:1406 -- WARNING: Not updating worker name since `setproctitle` is not installed. Install this with `pip install setproctitle` (or ray[debug]) to enable monitoring of worker processes.
2019-04-16 19:28:22,980	INFO node.py:423 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-04-16_19-28-22_24310/logs.
2019-04-16 19:28:23,106	INFO services.py:363 -- Waiting for redis server at 127.0.0.1:22839 to respond...
2019-04-16 19:28:23,251	INFO services.py:363 -- Waiting for redis server at 127.0.0.1:11763 to respond...
2019-04-16 19:28:23,256	INFO services.py:760 -- Starting Redis shard with 0.86 GB max memory.
2019-04-16 19:28:23,305	INFO services.py:1384 -- Starting the Plasma object store with 1.29 GB memory using /tmp.
2019-04-16 19:28:24,135	WARNING ppo.py:172 -- FYI: By default, the value function will not share layers with the policy model ('vf_share_layers': False).
2019-04-16 19:28:24,375	INFO policy_evaluator.py:278 -- Creating policy evaluation worker 0 on CPU (please ignore any CUDA init errors)
2019-04-16 19:28:24.377440: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.random.categorical instead.
WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
/Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
2019-04-16 19:28:31,488	INFO multi_gpu_optimizer.py:74 -- LocalMultiGPUOptimizer devices ['/cpu:0']
(pid=24330) 2019-04-16 19:28:44,682	WARNING compression.py:20 -- lz4 not available, disabling sample compression. This will significantly impact RLlib performance. To install lz4, run `pip install lz4`.
(pid=24333) 2019-04-16 19:28:44,682	WARNING compression.py:20 -- lz4 not available, disabling sample compression. This will significantly impact RLlib performance. To install lz4, run `pip install lz4`.
(pid=24330) 2019-04-16 19:28:45,941	INFO policy_evaluator.py:278 -- Creating policy evaluation worker 2 on CPU (please ignore any CUDA init errors)
(pid=24330) 2019-04-16 19:28:45.942628: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
(pid=24333) 2019-04-16 19:28:45,928	INFO policy_evaluator.py:278 -- Creating policy evaluation worker 1 on CPU (please ignore any CUDA init errors)
(pid=24333) 2019-04-16 19:28:45.931715: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
(pid=24330) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
(pid=24330) Instructions for updating:
(pid=24330) Colocations handled automatically by placer.
(pid=24333) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
(pid=24333) Instructions for updating:
(pid=24333) Colocations handled automatically by placer.
(pid=24330) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
(pid=24330) Instructions for updating:
(pid=24330) Use tf.random.categorical instead.
(pid=24333) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/models/action_dist.py:114: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
(pid=24333) Instructions for updating:
(pid=24333) Use tf.random.categorical instead.
(pid=24330) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=24330) Instructions for updating:
(pid=24330) Use tf.cast instead.
(pid=24330) /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
(pid=24330)   "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
(pid=24333) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/array_grad.py:425: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=24333) Instructions for updating:
(pid=24333) Use tf.cast instead.
(pid=24333) /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
(pid=24333)   "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
(pid=24330) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=24330) Instructions for updating:
(pid=24330) Deprecated in favor of operator or tf.math.divide.
(pid=24333) WARNING:tensorflow:From /Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
(pid=24333) Instructions for updating:
(pid=24333) Deprecated in favor of operator or tf.math.divide.
Traceback (most recent call last):
  File "/Users/apple/miniconda3/envs/gym_trading/bin/rllib", line 10, in <module>
    sys.exit(cli())
  File "/Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/scripts.py", line 40, in cli
    rollout.run(options, rollout_parser)
  File "/Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/rollout.py", line 102, in run
    rollout(agent, args.env, num_steps, args.out, args.no_render)
  File "/Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/ray/rllib/rollout.py", line 169, in rollout
    env.render()
  File "/Users/apple/miniconda3/envs/gym_trading/lib/python3.6/site-packages/gym/core.py", line 108, in render
    raise NotImplementedError
NotImplementedError

Test data same as train?

Is Fig 1 a test run on the same data used to train the model?

PyStan StanModel error

Hello Draichi!

A few days ago I have downloaded your very interesting project but got issues to execute forecast.py (Pyhton 3.6, Windows 10) - following error message:

WARNING:pystan:No module named 'stanfit4anon_model_861b75c6337e237650a61ae58c4385ef_2841341000846931969'
WARNING:pystan:Something went wrong while unpickling the StanModel. Consider recompiling.

Traceback (most recent call last):
File "forecast.py", line 20, in
df_prophet.fit(df)
File "C:\Users\Andreas Prosswimmer\Anaconda3\envs\crypto_env\lib\site-packages\fbprophet\forecaster.py", line 1105, in fit
params = model.optimizing(**args)
File "C:\Users\Andreas Prosswimmer\Anaconda3\envs\crypto_env\lib\site-packages\pystan\model.py", line 511, in optimizing
fit = self.fit_class(data, seed)
AttributeError: 'StanModel' object has no attribute 'fit_class'

---

Lookig forward testing your code!

Training

�[1m�[31mUsage: python3 train.py [asset] [window] [episodes]�[0m

None
Stack trace:

File "C:\Users\Amelize\Downloads\DL\New folder\cryptocurrency_prediction-masterDraichi\cryptocurrency_prediction-master\train.py", line 5, in
exit()
Loaded 'main'
Traceback (most recent call last):
File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_local.py", line 125, in _run

_pydevd.main()

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_vendored\pydevd\pydevd.py", line 1752, in main

globals = debugger.run(setup['file'], None, None, is_module)

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_vendored\pydevd\pydevd.py", line 1099, in run

return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_vendored\pydevd\pydevd.py", line 1106, in _exec

pydev_imports.execfile(file, globals, locals) # execute the script

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_vendored\pydevd_pydev_imps_pydev_execfile.py", line 25, in execfile

exec(compile(contents+"\n", file, 'exec'), glob, loc)

File "C:\Users\Amelize\Downloads\DL\New folder\cryptocurrency_prediction-masterDraichi\cryptocurrency_prediction-master\train.py", line 5, in

exit()

File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib_sitebuiltins.py", line 26, in call

raise SystemExit(code)

SystemExit
:
None

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\ptvsd_launcher.py", line 119, in

vspd.debug(filename, port_num, debug_id, debug_options, run_as)

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd\debugger.py", line 37, in debug

run(address, filename, *args, **kwargs)

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_local.py", line 64, in run_file

run(argv, addr, **kwargs)

File "c:\program files (x86)\microsoft visual studio\2017\community\common7\ide\extensions\microsoft\python\core\Packages\ptvsd_local.py", line 127, in _run

daemon.exitcode = int(ex.code)

TypeError
:
int() argument must be a string, a bytes-like object or a number, not 'NoneType'

The thread 'MainThread' (0x1) has exited with code 0 (0x0).

a few issues

Hi Lucas!

I found your repo from Adam King's project. First of all, great work and respect.

While my mac is training, there is a few issues I ran into.

1. When installing env with conda on my mac, matplotlib=2.2.3=py35h0e0179f_0 cannot be found.
   I removed the last part py35h ... and it fixed the problem.
   So, my question is: did you test this in ubuntu env, or mac?

2. I tried to train multi model as instructed on readme, python train_multi_model.py gave me error not finding SYMBOL_1 (from vars.py)
   I looked into what might caused this, but I could not find where you pass in the SYMBOLs from vars.py
   any idea how to fix this? or I did something wrong

3. While I was looking into train_multi_model.py, I noticed in experiment_spec, there is a line "local_dir": '/home/lucas/Documents/cryptocurrency_prediction/tensorboard'.
   I could not run this train_multi_model.py file, but if I did, I imagine it would give me an error about folder not exist.

4. So instead I tried the second command you provided, # single pair python train.py \ --algo PPO \ --pair XRP/USDT \ and it is still running as I am writing this post. But I could not find where is the output tensorflow folder, so I cannot really track its progress.
   Again, any idea? or did I do something wrong...?

Again, great project! Thx!

not using gpu when training

I am trying to train my model but it's not using any of the gpus available, I am wondering if it's on purpose or an error on your code which I can't find.

OS: Ubuntu 16.04
GPU: 2x V100

Suggestion

To Speed up data you can use high speed database like rethinkdb or even use modin to speed up pandas
https://rethinkdb.com/docs/guide/python/
https://modin.readthedocs.io/en/latest/

Using docker and docker-compose you may get more code contributter because make it easier to start and test project.

integrate directly on a broker is better, you can train in a practice environment, this way robot will mitigate issues when trading in a real account.

Binance, bitfinix and any other brokers has apis you may use to do it. I personally suggest you to search for one who have websocket because it is faster than common api.

you can run the robot directly on google colabs to use high speed gpu and train a model.
google colabs use ubuntu based distribution so you can install any package there.

use ! to use terminal commands there like this:

!git clone [email protected]:Draichi/T-1000.git 

How to apply the model to live trade ？

How to apply the model to live trade ？ thank you

Decouple reward function

I check self.trades in core_env.py , self.trades data caculate is loss,but total reward display Positive profit! thank you

replace ta-lib by https://github.com/bukosabino/ta

May i suggest pandas-ta instead?
I find i quite powerful and easy integration with pandas dataframe. Lot's of indicators.
https://github.com/twopirllc/pandas-ta

working with Binary Option like IqOption

First of all thanks for this project.

I'm learning RL and it can help a lot.

I'm begginer with gym and openai env. So my doubt is:

Is it possible to convert it to trade binary option of 1 minute ?
For instance:
Iqoption binary options allow you do performe some actions in 1, 5 or more minutes and if you choose right direction you earn percentage of this transation. In this case the payout is 85%

I already tried convetional machine learning like, svm, random forest, svc, xgboost but all doesn't performe good, then I tried deep learning with lstm and keras to classify those actions. worked but also not performing good.
maximum accuracy I got was 63% and running it directly on practice account I could not have and good return.
(there is a websocket where I connect to run directly on practice account)

I noticed one strategy doesn't performe for long time, so because of that I'm trying to learn o openai gym.

another doubt

Could you pleaase teach me how gym works?
I fallowed same tutorial as you but I could not understand every thing

And I promise you I help you in your project if you teach me

AttributeError: 'StanModel' object has no attribute 'fit_class'

Trying to run the forecast.py on Ubuntu 18.04 (Mate desktop)- following error:

loading bitcoin from cache
loading nano from cache
loading ripple from cache

Using TensorFlow backend.
/root/anaconda3/envs/crypto_env/lib/python3.6/site-packages/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning:

inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

INFO:fbprophet.forecaster:Disabling yearly seasonality. Run prophet with yearly_seasonality=True to override this.
Traceback (most recent call last):
File "forecast.py", line 20, in
df_prophet.fit(df)
File "/root/anaconda3/envs/crypto_env/lib/python3.6/site-packages/fbprophet/forecaster.py", line 1003, in fit
dat, init=stan_init, iter=1e4, **kwargs)
File "/root/anaconda3/envs/crypto_env/lib/python3.6/site-packages/pystan/model.py", line 512, in optimizing
fit = self.fit_class(data, seed)
AttributeError: 'StanModel' object has no attribute 'fit_class'

ImportError: cannot import name '_flatten_action' from 'ray.rllib.evaluation.episode'

Whenever I am trying to run main.py, I am getting this import error. Does anyone know how to fix this?

Traceback (most recent call last):
File "main.py", line 24, in
from t_1000 import T1000
File "/Users/tonmoyroy/CODE/T-1000/t_1000/init.py", line 1, in
from t_1000.application import T1000
File "/Users/tonmoyroy/CODE/T-1000/t_1000/application/init.py", line 1, in
from t_1000.application.core import T1000
File "/Users/tonmoyroy/CODE/T-1000/t_1000/application/core.py", line 13, in
from ray.rllib.evaluation.episode import _flatten_action
ImportError: cannot import name '_flatten_action' from 'ray.rllib.evaluation.episode' (/Users/tonmoyroy/opt/anaconda3/envs/tf/lib/python3.7/site-packages/ray/rllib/evaluation/episode.py)

T2000 - the T1000 reborn ?

R u really refactoring it ?
Need any help ?

Illegal instruction (core image recorded)

Could you help me ?

I installed all necessary package and used

# create env
conda env create -f t-1000.yml
# activate it
conda activate t-1000

and started with

python main.py -a btc eth bnb -c usd

but only this error is shown:

Illegal instruction (core image recorded)

did you already had this error?

potential lookahead bias

Nice work, thanks for sharing.
I was looking to your step function:

    def step(self, action):
        # Execute one time step within the environment
        self._take_action(action)
        self.current_step += 1

        net_worth_and_buyhold_mean = (self.net_worth + self.buy_and_hold) / 2
        reward = (self.net_worth - self.buy_and_hold) / net_worth_and_buyhold_mean
        done = self.net_worth <= 0 or self.balance <= 0 or self.current_step >= len(self.df1_features.loc[:, 'open'].values) -1
        obs = self._next_observation()

return obs, reward, done, {}

this calculate reward from values saved one step back.
Don't you think this is a lookahead bias?