druce / rl

Deep Reinforcement Learning For Trading

Druce Vertes

Tech veteran, quant analyst, data scientist, developer, tech strategist, CTO with leading hedge funds and Wall Street firms since 1992, consulting on machine learning, alternative data, predictive analytics.

"A life well lived is a series of personal obsessions shared without expectation of an audience." - some guy un1crom on Hacker News

• Résumé
• LinkedIn
• Twitter
• Email

rl's Issues

No issue ---resolved it

Trading With RL Pickle Error - TypeError: can't pickle _thread.RLock objects

Love the work you've posted! Super thorough.

I'm trying to implement the Trading with RL notebook into Google Colab but running into an issue. I will document my moves here for reproducibility.

Basically, I copied, cell for cell, the notebook from this repository into a Google Colab notebook.

First, you have to install backtrader into the environment. I did this by putting this at the very beginning of the imports:

try:
    import backtrader as bt
except:
    print('Backtrader not installed yet. Installing now...')
    !pip install backtrader
    print('Backtrader installed.')
    print('Restart and Run All now.')
    exit()

This way, it'll prompt the user to Restart and Run All once Backtrader is installed.

Second issue I figured out was to make sure this notebook is being run with eager execution disabled, so I added:

from tensorflow.compat import v1
v1.disable_eager_execution()

...to the imports at the top as well.

Now my issue is this error:

TypeError                                 Traceback (most recent call last)
<ipython-input-21-e8262df5188b> in <module>()
     38 
     39     if e and (e+1) % agent.save_interval == 0:
---> 40         agent.save()
     41 
     42 elapsed_time = time.time() - start_time

<ipython-input-20-21179c258b19> in save(self)
    149         self.predict_model.save("%s_predict.h5" % fullname)
    150         # can't save / load train model due to custom loss
--> 151         pickle.dump(self, open("%s.p" % fullname, "wb"))
    152 
    153     def load(filename, memory=True):

TypeError: can't pickle _thread.RLock objects

This happens when the cell with this at the beginning is run (i.e. the cell after the class REINFORCE_Agent(Agent): cell) :

N_EPISODES = 2000
ticks_per_episode = 1256
nstocks = 1
lag = 1

Googling that error, I found this answer that might help? But hoping to get some help troubleshooting this one? Would greatly appreciate it as I can't wait to get this working online.

Thanks!

DQN_Agent in shm

This is a work of art, it would be wonderful to find examples inspired by you with PPO and other agents.

I uncommented and tried to test the DQN_Agent in https://github.com/druce/rl/blob/master/Trading_with_RL.ipynb https://github.com/druce/rl/blob/master/Trading%20with%20RL.ipynb because even if it is slow, it still has great value for those who are learning, like myself. I only removed the “import resources” and its dependents because I am running it on Windows

I tried "
N_EPISODES = 10000
ticks_per_episode = 1256
nstocks = 1

env = Market(shm_market_gen,
nstocks=1,
episode_length=ticks_per_episode)

agent = DQN_Agent(state_size=nstocks*32,
action_size=2,
)

start_time = time.time()

for e in range(N_EPISODES):
agent.run_episode()
agent.score_episode(e, N_EPISODES)

if e and (e+1) % agent.save_interval == 0:
    agent.save()

elapsed_time = time.time() - start_time
print("Train time: ", elapsed_time)
agent.rlplot()"
Unfortunately, I got:

Traceback (most recent call last):
File "C:\Users\guest\AppData\Roaming\Python\Python310\site-packages\IPython\core\interactiveshell.py", line 3526, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "C:\Users\guest\AppData\Local\Temp\ipykernel_11800\4180752183.py", line 19, in
agent.run_episode()
File "C:\Users\guest\AppData\Local\Temp\ipykernel_11800\165701735.py", line 198, in run_episode
self.action = self.act(self.state.reshape([1, self.state_size]))
ValueError: cannot reshape array of size 8 into shape (1,32)"