yamatokataoka / reinforcement-learning-replications Goto Github PK
View Code? Open in Web Editor NEWReinforcement Learning Replications is a set of Pytorch implementations of reinforcement learning algorithms.
License: MIT License
Reinforcement Learning Replications is a set of Pytorch implementations of reinforcement learning algorithms.
License: MIT License
I'm thinking using rl-replicas on this project.
https://github.com/yamatokataoka/learning-from-human-preferences
In that case, learn
function should be customisable so swapping a reward function provided from an OpenAI gym environment with a learned reward function.
I also don't think Inheritance is easy to understand and extensible for the future RL implementations.
Todos
collect_one_epoch_experience
functions
collect_one_epoch_experience
functions as a samplerThe point is how much I can bring these two functions on on_policy_algorithm.py and off_policy_algorithm.py become closer together.
Now, the VPG model has ActorCriticPolicy for actor (policy) and critic (value function).
It's good to separate them for the separation of concerns and easy to extend policy and value functions in the near future.
Implement TRPO algorithm.
Refer #59
Saving model functionality should be a separated function.
Lint code before increasing the amount of code
It makes easy to read code and folder structure consistent.
Related
#75
The value loss is increasing while training compared to OpenAI Spinning Up implementation.
the current implementation is increasing from 178 to 451, whileSpinning Up's value loss is decreasing from 253 to 171 for example.
for now, I confirmed below parameters are the same between the implementations:
Implement Twin Delayed Deep Deterministic Policy Gradient (TD3).
Should use the same data class.
#59 (comment)
Provide some useful functions to return data. For example, returning flatted observations.
Add integration test for all RL algorithms
Test environments with either discrete or continuous action spaces.
I'm gonna use the src layout which is recommended in a pytest document.
This layout prevents a lot of common pitfalls and has many benefits such as:
You can find more on:
tox --workdir /content/tox
you don't need to do anything with tox but pytest.
https://www.stefaanlippens.net/pytest-disable-log-capturing.html
use these options
pytest --capture=no --log-cli-level=INFO
I found it's still printing captured logs.
in that case, use --show-capture=no
and change the log format with --log-cli-format
https://docs.pytest.org/en/latest/how-to/logging.html
Implement the Gaussian policy as one of the two most common kinds of stochastic policies.
Bump up Python version from 3.6 to 3.10
experiment stats
in #75, I bumped up the version to 3.10 but I found it's inconvenience when I run it in Colab.
As of 7.2022, the Python version in Colab is 3.7.
Thus, I will bump down to 3.7.
At the end of the training, it will store policy model for inference later.
My initial implementation of the VPG tooks around 300 sec to excute 200,000 environment interations.
It's about 2x slower than OpenAI Spinning Up implementation.
ran this with cProfile module.
import gym
from rl_replicas.vpg.vpg import VPG
from rl_replicas.common.policies import ActorCriticPolicy
env = gym.make('CartPole-v0')
model = VPG(ActorCriticPolicy, env, seed=1)
model.learn()
It can introduce common functions and extend for VPG and TRPO.
separate network and optimizer.
Refer #59
ex. epochs -> num_epochs
let's visualise experiment result.
now, vpg uses list store experiences like observation and reward.
it seems deque is much more efficient.
https://stackoverflow.com/questions/47493446/should-i-use-a-python-deque-or-list-as-a-stack/47493474
research python logging for machine learning library.
improve logging.
The policies should determine types of policy; deterministic or stochastic, and categorical or normal distribution.
It's the preparation of implementing DDPG and continuous action space support for on-policy algorithms.
Try out the distribution tools and process without worrying about affecting the real index.
Implement Proximal Policy Optimization (PPO) (by clipping)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.