yamatokataoka / reinforcement-learning-replications Goto Github PK

I'm thinking using rl-replicas on this project.
https://github.com/yamatokataoka/learning-from-human-preferences

In that case, learn function should be customisable so swapping a reward function provided from an OpenAI gym environment with a learned reward function.

I also don't think Inheritance is easy to understand and extensible for the future RL implementations.

Todos

• Refactor collect_one_epoch_experience functions
  • #64
  • #66
• Use the same data class for OneEpochExperience #71
• Extract collect_one_epoch_experience functions as a sampler

The point is how much I can bring these two functions on on_policy_algorithm.py and off_policy_algorithm.py become closer together.

• DIY for scipy function

Now, the VPG model has ActorCriticPolicy for actor (policy) and critic (value function).
It's good to separate them for the separation of concerns and easy to extend policy and value functions in the near future.

Implement TRPO algorithm.

Refer #59

Saving model functionality should be a separated function.

Lint code before increasing the amount of code

• Lint code with isort, black, flake8, mypy
• Automate checking and linting

It makes easy to read code and folder structure consistent.

• Use built-in types in Python 3.10

Related
#75

• some warning logs should be debug

• note here how to run RL experiment with Spinning Up on Colab to compare results between Spinning Up's and mine.

The value loss is increasing while training compared to OpenAI Spinning Up implementation.

the current implementation is increasing from 178 to 451, whileSpinning Up's value loss is decreasing from 253 to 171 for example.

for now, I confirmed below parameters are the same between the implementations:

• the number of the network parameters: policy: 4610, value function: 4545
• network architecture (two hidden layers with 64 units)
• total environment interactions
• number of value function updates
• learing rate both on policy and value function
• gym environment: CartPole-v0

Implement Twin Delayed Deep Deterministic Policy Gradient (TD3).

Should use the same data class.
#59 (comment)

Provide some useful functions to return data. For example, returning flatted observations.

Add integration test for all RL algorithms

• VPG
• TRPO
• PPO
• DDPG
• TD3

Test environments with either discrete or continuous action spaces.

I'm gonna use the src layout which is recommended in a pytest document.

This layout prevents a lot of common pitfalls and has many benefits such as:

• I can force to test the installed version of our package, not the local code from the repository,
• good tool integration with tox, pytest which I'm planning to use

You can find more on:

• https://blog.ionelmc.ro/2014/05/25/python-packaging/
• https://hynek.me/articles/testing-packaging/

you don't need to do anything with tox but pytest.

https://www.stefaanlippens.net/pytest-disable-log-capturing.html
use these options

pytest --capture=no --log-cli-level=INFO

I found it's still printing captured logs.
in that case, use --show-capture=no

and change the log format with --log-cli-format
https://docs.pytest.org/en/latest/how-to/logging.html

Implement the Gaussian policy as one of the two most common kinds of stochastic policies.

• transition from setup.py to pyproject.toml and setup.cfg to avoid executing arbitrary scripts and boilerplate code.
• single-source the package version
• set up GitHub Actions for PyPI and TestPyPI

• transition from setup.py to pyproject.toml and setup.cfg to avoid executing arbitrary scripts and boilerplate code.
• single-source the package version

Bump up Python version from 3.6 to 3.10

experiment stats

• epochs (number of updates policy): 1000
• steps_per_epoch: 4000
• n_value_gradients: 80
• gamma: 0.99
• gae_lambda: 0.97
• value function learning_rate: 1e-3
• policy learning_rate: 3e-4
• seed: (0 ~ 2)

in #75, I bumped up the version to 3.10 but I found it's inconvenience when I run it in Colab.
As of 7.2022, the Python version in Colab is 3.7.
Thus, I will bump down to 3.7.

At the end of the training, it will store policy model for inference later.

My initial implementation of the VPG tooks around 300 sec to excute 200,000 environment interations.
It's about 2x slower than OpenAI Spinning Up implementation.

ran this with cProfile module.

import gym
from rl_replicas.vpg.vpg import VPG
from rl_replicas.common.policies import ActorCriticPolicy

env = gym.make('CartPole-v0')

model = VPG(ActorCriticPolicy, env, seed=1)

model.learn()

07102020_performance_spiningup_vpg.txt

07102020_performance_rl_replicas_vpg.txt

It can introduce common functions and extend for VPG and TRPO.

separate network and optimizer.

Refer #59

ex. epochs -> num_epochs

let's visualise experiment result.

now, vpg uses list store experiences like observation and reward.

it seems deque is much more efficient.
https://stackoverflow.com/questions/47493446/should-i-use-a-python-deque-or-list-as-a-stack/47493474

research python logging for machine learning library.
improve logging.

For reply buffer
https://stackoverflow.com/questions/24245057/pop-multiple-items-from-the-beginning-and-end-of-a-list

The policies should determine types of policy; deterministic or stochastic, and categorical or normal distribution.
It's the preparation of implementing DDPG and continuous action space support for on-policy algorithms.

• it doesn't need to publish to TestPyPI
• add release flow in README

Try out the distribution tools and process without worrying about affecting the real index.

• set up GitHub Actions for PyPI and TestPyPI

Implement Proximal Policy Optimization (PPO) (by clipping)

• Set up mypy for type checking.
• Resolve all errors and warnings from mypy.
• Remove unnecessary type hints.