lucadellalib / actorch

Deep reinforcement learning framework for fast prototyping based on PyTorch

Welcome to actorch, a deep reinforcement learning framework for fast prototyping based on PyTorch. The following algorithms have been implemented so far:

• REINFORCE
• Advantage Actor-Critic (A2C)
• Actor-Critic Kronecker-Factored Trust Region (ACKTR)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Advantage-Weighted Regression (AWR)
• Deep Deterministic Policy Gradient (DDPG)
• Distributional Deep Deterministic Policy Gradient (D3PG)
• Twin Delayed Deep Deterministic Policy Gradient (TD3)
• Soft Actor-Critic (SAC)

---

💡 Key features

• Support for OpenAI Gymnasium environments
• Support for custom observation/action spaces
• Support for custom multimodal input multimodal output models
• Support for recurrent models (e.g. RNNs, LSTMs, GRUs, etc.)
• Support for custom policy/value distributions
• Support for custom preprocessing/postprocessing pipelines
• Support for custom exploration strategies
• Support for normalizing flows
• Batched environments (both for training and evaluation)
• Batched trajectory replay
• Batched and distributional value estimation (e.g. batched and distributional Retrace and V-trace)
• Data parallel and distributed data parallel multi-GPU training and evaluation
• Automatic mixed precision training
• Integration with Ray Tune for experiment execution and hyperparameter tuning at any scale
• Effortless experiment definition through Python-based configuration files
• Built-in visualization tool to plot performance metrics
• Modular object-oriented design
• Detailed API documentation

---

🛠️️ Installation

For Windows, make sure the latest Visual C++ runtime is installed.

Using Pip

First of all, install Python 3.6 or later. Open a terminal and run:

pip install actorch

Using Conda virtual environment

Clone or download and extract the repository, navigate to <path-to-repository>/bin and run the installation script (install.sh for Linux/macOS, install.bat for Windows). actorch and its dependencies (pinned to a specific version) will be installed in a Conda virtual environment named actorch-env.

NOTE: you can directly use actorch-env and the actorch package in the local project directory for development (see For development).

Using Docker (Linux/macOS only)

First of all, install Docker and NVIDIA Container Runtime. Clone or download and extract the repository, navigate to <path-to-repository>, open a terminal and run:

docker build -t <desired-image-name> .                  # Build image
docker run -it --runtime=nvidia <desired-image-name>    # Run container from image

actorch and its dependencies (pinned to a specific version) will be installed in the specified Docker image.

NOTE: you can directly use the actorch package in the local project directory inside a Docker container run from the specified Docker image for development (see For development).

From source

First of all, install Python 3.6 or later. Clone or download and extract the repository, navigate to <path-to-repository>, open a terminal and run:

pip install .

For development

First of all, install Python 3.6 or later and Git. Clone or download and extract the repository, navigate to <path-to-repository>, open a terminal and run:

pip install -e .[all]
pre-commit install -f

This will install the package in editable mode (any change to the package in the local project directory will automatically reflect on the environment-wide package installed in the site-packages directory of your environment) along with its development, test and optional dependencies. Additionally, it installs a git commit hook. Each time you commit, unit tests, static type checkers, code formatters and linters are run automatically. Run pre-commit run --all-files to check that the hook was successfully installed. For more details, see pre-commit's documentation.

---

▶️ Quickstart

In this example we will solve the OpenAI Gymnasium environment CartPole-v1 using REINFORCE. Copy the following configuration in a file named REINFORCE_CartPole-v1.py (with the same indentation):

import gymnasium as gym
from torch.optim import Adam

from actorch import *


experiment_params = ExperimentParams(
    run_or_experiment=REINFORCE,
    stop={"training_iteration": 50},
    resources_per_trial={"cpu": 1, "gpu": 0},
    checkpoint_freq=10,
    checkpoint_at_end=True,
    log_to_file=True,
    export_formats=["checkpoint", "model"],
    config=REINFORCE.Config(
        train_env_builder=lambda **config: ParallelBatchedEnv(
            lambda **kwargs: gym.make("CartPole-v1", **kwargs),
            config,
            num_workers=2,
        ),
        train_num_episodes_per_iter=5,
        eval_freq=10,
        eval_env_config={"render_mode": None},
        eval_num_episodes_per_iter=10,
        policy_network_model_builder=FCNet,
        policy_network_model_config={
            "torso_fc_configs": [{"out_features": 64, "bias": True}],
        },
        policy_network_optimizer_builder=Adam,
        policy_network_optimizer_config={"lr": 1e-1},
        discount=0.99,
        entropy_coeff=0.001,
        max_grad_l2_norm=0.5,
        seed=0,
        enable_amp=False,
        enable_reproducibility=True,
        log_sys_usage=True,
        suppress_warnings=True,
    ),
)

Open a terminal in the directory where you saved the configuration file and run (if you installed actorch in a virtual environment, you first need to activate it, e.g. conda activate actorch-env if you installed actorch using Conda):

pip install gymnasium[classic_control]  # Install dependencies for CartPole-v1
actorch run REINFORCE_CartPole-v1.py    # Run experiment

NOTE: training artifacts (e.g. checkpoints, metrics, etc.) are saved in nested subdirectories. This might cause issues on Windows, since the maximum path length is 260 characters. In that case, move the configuration file (or set local_dir) to an upper level directory (e.g. Desktop), shorten the configuration file name, and/or shorten the algorithm name (e.g. DistributedDataParallelREINFORCE.rename("DDPR")).

Wait for a few minutes until the training ends. The mean cumulative reward over the last 100 episodes should exceed 475, which means that the environment was successfully solved. You can now plot the performance metrics saved in the auto-generated TensorBoard (or CSV) log files using Plotly (or Matplotlib):

pip install actorch[vistool]  # Install dependencies for VisTool
cd experiments/REINFORCE_CartPole-v1/<auto-generated-experiment-name>
actorch vistool plotly tensorboard

You can find the generated plots in plots.

Congratulations, you ran your first experiment!

See examples for additional configuration file examples.

HINT: since a configuration file is a regular Python script, you can use all the features of the language (e.g. inheritance).

---

🔗 Useful links

• Introduction to deep reinforcement learning

• Hyperparameter tuning with Ray Tune and Optuna integration

• Logging with Ray Tune

• Monitoring jobs with Ray Dashboard

• Setting up a cluster with Ray Cluster

---

@ Citation

@misc{DellaLibera2022ACTorch,
  author = {Luca Della Libera},
  title = {{ACTorch}: a Deep Reinforcement Learning Framework for Fast Prototyping},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/lucadellalib/actorch}},
}

---

📧 Contact

[email protected]

---