continualai / avalanche-rl

Avalanche fork adding RL support

ContinualAI - Jekyll Website now DEPRECATED

We moved to GitBook. If you want to help us maintain the new website, please send and email to [email protected].

Continual AI is the first hub on Continual / Lifelong Deep Learning in AI! :-) The aim of the project is to provide a starting point for researchers, developers and AI enthusiasts who share an interest or are willing to learn more and/or contribute to Continual / Lifelong Learning. We are building an open-source, collaborative wiki at continualai.org as well as creating a community of CL enthusiasts! Join us today on slack! :D

How to contribute

1. Star the project :-)

2. Join our community on Slack: https://continualai.herokuapp.com/

3. Start making changes to the *.md files from the browser (use the 'Preview' button)

4. Commit the changes!

How to contribute (like a pro)

1. Star the project :-)

2. Join our community on Slack: https://continualai.herokuapp.com/

3. Fork the repo on GitHub and clone it locally

4. Enter the folder:

   cd website-wiki

5. If you don't have gem and bundler installed:

   apt-get install rubygems
gem install bundler

6. Install Ruby gems:

   bundle install

7. Start Jekyll server:

   jekyll serve --incremental

8. Now you can start making changes on the see the result in your browser at http://localhost:4000/

9. Make a Pull Request (with only the .md or original .html files)! :D

avalanche-rl's Issues

'DQNStrategy' object has no attribute 'clock'

Hello,

I am writing to inform you that there is a problem between the avalanche-rl and DQNStrategy.
Unfortunately, in avalanche-rl when a strategy uses EWC plugin, below error will be raise:
Error for Compute EWC penalty and add it to the loss.
exp_counter = strategy.clock.train_exp_counter
if exp_counter == 0:
return
'DQNStrategy' object has no attribute 'clock'.

It needs to add 'clock' attribute in 'DQNStrategy'. I will appreciate if you update this bug in the avalanhe-rl.

avalanche-lib==0.3.1

Add fancy badges in Readme

The repo doesn't look too pretty atm, we could use some Ops

PPO Implementation

Implement PPO algorithm from scratch following the framework guidelines so that we can easily integrate it with all available features.

Add citation to Avalanche-RL paper

HI Niccolò! Please add a ref in the main readme about our recent pub on Avalanche-RL :)
You can get inspired on how we did it in the main avalanche repo!

Moving benchmarks to avalanche

I think Avalanche is ready to integrate the CRL benchmarks.

@NickLucche do you agree with moving the benchmarks in avalanche? Once they are in the main repository, they would be considered "stable" and possibly receive more attention. There are a lot of people using Avalanche just for the benchmarks that may not know about avalanche-rl.

Support new Avalanche version

This new version (ContinualAI/avalanche@4380ebc) includes a refactoring of the RLScenario and RLExperience using the newly defined benchmark stack components.

In order to support it, we must also modify our RLStrategy to inherit from the "BaseTemplate" (https://github.com/ContinualAI/avalanche/tree/master/avalanche/training/templates), this is the newest "hierarchy in town" that just joined the Avalanche training module.

This integration should represent a significant step toward an eventual merge of Avalanche RL into Avalanche.

Error in forgetting_metrics

Hello,
I would like to save logs of some metrics such as the forgetting_metrics() using CSVLogher class, but it can not save the metrics in CSV file! Do you know what is the problem with this issue?

The code used for the logger was according to the below:

from avalanche.evaluation.metrics import (
forgetting_metrics,
)

text_logger = TextLogger(open("log.txt", "a"))
interactive_logger = InteractiveLogger()
csv_logger = CSVLogger()
eval_plugin = EvaluationPlugin(
forgetting_metrics(experience=True, stream=True),
loggers=[interactive_logger, text_logger, csv_logger], or loggers=[csv_logger]
collect_all=True,
)

Also, I get the below error:

AttributeError: 'DQNStrategy' object has no attribute 'mb_y'

Thanks in advance for your help.

avalanche-lib==0.3.1

Support latest version of Avalanche

Update all different classes to match the current structure in Avalanche, introducting Templates and re-use possible coomponents.
Starting from the work developed in #8, the first goal is to update Avalanche-RL without adding new features.

Memory Management

Hello,
In avalanhce-rl, after 1000 experiences and 1000 steps per experience, the below error will be risen. It is necessary to manage memory during training correctly.

torch.cuda.OutOfMemoryError
/Avalanche/avalanche-rl/avalanche_rl/training/strategies/rl_base_strategy.py", line 377, in train_exp
self.loss.backward()

Thanks.

Error in evaluation of a strategy

Hello,

I am writing to inform you that unfortunately there is a problem when a scenario will be evaluated after training. In other words, when the below function will be called:

strategy.eval(scenario.test_stream)

This error will be shown:

AttributeError: 'RLScenario' object has no attribute 'test_stream'

Is there any update or solution for this issue?

Thanks in advance

avalanche-lib==0.3.1

Restructure file location

Current module structure is borrowed from avalanche but file location for rl utils isn't super-obvious. I believe we should move stuff around in order to make it more predictable for the user, probably with a less-nested and simpler structure.

Run examples in github actions and check exit code

Error related to avalanche.training.strategies

I have to work on CRL for path generation of a drone. For learning, I tried to run 'simple_dqn.py' from this repository but it shows error related to avalanche training strategies. I am using version avalanche version 0.3.1 and pyhton 3.9

FIX CI

we should fix the CI on this repo.

Also, if the examples reproduce some papers, it would be nice to state the expected results so that we can check reproducibility with new commits, like we want to do with the reproducible-cl repo.