universome / firelab

Experimental framework for running pytorch experiments

firelab (version 0.0.26)

About

Framework for running DL experiments with pytorch. Provides the following useful stuff:

• parallel hyperparameters optimization
• allows to start/continue your experiment with easy commands from yml config file
• easier to save checkpoints, write logs and visualize training
• useful utils for HP tuning and working with pytorch (look them up in utils.py)

Installation

pip install firelab

Future plans

[ ] Run in daemon. [ ] Implement firelab ls command [ ] Easier profiling (via contexts?) [ ] There are some interseting features in https://github.com/vpj/lab. [ ] Add commit hash to summary [ ] Create new branch/commit for each experiment? [ ] More meaningful error messages. [ ] Does model release GPU after training is finished (when we do not use HPO)? [ ] Proper handling of errors in HPO: should we fail on the first exception? Should we try/catch result.get() in process pool? [x] Make trainers run without config.firelab, this will make it possible to run trainer from python [ ] Does continue_from_iter work?

Useful commands:

• firelab ls — lists all running experiments
• firelab start / firelab stop / firelab pause / firelab continue — starts/stops/pauses/continues experiments

Useful classes

• BaseTrainer — controls the experiment: loads data, runs/stops training, performs logging, etc

Cool staff firelab can do:

• Reduces amount of boilerplate code you write for training/running experiments
• Keep all experiment arguments and hyperparameters in a expressive config files
• Visualize your metrics with tensorboard through tensorboardX
• Save checkpoints and logs with ease.
• Fixes random seeds for you by default (in numpy, pytorch and random). Attention: if you use other libs with other random generators, you should fix random seeds by yourself (we recommend taking it from hyperparams)

Usage:

Configs

Besides your own configs, firelab adds its inner staff, which you can use or change as hyperparameter:

• name of the experiment
• random_seed

Experiment name determines where config is. Experiment name can't duplicate each other.

TODO

• Interactive config builder
• Clone experiment/config
• Add examples with several trainers in them
• Why do we pass both exp_dir and exp_name everywhere in manager.py? We should care only about exp_path I suppose?
• Looks like we do not need the dublicating logic of directories creation in manager.py anymore since it is in BaseTrainer
• Rename BaseTrainer into Trainer?

firelab's Issues

Do not start an experiment with unstaged changes

It's hard to recover what was the config if you are constantly changing the things and running experiments without committing the changes. You can squash the commits after that, so git history won't be dirty

Run validation via callbacks

Use callbacks for on_epoch_done() and on_iter_done() to run validation instead of doing weird computations inside _try_to_validate()

Config inheritance

There are a lot of copypasting in configs. We should implement inheritance

Recursive set in Config

.get() and .has() methods can work recursively (by splitting property name with dots). Add analogous recursive .set() method.

GPU is not released for normal (non-HPO) experiment after training is done