jexio / fulmo

A very general, feature-rich template for rapid and scalable ML experimentation with best practices.

fulmo

Template to start your deep learning project based on PyTorchLightning for rapid prototyping.

Contents

• fulmo
  • Why Lightning + Hydra + Albumentations?
  • Features
  • Project Structure
  • Workflow
  • Experiment Tracking
  • Quick start
  • Todo
  • Credits

Why Lightning + Hydra + Albumentations?

• PyTorch Lightning provides great abstractions for well structured ML code and advanced features like checkpointing, gradient accumulation, distributed training, etc.
• Hydra provides convenient way to manage experiment configurations and advanced features like overriding any config parameter from command line, scheduling execution of many runs, etc.
• Albumentations (Optional) provides many image augmentation. Albumentations supports all common computer vision tasks such as classification, semantic segmentation, instance segmentation, object detection, and pose estimation.

Features

Pipelines based on hydra-core configs and PytorchLightning modules

• Predefined folder structure. Modularity: all abstractions are split into different submodule
• Rapid Experimentation. Thanks to automating pipeline with config files and hydra command line superpowers
• Little Boilerplate. So pipeline can be easily modified
• Main Configuration. Main config file specifies default training configuration
• Experiment Configurations. Stored in a separate folder, they can be composed out of smaller configs, override chosen parameters or define everything from scratch
• Experiment Tracking. Many logging frameworks can be easily integrated
• Logs. All logs (checkpoints, data from loggers, chosen hparams, etc.) are stored in a convenient folder structure imposed by Hydra
• Automates PyTorch Lightning training pipeline with little boilerplate, so it can be easily modified
• Augmentations with albumentations described in a yaml config.
• Support of timm models, pytorch-optimizer and TorchMetrics
• Exponential Moving Average for a more stable training, and Stochastic Moving Average for a better generalization and just overall performance.

Project structure

The directory structure of new project looks like this:

├── src
│   ├── fulmo
│   │   ├── callbacks               <- PyTorch Lightning callbacks
│   │   ├── core                    <- PyTorch Lightning models
│   │   ├── datasets                <- PyTorch datasets
│   │   ├── losses                  <- PyTorch losses
│   │   ├── metrics                 <- PyTorch metrics  
│   │   ├── models                  <- PyTorch model architectures
│   │   ├── optimizers              <- PyTorch optimizers
│   │   ├── readers                 <- Data readers
│   │   ├── samples                 <- PyTorch samplers
│   │   ├── schedulers              <- PyTorch schedulers
│   │   └── utils
├── tests
│   ├── test_fulmo                  <- Tests
│
├── .bumpversion.cfg
├── .darglint
├── .gitignore
├── .pre-commit-config.yaml <- Configuration of hooks for automatic code formatting
├── CHANGELOG.md
├── mypy.ini
├── noxfile.py
├── poetry.lock             <- File for installing python dependencies
├── pyproject.toml          <- File for installing python dependencies
├── README.md
└── tasks.py

Workflow

1. Write your PyTorch model
2. Write your PyTorch Lightning datamodule
3. Write your experiment config, containing paths to your model and datamodule
4. Run training with chosen experiment config:
   

python train.py +experiment=experiment_name

Experiment Tracking

PyTorch Lightning provides built in loggers for Weights&Biases, Neptune, Comet, MLFlow, Tensorboard and CSV. To use one of them, simply add its config to configs/logger and run:

python train.py logger=logger_name

Quickstart

First, install dependencies

pip install fulmo | poetry add fulmo

Second, create your project

See examples folder.

Next, you can train model with default configuration without logging

python train.py

Or you can train model with chosen experiment config

python train.py +experiment=experiment_name

Resume from a checkpoint

# checkpoint can be either path or URL
# path should be either absolute or prefixed with `${work_dir}/`
# use quotes '' around argument or otherwise $ symbol breaks it
python train.py '+trainer.resume_from_checkpoint=${work_dir}/logs/runs/2021-06-23/16-50-49/checkpoints/last.ckpt'

TODO

• Data version control
• Metric learning pipeline
• Integrate Cross-Batch Memory for Embedding Learning (XBM)
• Image augmentation policies

Credits

• This package was created with Cookiecutter and the fedejaure/cookiecutter-modern-pypackage project template.
• hobogalaxy/lightning-hydra-template
• Erlemar/pytorch_tempest
• bonlime/pytorch-tools