csuhyd / lamda-pilot

🎉 PILOT: A Pre-trained Model-Based Continual Learning Toolbox

🎉Introduction • 🌟Methods Reproduced • 📝Reproduced Results
☄️How to Use • 👨‍🏫Acknowledgments • 🤗Contact

---

🎉 Introduction

Welcome to PILOT, a pre-trained model-based continual learning toolbox [Paper]. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT also fits typical class-incremental learning algorithms (e.g., DER, FOSTER, and MEMO) within the context of pre-trained models to evaluate their effectiveness.

If you use any content of this repo for your work, please cite the following bib entry:

@article{sun2023pilot,
  title={PILOT: A Pre-Trained Model-Based Continual Learning Toolbox},
  author={Sun, Hai-Long and Zhou, Da-Wei and Ye, Han-Jia and Zhan, De-Chuan},
  journal={arXiv preprint arXiv:2309.07117},
  year={2023}
}

@article{zhou2024continual,
  title={Continual Learning with Pre-Trained Models: A Survey},
  author={Zhou, Da-Wei and Sun, Hai-Long and Ning, Jingyi and Ye, Han-Jia and Zhan, De-Chuan},
  journal={arXiv preprint arXiv:2401.16386},
  year={2024}
}

🌟 Methods Reproduced

• FineTune: Baseline method which simply updates parameters on new tasks.
• iCaRL: iCaRL: Incremental Classifier and Representation Learning. CVPR 2017 [paper]
• Coil: Co-Transport for Class-Incremental Learning. ACMMM 2021 [paper]
• DER: DER: Dynamically Expandable Representation for Class Incremental Learning. CVPR 2021 [paper]
• FOSTER: Feature Boosting and Compression for Class-incremental Learning. ECCV 2022 [paper]
• MEMO: A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning. ICLR 2023 Spotlight [paper]
• SimpleCIL: Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need. arXiv 2023 [paper]
• L2P: Learning to Prompt for Continual Learning. CVPR 2022 [paper]
• DualPrompt: DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning. ECCV 2022 [paper]
• CODA-Prompt: CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning. CVPR 2023 [paper]
• ADAM: Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need. arXiv 2023 [paper]
• RanPAC: RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS 2023 [paper]
• Ease: Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. CVPR 2024 [paper]

Intended authors are welcome to contact us to reproduce your methods in our repo. Feel free to merge your algorithm into PILOT if you are using our codebase!

📝 Reproduced Results

CIFAR-100

ImageNet-R

For exemplar parameters, Coil, DER, iCaRL, MEMO, and FOSTER set the fixed_memory option to false and retain the memory_size of 2000 for CIFAR100, while setting fixed_memory option to true and retaining the memory_per_class of 20 for ImageNet-R. On the contrary, other models are exemplar-free.

☄️ how to use

🕹️ Clone

Clone this GitHub repository:

git clone https://github.com/sun-hailong/LAMDA-PILOT
cd LAMDA-PILOT

🗂️ Dependencies

1. torch 2.0.1
2. torchvision 0.15.2
3. timm 0.6.12
4. tqdm
5. numpy
6. scipy

🔑 Run experiment

1. Edit the [MODEL NAME].json file for global settings and hyperparameters.

2. Run:

   python main.py --config=./exps/[MODEL NAME].json
   

3. hyper-parameters

   When using PILOT, you can edit the global parameters and algorithm-specific hyper-parameter in the corresponding json file.

   These parameters include:

   • model_name: The model's name should be selected from the 11 methods listed above, i.e., finetune, icarl, coil, der, foster, memo, simplecil, l2p, dualprompt, coda-prompt and adam.
   • init_cls: The number of classes in the initial incremental stage. As the configuration of CIL includes different settings with varying class numbers at the outset, our framework accommodates diverse options for defining the initial stage.
   • increment: The number of classes in each incremental stage $i$, $i$ > 1. By default, the number of classes is equal across all incremental stages.
   • backbone_type: The backbone network of the incremental model. It can be selected from a variety of pre-trained models available in the Timm library, such as ViT-B/16-IN1K and ViT-B/16-IN21K. Both are pre-trained on ImageNet21K, while the former is additionally fine-tuned on ImageNet1K.
   • seed: The random seed is utilized for shuffling the class order. It is set to 1993 by default, following the benchmark setting iCaRL.
   • fixed_memory: a Boolean parameter. When set to true, the model will maintain a fixed amount of memory per class. Alternatively, when set to false, the model will preserve dynamic memory allocation per class.
   • memory_size: The total number of exemplars in the incremental learning process. If fixed_memory is set to false, assuming there are $K$ classes at the current stage, the model will preserve $\left[\frac{{memory-size}}{K}\right]$ exemplars for each class. L2P, DualPrompt, SimpleCIL, ADAM, and CODA-Prompt do not require exemplars. Therefore, parameters related to the exemplar are not utilized.
   • memory_per_class: If fixed memory is set to true, the model will preserve a fixed number of memory_per_class exemplars for each class.

🔎 Datasets

We have implemented the pre-processing datasets as follows:

• CIFAR100: will be automatically downloaded by the code.
• CUB200: Google Drive: link or Onedrive: link
• ImageNet-R: Google Drive: link or Onedrive: link
• ImageNet-A: Google Drive: link or Onedrive: link
• OmniBenchmark: Google Drive: link or Onedrive: link
• VTAB: Google Drive: link or Onedrive: link
• ObjectNet: Onedrive: link You can also refer to the filelist if the file is too large to download.

These subsets are sampled from the original datasets. Please note that I do not have the right to distribute these datasets. If the distribution violates the license, I shall provide the filenames instead.

When training not on CIFAR100, you should specify the folder of your dataset in utils/data.py.

    def download_data(self):
        assert 0,"You should specify the folder of your dataset"
        train_dir = '[DATA-PATH]/train/'
        test_dir = '[DATA-PATH]/val/'

👨‍🏫 Acknowledgments

We thank the following repos providing helpful components/functions in our work.

• PyCIL
• RevisitingCIL
• l2p-pytorch
• CODA-Prompt
• RanPAC

🤗 Contact

If there are any questions, please feel free to propose new features by opening an issue or contact with the author: Hai-Long Sun([email protected]) and Da-Wei Zhou([email protected]). Enjoy the code.

🚀 Star History