continualai / clvision-challenge-2022

CVPR 2022 Continual Learning in Computer Vision Workshop Challenge

3rd CLVision Workshop @ CVPR 2022 Challenge

This is the official starting repository for the Continual Learning Challenge held in the 3rd CLVision Workshop @ CVPR 2022.

Please refer to the challenge website for more details!

Updates:

• The competition report paper is available on arXiv!
• Reports from the top-3 teams per track are now available under the reports folder.

Getting started

The devkit is based on the Avalanche library. We warmly recommend looking at the documentation (especially the "Zero To Hero tutorials") if this is your first time using it! Avalanche is added as a Git submodule of this repository.

The recommended setup steps are as follows:

1. Install conda (and mamba; recommended)

2. Clone the repo and create the conda environment:

git clone --recurse-submodules https://github.com/ContinualAI/clvision-challenge-2022.git
cd clvision-challenge-2022
conda env create -f environment.yml

3. Setup your IDE so that the avalanche submodule is included in the PYTHONPATH. Note: you have to include the top-level folder, not avalanche/avalanche!

   1. For Jetbrains IDEs (PyCharm), this can be done from the Project pane (usually on the right) by right-clicking on the "avalanche" folder -> "Mark Directory as" -> "Sources Root".
   2. For VS Code, follow the official documentation.

4. Download and extract the dataset: in order to download the dataset, we ask all participants to accept the dataset terms and provide their email addresses through this form. You will immediately receive the download instructions at the provided address. We recommend extracting the dataset in the default folder $HOME/3rd_clvision_challenge/demo_dataset/. The final directory structure should be like this:

$HOME/3rd_clvision_challenge/challenge/
├── ego_objects_challenge_test.json
├── ego_objects_challenge_train.json
├── images
│   ├── 07A28C4666133270E9D65BAB3BCBB094_0.png
│   ├── 07A28C4666133270E9D65BAB3BCBB094_100.png
│   ├── 07A28C4666133270E9D65BAB3BCBB094_101.png
│   ├── ...

The aforementioned steps should be OS-agnostic. However, we recommend setting up your dev environment using a mainstream Linux distribution.

Object Classification (Track 1) Training Template

The starting template for the object classification track is based on the tried and tested strategies from Avalanche.

In particular, the starting template can be found in starting_template_instance_classification.py. The default template implements a working train/eval loop that uses the Naive strategy.

The Naive strategy is a plain fine-tuning loop that, given the optimizer, number of epochs, the minibatch size, and the loss function, will just run a very forgetting-prone training loop. This should be taken as the lower bound for a solution. This means that the basic loop is already there, ready to be customized. There are two main ways to implement your solution:

• Override parts of the base class SupervisedTemplate in order to customize the epoch loop, the backward and forward operations, etcetera.
• Implement your solution as a plugin (many mainstream techniques are implemented in Avalanche as plugins, see the documentation).

We suggest you to study the From Zero To Hero tutorials to learn about Avalanche.

Object Detection (Track 2 and 3) Training Templates

The starting point for the detection tracks are starting_template_category_detection.py and starting_template_instance_detection.py. That entry point uses the (ObjectDetectionTemplate) template, which is the template you should customize to implement your CL strategy. The recommended way to do this is to create a child class.

That training template is based on Avalanche training templates. The training loop is an almost exact implementation of the one shown in the official TorchVision Object Detection Finetuning Tutorial, especially the train_one_epoch method.

A schematic visualization of the training loop, its events, and an example of a plugin implementing EWC is shown below:

Submitting a solution

Solutions must be submitted through the CodaLab portal:

• Instance Classification track: submissions portal
• Category Detection track: submissions portal
• Instance Detection track: submissions portal

A solution consists of a zip file containing "N" (track-dependent) files. Each file must contain the predictions obtained on the full test set by running an eval pass after each training experience. The devkit already contains a plugin that will take care of storing such output. Make sure you do NOT include intermediate directories inside the archive. Do not change the name of files produced by the plugin.

The maximum number of allowed submissions is 20. Only 3 solutions can be submitted each day.

Note: the evaluation for detection tracks may take some minutes to complete.

Hints

• The devkit will be updated quite often. We recommend checking if there are new updates frequently.
• The InteractiveLogger will just print the progress to stdout (and it is quite verbose). Consider using dashboard loggers, such as Tensorboard or Weights & Biases. See the tutorial on loggers here. You can use more than one logger at the same time!

clvision-challenge-2022's Issues

#track 2: 'Solutions can exploit a replay buffer containing data coming from up to 3500 training instances.' For example, in task1, can we randomly sample 3500 images from the full training set (ego_objects_challenge_train.json)?

What's this repo actually for?

Does it mean we have to build our training and inference procedure with this repo?

Track2 Submission Problem: AttributeError: 'list' object has no attribute 'items'

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
File "/tmp/codalab/tmpYQV4mo/run/program/track2_evaluator.py", line 197, in
evaluate_track2(args.input_dir, args.output_dir)
File "/tmp/codalab/tmpYQV4mo/run/program/track2_evaluator.py", line 110, in evaluate_track2
score_str = evaluate_track2_jsons(truth_file, submit_dir)
File "/tmp/codalab/tmpYQV4mo/run/program/track2_evaluator.py", line 49, in evaluate_track2_jsons
predictions_json = prepare_for_ego_objects_detection(predictions_json)
File "/tmp/codalab/tmpYQV4mo/run/program/track2_evaluator.py", line 119, in prepare_for_ego_objects_detection
for original_id, prediction in predictions.items():
AttributeError: 'list' object has no attribute 'items'

When we tried to submit results,it said the problem like this. We check the json files and find it is list,not dict. I want to know where I can change this part code,thanks a lot.

mandatory metric only generate output of 1 json file for last experience

For detection track, the mandatory metric output .json file with same filename. In the end, they overwrite previous outputs and only 1 track2_output_eval_exp0.json file is left in the output dir.
This is tested using the given starting_template_category_detection.py

Detection test set all instances have "category_id": 2, "instance_id": "unlabeled"

In the detection test set, ego_objects_challenge_train.json, all instances are of "category_id": 2, "instance_id": "unlabeled"

can you release the metrics of task1-task5 in leadboard?

Different classes in train/test experiences

Hello,

I am experimenting with different CL metrics which require access to each separate test experience (in addition to the default concatenated one). For this, I modified the demo_detection_benchmark to also output an additional DetectionCLScenario, which contains 4 test experiences. However, it seems that each test experience contains a different number of classes compared to the corresponding train experience, i.e. [48, 47, 49, 52] vs [54, 52, 56, 53]. Is it because we have several objects per image that the classes cannot be cleanly separated?

The code to reproduce this can be found below.

import itertools
from pathlib import Path
from typing import Union

from avalanche.benchmarks import NCScenario, StreamUserDef
from avalanche.benchmarks.utils import AvalancheDataset
from devkit_tools import ChallengeClassificationDataset, \
    ChallengeDetectionDataset
from devkit_tools.benchmarks.classification_benchmark import \
    demo_classification_benchmark
from devkit_tools.benchmarks.detection_scenario import DetectionCLScenario, \
    det_exp_factory
from devkit_tools.benchmarks.remap_category_ids import \
    make_compact_category_ids_mapping, remap_category_ids
from devkit_tools.challenge_constants import \
    DEMO_DETECTION_FORCED_TRANSFORMS, DEFAULT_DEMO_TEST_JSON, \
    DEFAULT_DEMO_TRAIN_JSON, DEMO_DETECTION_EXPERIENCES
from ego_objects import EgoObjects
from examples.tvdetection.transforms import ToTensor


def dev_demo_detection_benchmark(
        dataset_path: Union[str, Path],
        class_order_seed: int,
        *,
        train_transform=None,
        eval_transform=None,
        train_json_name=None,
        test_json_name=None,
        return_dev=False):

    # Use the classification benchmark creator to generate the correct order
    cls_benchmark: NCScenario = demo_classification_benchmark(
        dataset_path,
        class_order_seed,
        train_json_name=train_json_name,
        test_json_name=test_json_name,
        instance_level=False,
        n_exps=DEMO_DETECTION_EXPERIENCES
    )

    # Create aligned datasets
    if train_json_name is None:
        train_json_name = DEFAULT_DEMO_TRAIN_JSON
    train_ego_api = EgoObjects(str(Path(dataset_path) / train_json_name))

    if test_json_name is None:
        test_json_name = DEFAULT_DEMO_TEST_JSON
    test_ego_api = EgoObjects(str(Path(dataset_path) / test_json_name))

    train_dataset = ChallengeClassificationDataset(
        dataset_path,
        ego_api=train_ego_api,
        train=True,
        bbox_margin=20,
        instance_level=False
    )

    test_dataset = ChallengeClassificationDataset(
        dataset_path,
        ego_api=test_ego_api,
        train=False,
        bbox_margin=20,
        instance_level=False
    )

    # Keep this order
    train_order = cls_benchmark.train_exps_patterns_assignment
    test_order = list(itertools.chain.from_iterable(
        cls_benchmark.test_exps_patterns_assignment))

    dev_test_order = cls_benchmark.test_exps_patterns_assignment

    train_img_ids = []
    for exp_id in range(len(train_order)):
        img_id_in_exp = []
        for instance_idx in train_order[exp_id]:
            img_id_in_exp.append(train_dataset.img_ids[instance_idx])
        train_img_ids.append(img_id_in_exp)

    test_img_ids = []
    for instance_idx in test_order:
        test_img_ids.append(test_dataset.img_ids[instance_idx])

    dev_test_img_ids = []
    for exp_id in range(len(dev_test_order)):
        img_id_in_exp = []
        for instance_idx in dev_test_order[exp_id]:
            img_id_in_exp.append(test_dataset.img_ids[instance_idx])
        dev_test_img_ids.append(img_id_in_exp)

    base_transforms = dict(
        train=(DEMO_DETECTION_FORCED_TRANSFORMS, None),
        eval=(DEMO_DETECTION_FORCED_TRANSFORMS, None)
    )

    # Align categories IDs
    # The JSON may contain categories with sparse IDs
    # In this way max(categories_ids) >= len(categories), which is not ok!
    # For instance, if category IDs are [0, 1, 2, 3, 5], then initializing
    # the ROI head with n_categories=5 is wrong and it will trigger errors
    # when computing losses (as logits must have 6 elements, not 5)
    # To prevent issues, we just remap categories IDs to range [0, n_categories)
    categories_id_mapping = make_compact_category_ids_mapping(
        train_ego_api, test_ego_api)

    remap_category_ids(train_ego_api, categories_id_mapping)
    remap_category_ids(test_ego_api, categories_id_mapping)

    train_exps = []
    for exp_id, exp_img_ids in enumerate(train_img_ids):
        exp_dataset = ChallengeDetectionDataset(
            dataset_path,
            train=True,
            ego_api=train_ego_api,
            img_ids=exp_img_ids,
        )

        avl_exp_dataset = AvalancheDataset(
            exp_dataset,
            transform_groups=base_transforms,
            initial_transform_group='train'
        ).freeze_transforms(
        ).add_transforms_to_group(
            'train', transform=train_transform
        ).add_transforms_to_group(
            'eval', transform=eval_transform
        )

        train_exps.append(avl_exp_dataset)

    test_exps = []
    exp_dataset = ChallengeDetectionDataset(
        dataset_path,
        train=False,
        ego_api=test_ego_api,
        img_ids=test_img_ids,
    )

    avl_exp_dataset = AvalancheDataset(
        exp_dataset,
        transform_groups=base_transforms,
        initial_transform_group='eval'
    ).freeze_transforms(
    ).add_transforms_to_group(
        'train', transform=train_transform
    ).add_transforms_to_group(
        'eval', transform=eval_transform
    )

    test_exps.append(avl_exp_dataset)

    dev_test_exps = []
    for exp_id, exp_img_ids in enumerate(dev_test_img_ids):
        exp_dataset = ChallengeDetectionDataset(
            dataset_path,
            train=False,
            ego_api=test_ego_api,
            img_ids=exp_img_ids,
        )
        avl_exp_dataet = AvalancheDataset(
            exp_dataset,
            transform_groups=base_transforms,
            initial_transform_group='eval',
        ).freeze_transforms(
        ).add_transforms_to_group(
            'train', transform=train_transform
        ).add_transforms_to_group(
            'eval', transform=eval_transform
        )
        dev_test_exps.append(avl_exp_dataet)

    all_cat_ids = set(train_ego_api.get_cat_ids())
    all_cat_ids.union(test_ego_api.get_cat_ids())

    assert set(train_ego_api.get_cat_ids()) == set(test_ego_api.get_cat_ids())

    train_def = StreamUserDef(
        exps_data=train_exps,
        exps_task_labels=[0 for _ in range(len(train_exps))],
        origin_dataset=None,
        is_lazy=False
    )

    test_def = StreamUserDef(
        exps_data=test_exps,
        exps_task_labels=[0],
        origin_dataset=None,
        is_lazy=False
    )

    dev_test_def = StreamUserDef(
        exps_data=dev_test_exps,
        exps_task_labels=[0 for _ in range(len(dev_test_exps))],
        origin_dataset=None,
        is_lazy=False
    )

    dev = {
        "dev_scenario": DetectionCLScenario(
            n_classes=len(all_cat_ids),
            stream_definitions={
                'train': train_def,
                'test': dev_test_def
            },
            complete_test_set_only=False,
            experience_factory=det_exp_factory
        ), 
        "categories_id_mapping": categories_id_mapping,
        "cls_benchmark": cls_benchmark,
    }

    original_return = DetectionCLScenario(
        n_classes=len(all_cat_ids),
        stream_definitions={
            'train': train_def,
            'test': test_def
        },
        complete_test_set_only=True,
        experience_factory=det_exp_factory
    )

    if return_dev:
        return original_return, dev
    
    return original_return


__all__ = [
    'demo_detection_benchmark'
]


if __name__=="__main__":

    from devkit_tools.challenge_constants import DEFAULT_DEMO_CLASS_ORDER_SEED
    from avalanche.benchmarks.utils import Compose


    train_transform = Compose(
        [   
            ToTensor(),
        ]
    )

    eval_transform = Compose(
        [
            ToTensor(),
        ]
    )

    
    _, dev = dev_demo_detection_benchmark(
        dataset_path="data/demo",
        class_order_seed=DEFAULT_DEMO_CLASS_ORDER_SEED,
        train_transform=train_transform,
        eval_transform=eval_transform,
        return_dev=True,
    )

    dev_benchmark = dev["dev_scenario"]
    def get_stream_classes(stream):
        classes = []

        for i in range(len(stream)):
            print(f"Experience={i}")
            exp_classes = set()
            avl_dataset = stream[i].dataset
            print(f"dataset_len={len(avl_dataset)}")
            assert len(avl_dataset.targets) == len(avl_dataset._dataset)
            for _img, targets, _task in avl_dataset:
                exp_classes.update(targets["labels"].tolist())
            classes.append(exp_classes)

        return classes

    print("Test")
    test_classes = get_stream_classes(dev_benchmark.test_stream)
    print([len(t) for t in test_classes]) # this print: [54, 52, 56, 53]

    print("Train")
    train_classes = get_stream_classes(dev_benchmark.train_stream)
    print(train_classes)
    print([len(t) for t in train_classes]) # this print: [48, 47, 49, 52]

Can you provide the evaluation server (RTX 5000 with 16G GPU-memory)?

Why track2 demo results AP seems strange?

Eval on experience 0 (Task 0) from test stream ended.
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/AP = 0.0008
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/AP50 = 0.0014
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/AP75 = 0.0009
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APc = -1
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APf = 0.0008
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APl = 0.0009
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APm = 0.0000
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APr = -1
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/APs = 0.0000
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/AR@300 = 0.0005
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/ARl@300 = 0.0006
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/ARm@300 = 0.0000
LvisMetrics/eval_phase/test_stream/Task000/Exp000/bbox/ARs@300 = 0.0000
Time_Exp/eval_phase/test_stream/Task000/Exp000 = 163.8030
-- >> End of eval phase << --
Time_Stream/eval_phase/test_stream/Task000 = 165.282

I don't know if the AP is correct when i run starting_template_category_detection.py. Because it is so low. Hope for your reply,thanks

the challenge dataset download failed.

We have tried many times, when the dataset is downloaded to 3G, the download speed becomes 0, download failed, can you upload the dataset to other platform. etc Onedrive, google drive, baidu pan.

Track1: Could you release the upper-bound performance(average mAP) during each experience?

Can we use multiple GPUs in training ? or we must use one GPU (RTX 5000 with 16G GPU-memory)?

training set of object detection containing bboxes of 0 area

There are some instances in the object detection training set containing bboxes of 0 area, e.g.
{"id": 127172, "image_id": 569716681169025, "bbox": [940.79, 197.9, 77.73, 0], "category_id": 351, "instance_id": "60E87098DACA63C09F51E202A6B813E3_351_0", "area": 0.0}
Is this a bug or intended behavior?

Thanks.

Track3 baseline doesn't work correctly

Runned the track3 baseline, create only one json file for submit but five json files are needed for submit.

Cannot iterate all data streams in one epoch.

Hello. When I use the replay plugin with the default strategy, the terminal outputs the following training logs. It doesn't seem like all the data streams are being used (stop at 50th iteration).

        RunningLoss_Epoch/train_phase/train_stream/Task000 = 3.2020
        Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9686
 71%|███████████████████████████████████████████████▏                  | 50/70 [01:03<00:25,  1.27s/it]
Epoch 21 ended.
        RunningLoss_Epoch/train_phase/train_stream/Task000 = 3.0610
        Top1_Acc_Epoch/train_phase/train_stream/Task000 = 0.9662
 71%|███████████████████████████████████████████████▏                  | 50/70 [01:03<00:25,  1.27s/it]

#track 2: can you release the upper-bound performance of task1-task5? The test set contains 277 categories, while the training set of task1 only has 159 categories, then the upper-bound of task1 is?

Can we add augmentation transforms to the eval transformations?

We found two comments about this problem in the Challenge Repo, but their content conflicted with each other.

can you provide a result file JSON format?

Are you modified the evaluation code on the server? The results submitted before were normal, but now they are wrong.

where is the examples.tvdetection.transforms

Hi, in the starting_template_track2.py , where is the examples.tvdetection.transforms ?

If Using multiple accounts on CodaLab, but the overall number of submissions are less than 20, is ok?