maheshkkumar / fscc

Few-Shot Scene Adaptive Crowd Counting Using Meta-Learning (WACV 2020)

Few-Shot Scene Adaptive Crowd Counting Using Meta-Learning

This repository contains the PyTorch implementation for "Few-Shot Scene Adaptive Crowd Counting Using Meta-Learning" by Mahesh Kumar Krishna Reddy, Mohammad Hossain, Mrigank Rochan, and Yang Wang. If you make use of this code in your work, please cite the paper.

Code | Paper

Abstract

We consider the problem of few-shot scene adaptive crowd counting. Given a target camera scene, our goal is to adapt a model to this specific scene with only a few labeled images of that scene. The solution to this problem has potential applications in numerous real-world scenarios, where we ideally like to deploy a crowd counting model specially adapted to a target camera. We accomplish this challenge by taking inspiration from the recently introduced learning-to-learn paradigm in the context of few-shot regime. In training, our method learns the model parameters in a way that facilitates the fast adaptation to the target scene. At test time, given a target scene with a small number of labeled data, our method quickly adapts to that scene with a few gradient updates to the learned parameters. Our extensive experimental results show that the proposed approach outperforms other alternatives in few-shot scene adaptive crowd counting.

Setup

pip install -r requirements.txt

Datasets

The details related to all the crowd counting datasets can be found in the following links.

1. WorldExpo'10
2. Mall
3. UCSD

Training

First, to generate a pre-trained backbone CSRNet network, please refer to CSRNet documentation. Then, the command line arguments for the meta-learning train.py is as follows:

>> python train.py --help

usage: train.py [-h] [-d DATASET] -trp DATA_PATH [-nt NUM_TASKS] [-ni NUM_INSTANCES] [-mb META_BATCH] [-bb BASE_BATCH] [-mlr META_LR] [-blr BASE_LR] [-e EPOCHS] [-bu BASE_UPDATES] [-exp EXPERIMENT] -log LOG_NAME

optional arguments:
  -h, --help            show this help message and exit
  -d DATASET, --dataset DATASET
                        Name of the dataset
  -trp DATA_PATH, --data_path DATA_PATH
                        Path of the dataset
  -nt NUM_TASKS, --num_tasks NUM_TASKS
                        Number of tasks for training
  -ni NUM_INSTANCES, --num_instances NUM_INSTANCES
                        Number of instances per task for training
  -mb META_BATCH, --meta_batch META_BATCH
                        Batch size for meta network
  -bb BASE_BATCH, --base_batch BASE_BATCH
                        Batch size for base network
  -mlr META_LR, --meta_lr META_LR
                        Meta learning rate
  -blr BASE_LR, --base_lr BASE_LR
                        Base learning rate
  -e EPOCHS, --epochs EPOCHS
                        Number of training epochs
  -bu BASE_UPDATES, --base_updates BASE_UPDATES
                        Iterations for base network to train
  -exp EXPERIMENT, --experiment EXPERIMENT
                        Experiment number
  -log LOG_NAME, --log_name LOG_NAME
                        Name of logging file

You can either train the network using the run.sh or by using the following command:

python train.py --dataset=<dataset_name> \
                --data_path=<data_path> \
                --num_tasks=<num_tasks> \
                --num_instances=<num_instances> \
                --meta_batch=<meta_batch> \
                --base_batch=<base_batch> \
                --meta_lr=0.001 \
                --base_lr=0.001 \
                --epochs=1000 \
                --base_updates=<base_updates> \
                --exp=<experiment_name> \
                --log=<log_path>

Citation

@inproceedings{reddy2020few,
    author      =  {Reddy, Mahesh Kumar Krishna and Hossain, Mohammad and Rochan, Mrigank and Wang, Yang},
    title       =  {Few-Shot Scene Adaptive Crowd Counting Using Meta-Learning},
    booktitle   =  {The IEEE Winter Conference on Applications of Computer Vision (WACV)},
    month       =  {March},
    year        =  {2020}
}

Acknowledgements

We have borrowed code from the following repositories:

1. https://github.com/leeyeehoo/CSRNet-pytorch
2. https://github.com/katerakelly/pytorch-maml

License

The project is licensed under MIT license (please refer to LICENSE.txt for more details).

fscc's Issues

Bug in metalearner.py line 85

Hi guys,

I think there is an indentation bug in metalearner.py line 85. the call to add the hooks should be within the for loop.

Greetz
M

https://github.com/maheshkkumar/fscc/blob/master/networks/meta_learner.py

Pretrained model for CSRNet

In meta_learner.py, there are # TODO: path of the pre-trained backbone CSRNet.
In the paper, the model use VGG-16 pretrained model on ImageNet. Does pretrained backbone CSRNet mean the VGG-16 pretrained model on ImageNet?
Can you provide the pretrained model?

The directory structure of the WorldExpo'10 dataset

I am puzzled if the directory structure of the WorldExpo'10 dataset can be described,especially what directory should groundtruth be in.It takes time to place them correctly to match the code

code help

Can you provide me with the code of the comparison experiment（Reptile and Meta-LSTM)? This will reduce my workload

"The size of tensor a (1024) must match the size of tensor b(128) at non-singleton dimension 3"

Hi, When I try to train the project, I find a error as follows:
RuntimeError: "The size of tensor a (1024) must match the size of tensor b(128) at non-singleton dimension 3"

Can you give some solutions? Thank you!

About the experiment on Meta-LSTM

Hi,Thanks for your great work here!
I wonder if you have the code for Meta-LSTM,and can you share it with me.
Hope to receive your reply,thanks.

About dataset

Could you share the structure of WorldExpo'10 dataset?
I want to know the train, test data file structure.
And Could you provide sample annotation file? I don't know how the annotation file is organized.
If you could share the dataset structure and sample annotation file, it will be very helpful for me.

Pretrained CSRNet

Hey,
By pretrained CSRNet, do you mean Imagenet pretrained Vgg16 layers plus random initialization for rest of the CSRNet layers ? (Looks like the pretrained net is trained on world expo. If that's the case, could you please share the pretrained net as well ? )

Thanks,
Viresh

What's the TestDataset?

Hi, thanks for your great job! I have a problem when reading the code, that is how should we save the best model during training? And I found that you save the best model by evaluating the model in the testdataset, but I didn't find any descripiton about it, can you tell me what it is?

NameError: name 'CustomDataset' is not defined

Hi, When I try to train the project, I find a error as follows:
NameError: name 'CustomDataset' is not defined

Can you give some solutions? Thank you!

Preprocessing on the dataset

Hi,

Great application of MAML. I wonder if you haven't released some preprocessing operations. I found I cannot directly run the code, The original dataset has not been divided into 'tasks (classes)', so I will get errors in the get_task.py.
Hope to hear from you.

Best.