ylsung / vl_adapter

PyTorch code for "VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks" (CVPR2022)

VL-Adapter

• Authors: Yi-Lin Sung, Jaemin Cho, Mohit Bansal
• Paper: "VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks" (CVPR 2022)

We evaluate VL-adapter in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2, and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C.

Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model.

** Note ** Please go into CLIP-ViL folder and follow the README there for running the experiments of adapters on CLIP-ViL. This README is for adapters on VL-Bart.

Installation

# Create python environment (optional)
conda create -n vlt5 python=3.8
source activate vlt5

# Install python dependencies
pip install -r requirements.txt

# Download T5/BART backbone checkpoint
python download_backbones.py

# For MSCOCO captioning evaluation (optional; for captioning only)
python -c "import language_evaluation; language_evaluation.download('coco')"

Code structure

# Store images, features, and annotations
./datasets
    COCO/
        images/
        clip_featuers/
    VG/
        images/
        clip_features/
    GQA/
        images/
        clip_features/
    nlvr/
        images/
        clip_features/
    vqa/
    lxmert/

    video/
        ann/
        vis_features

# Train VL-T5 with adapters
./VL-T5/
    src/
        modeling_t5.py modeling_bart.py                       <= VL-T5/VL-BART model classes
        pretrain.py, pretrain_data.py, pretrain_model.py      <= pretraining
        vqa.py, vqa_data.py vqa_model.py ...                  <= fine-tuning on downstream tasks (ex. VQA, GQA, NLVR2)
        multitask.py, multitask_data.py multiask_model.py     <= multitask learning on 7 downstream tasks
        param.py                                              <= (argparse) configuration
        tokenization.py                                       <= custom tokenizer
        utils.py, dist_utils.py                               <= utility functions
    snap/                                                     <= store weight checkpoints
    scripts/                                                  <= bash scripts for pretraining and finetuning

Data

Image-text dataset

Please go to link to download the processed CLIP features. We suggest to use gdrive to download it. Unzip the downloaded file and arrange the folders following the format which is shown in the "Code Structure."

If you would like to use dgrive to download the data, please try the following command

gdrive download 1O_RU1iFh_sbItZCTkOHUrbVIQQ_89Djj

Extract your own CLIP features

Please refer to feature_extraction for more details.

Video-text dataset

Please go to VALUE to download the ViT processed data.

Run different approaches

The following scripts can run every approach with the best hyper-parameters.

Image dataset

# Full fine-tuning
cd VL-T5/
bash scripts/image/full_finetuning.sh 1

# Single Adapter
cd VL-T5/
bash scripts/image/single_adapter.sh 1

# Multiple Adapters
cd VL-T5/
bash scripts/image/multiple_adapters.sh 1

# Hyperformer
cd VL-T5/
bash scripts/image/hyperformer.sh 1

# Single Compacter
cd VL-T5/
bash scripts/image/single_compacter.sh 1

# Multiple Compacters
cd VL-T5/
bash scripts/image/multiple_compacters.sh 1

# Single LoRA
cd VL-T5/
bash scripts/image/single_lora.sh 1

# Multiple LoRA
cd VL-T5/
bash scripts/image/multiple_lora.sh 1

# Single Prompt
cd VL-T5/
bash scripts/image/single_prompt.sh 1

# Multiple Prompts
cd VL-T5/
bash scripts/image/multiple_prompts.sh 1

Video dataset

# Full fine-tuning
cd VL-T5/
bash scripts/video/full_finetuning.sh 1

# Single Adapter
cd VL-T5/
bash scripts/video/single_adapter.sh 1

# Single LoRA
cd VL-T5/
bash scripts/video/single_lora.sh 1

# Single Prompt
cd VL-T5/
bash scripts/video/single_prompt.sh 1

Acknowledgement

This repo is adapted from VLT5. I also borrow some codes from CLIP, CLIP-ViL, Compacter, Hyperformer and Prefix-tuning.

Reference

Please cite our paper if you use our models in your project.

@inproceedings{sung2022vladapter,
  title     = {VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks},
  author    = {Yi-Lin Sung, Jaemin Cho, Mohit Bansal},
  booktitle = {CVPR},
  year      = {2022}
}

vl_adapter's Issues

Do you plan to also release the adapted checkpoints for each experiment?

Hi guys, thanks a lot for this great repo and work!

I wonder whether you are planning to also release the adapted model checkpoints used for each final evaluation.

Thanks again and looking forward to your reply~

A question about zero-grad settings in VL-adapter's multitask.py file.

Thanks for your brilliant work.

                batch['log_train_accuracy'] = self.args.log_train_accuracy

                # self.optim.zero_grad()
                if self.args.fp16 and _use_native_amp:
                    with autocast():
                        if self.args.distributed:
                            results = self.model.module.train_step(batch)
                        else:
                            results = self.model.train_step(batch)
                else:
                    if self.args.distributed:
                        results = self.model.module.train_step(batch)
                    else:
                        results = self.model.train_step(batch)

                loss = results['loss']

Looking at the code, it appears that you are training without initializing the gradients before performing backpropagation.

Is there a reason why this works?

unsuccessful clip feature extraction

Hi, Guys,

Thank you for providing the details in feature extraction.

I encountered a problem in feature_extraction/coco_CLIP.py. I used the following arguments to be consistent with your paper.

 'att_size': 7,
  'model_type': 'RN101' 

The following error occurs as running line 152 in coco_CLIP.py:

tmp_att, tmp_fc = model.encode_image(image)

Exception has occurred: ValueError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
not enough values to unpack (expected 2, got 1)
  File "/workspace/VL_adapter/feature_extraction/Omni_Benchmark_feature_extraction-Copy1.py", line 173, in main
    tmp_att, tmp_fc = model.encode_image(image) # expected 1 got 2
  File "/workspace/VL_adapter/feature_extraction/Omni_Benchmark_feature_extraction-Copy1.py", line 252, in <module>
    main(params)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main (Current frame)
    "__main__", mod_spec)

The model is imported using VL_adapter/CLIP-ViL/clip/

Can you help me with that?

Thank you so much!!

Where should I set a different $d$ for the Single Adapter?

Hi there, thanks a lot for your great work!
I am wondering where I can set a different $d$ for Single Adapter. In Table 11 of the paper, you listed several results from different $d$, but I didn't find this argument in the script file. Could you please tell me the right place?

Thanks again and looking forward to your reply ❤️‍🔥

About boxes in video

Hi, @ylsung
Thanks for your sharing the code. I would like to know why boxes in video are all set as [0, 0, 0, 0] rather than [0, 0, 1, 1]. Any motivation for this setting? In my view, [0, 0, 1, 1] seems a more rational choice, as the whole frame is taken from video.

train problem

it's a nice work,but there is a problem when i train the shell(bash scripts/image/multiple_adapters.sh 1).i find the problem in github,it seems about out of memory though i set the batch_size very small(20 or 10,my GPU is 2080ti(11G)).Do you know why is that?

,

Could you release pretrained models?

How you fed the VQA v2 each question and answers list to VL T5 or VL BART ?

Hey Authors,

Thank you for the repo @ylsung .

Can you please explain a bit how you guys sent the question and paired answers to the model as each question has multiple answers from the code it seems you guys are selecting one random answer (

VL_adapter/VL-T5/src/vqa_data.py

Line 229 in 545fcbb

answer = random.choice(answers)['answer']

). Can you please explain your approach a bit?

Clip feature extraction

This CVPR work is really interesting.
Btw, could you please also provide a script for extracting clip features?

Thank you for sharing this great work!

Where is the entrance to the setting of half-shared adapters?

Hi again, I would like to ask where the entrance to the half-shread adapters is.

On the README, there are only single and multiple adapters.

Thanks in advance!

No such file or directory: '/root/VL_adapter/datasets/vqa/train.json'

how to solve it...there is no 'train.json' file in the download link

When do you expect to release the code？

Hi, can you share your plan to release the code for VL_adapter?
Thanks!

DDP degrades the performance

Thank you for sharing this code!

I am testing your code for multitask video with BART on 24GB GPUs.
To run your code on 24GB GPUs, I used below command to enable DDP. (batch size:50 -> 25)

bash scripts/video/single_adapter.sh 2

However, it showed worse results than the performance on a single 48GB GPU.
When I increased the number of GPUs, the performance was getting worse.
Because the model doesn't have BatchNorm, I thought the performance should be similar.

Have you tried DDP? Or do you have any intuition about the problem?

Question about the design choice of the unified framework.

Thanks for the great work!
I had read the paper and raised a question about the choice of the unified framework. The author chose BART as a textual encoder and CLIP model as a visual encoder. I initially thought that that we could directly use CLIP textual and visual encoders. It could produce much better results since it was pre-trained on a cross-modal alignment task. Why is BART chosen instead?

training hangs every few seconds

Thanks for sharing your implementation for this awesome work!

I was trying to reproduce some of the exp results and I found that my training process was very slow. I noticed the training hang after every few iterations for seconds and then resumed. I tried tuning "number_of_workers" which didn't help in general. Reducing or increasing the number of GPUs doesn't resolve the issue either. When I monitored the gpu memory usage, I found when the training process hang, the memory usage is 0% for all gpus. I suspect there is something wrong with the dataloader but I cannot pin down the cause. Any comment is appreciated!

Environment:
Ubuntu 18.04
Single node with 8 Tesla V100 GPUs
Pytorch 1.13
CUDA 11.6

Sample command I used for running exp

cd VL-T5/
bash scripts/image/multiple_adapters.sh 8

Edit:
It turns out the bottleneck is on the reading of clip features which were saved in hdf5 format. I have to convert all clip features to ".npy" format and it gives ~7x speed up for data loading.

COCO Cap. Karpathy test CIDEr score is super low

Hi @ylsung

Thanks for sharing the codebase. I was trying to reproduce the results from training a single adapter for multiple datasets. I found that after training, the performance on the test set for COCO Captioning dataset is super low, e.g., CIDER is 1.16 while the paper reports 114.9. The validation results looks normal, val CIDER reaches 114 after training. The val/test results for the other 3 dataset seem normal as well. I was using the exact same code you provided for training and followed the instruction to create the environment. Did I miss anything?

FileNotFoundError

When I run ‘bash scripts/image/single_adapter.sh 1’, I met error like this:
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/data/VL_adapter-main/datasets/COCO/clip_features/data_clip_RN101_att/COCO_val2014_000000007414.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Would you mind telling me how can I get this .h5 file?
Thank you.