salesforce / albef Goto Github PK

I don't have AWS account. Can you share some valid method?

Get the following error:
PIL.UnidentifiedImageError: cannot identify image file '/home/ubuntu/data/CC3M/DownloadConceptualCaptions/validation/10481_3355970027'

The error is generated by this code in caption_dataset.py:
image = Image.open(ann['image']).convert('RGB')

BTW, I can only download 2.4M images from CC3M/training, how did you download 2.95M images? Thanks.

I have a problem at pretrain phase, such as:
Traceback (most recent call last):
File "Pretrain.py", line 215, in
Traceback (most recent call last):
File "Pretrain.py", line 215, in
main(args, config)
File "Pretrain.py", line 93, in main
main(args, config)utils.init_distributed_mode(args)

File "Pretrain.py", line 93, in main
File "/root/albef/ALBEF/utils.py", line 257, in init_distributed_mode
utils.init_distributed_mode(args)
File "/root/albef/ALBEF/utils.py", line 257, in init_distributed_mode
torch.distributed.barrier()
File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 2420, in barrier
torch.distributed.barrier()
File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 2420, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled system error, NCCL version 2.7.8
ncclSystemError: System call (socket, malloc, munmap, etc) failed.

Have you ever had a similar problem or give me some advice? Thanks!

I want to do the VQA finetuning. I do not have A100 GPUs, and I could only run on 4 2080Ti GPUs with 11019MB memory. I tried to reduce the batch size to 1 but still got "CUDA out of memory". I want to inquire whether it is normal?

In your grounding task, you used 'dets.json' to evaluate your results, how do you get the 'dets.json' file? which object detector did you use?

While fine-tuning the network on larger dataset (on 8 A100 GPUs) for text to image retrieval, I always face the issue of CUDA OUT OF MEMORY, after running for 1000-1500 batches. If the batch size is reduced to 16 (from 32), this issue is faced after 2500-3000 batches.

I am currently using -

• pytorch 1.8.0
• transformers 4.8.1
• timm 0.4.9
• batch_size_train: 32
• batch_size_test: 32
  E.g.:
  RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 5; 39.59 GiB total capacity; 34.18 GiB already allocated;111.19 MiB free; 37.37 GiB reserved in total by PyTorch).

Did you face any of these issues during fine-tuning?

when I test in VQA task, I run :
python -m torch.distributed.launch --nproc_per_node=8 --use_env VQA.py
--config ./configs/VQA.yaml
--output_dir output/vqa
--checkpoint ./ALBEF.pth
But, when I run one epoch, I get the result flood is empty?
How to get the result? when all epoch training end?

Hi Junnan, I have the following questions about the result. Hope that you can help to clarify them, thanks.

1. VQA:
   I get the result folder after fine-tuning on VQA dataset. Which json file should I use to get test-dev and test-std?
   

2. SNLI-VE:
   This is the log file after fine-tuning on SNLI-VE dataset. You didn't update the best-epoch, so it is always 0. Should I pick the row which has the best val accuracy as the final result?
   

3. Grouding
   This is the log file after fine-tuning on Ref-COCO. Should I pick the row which has the best val_d as the final result?
   

4. NLVR2
   This is the log file after fine-tuning on NLVR2, but I did't find dev and test-P as shown in your paper, any idea?
   

Hi, I tried to visualize the whole sentence attention map by [CLS], but the result is inaccurate.
For example: text tokens are [[CLS], "grey", "sweater"], the "grey" "sweater" tokens fits well in the image, but the [CLS] token heat map is incorrect. How did you get the result of Figure 4? Thank you!

I've removed text attention mask:

    if show_CLS:
        cams = cams[:, :, :, 1:].reshape(image.size(0), 12, -1, 24, 24)
        grads = grads[:, :, :, 1:].clamp(0).reshape(image.size(0), 12, -1, 24, 24)
    else:
        cams = cams[:, :, :, 1:].reshape(image.size(0), 12, -1, 24, 24) * mask
        grads = grads[:, :, :, 1:].clamp(0).reshape(image.size(0), 12, -1, 24, 24) * mask

hi, I notice that you set max length of Tokenizer is 25. However, Clip and Align, Filip set 76. Three times exist. You can't deal with long description, which may leads to bad performance？

Traceback (most recent call last):
File "Retrieval.py", line 397, in
main(args, config)
File "Retrieval.py", line 265, in main
state_dict = checkpoint['model']
KeyError: 'model'

Hi, thanks for your sharing ALBEF.
I have got that the checkpoint released on Github is pretrained on 14M dataset at issue12. I wonder if you would release the pretrained checkpoint on 4M dataset?

Thanks! ^_^

80GB or 40GB.?
For accelerating data reading, which method do you apply?
Thank you

Hi, thanks for the great work!

I wonder why Table 8 says the number of captions of VG is 769K while in previous papers (e.g. ViLT, UNITER) it is ~5M?

Thanks a lot for the interesting work.
BTW, I see in configs. you always make opt as adamw, did you select apex with fp16 for training or inference? I see in your code only when ope == 'fusedadamw' will use the apex optimizer.

I know there is no official split config. what criterion do you apply when you split the dataset.

Is this going to be released?

Hi,
Thanks for your nice work! The released code is also very useful.

I am wondering how did you get answer_list.json. It contains 3,128 answers. However, your paper says that you use 3,192 candidate answers. Meanwhile, in fact existing methods usually use 3,129 answers as reported in their papers.

I have tried to get top-3,129 most frequent answers on VQA train set or train&validation set. It yields a list that only has ~2700 answers overlapped with your list.

Thanks!

Hi Junnan,

I noticed the answers in the VQA dataset are assigned a weight value based on their frequency, and all answers to a question have a total weight of 1. For VG dataset, it looks like the answers are assigned a weight of 0.5 (see https://github.com/salesforce/ALBEF/blob/main/dataset/vqa_dataset.py#L66). Since there is always a single answer to VG questions, if using the same rules as in VQA dataset, the weight should be 1. Do you have any reason not to use a weight value of 1 for VG answers?

Best,
Jie

I want to reproduce your result using your code. I download datasets excluding CC12M. And I find the three losses vary dramaticly. I don't know whether normal or not?

First of all, thanks for your great work!I am trying to speed up the pretraining process by using more GPUs(such as 32 or 64 A100), but it turns out a performance drop in the original setting. i wonder the 8 GPUs version config(such as batch size or epoch num) is not suitable for more GPUs. have you tried pretraining in more GPUs and could you please share the config?

Hi, @LiJunnan1992

Thanks for your work and release!

However, I got a lower zero-shot performance on flickr30k, which seems to be the same as the performance reported in #23 ,

# val set
{'txt_r1': 90.23668639053254, 'txt_r5': 98.0276134122288, 'txt_r10': 99.11242603550296, 'txt_r_mean': 95.7922419460881, 'img_r1': 75.75936883629191, 'img_r5': 92.50493096646943, 'img_r10': 95.75936883629191, 'img_r_mean': 88.00788954635108, 'r_mean': 91.90006574621958}
# test set
{'txt_r1': 88.5, 'txt_r5': 98.5, 'txt_r10': 99.2, 'txt_r_mean': 95.39999999999999, 'img_r1': 75.92, 'img_r5': 93.34, 'img_r10': 96.66, 'img_r_mean': 88.63999999999999, 'r_mean': 92.01999999999998}

and used this command:

python -m torch.distributed.launch --nproc_per_node=8 --use_env Retrieval.py \
--config ./configs/Retrieval_flickr.yaml \
--output_dir output/Retrieval_flickr \
--checkpoint sources/ALBEF.pth --evaluate

Hi @LiJunnan1992,

A few questions on pretraining: (1) how do you decide which checkpoint to use for each run, are you always using the last checkpoint? (2) and how to compare pretrained checkpoint across runs (3) do you monitor pretraining performance using by using one of the downstream tasks? If so, could you provide some more details?

Best,
Jie

hi, I just want to test the effect of the model.
here is my test code:

python Retrieval.py --config ./configs/Retrieval_flickr.yaml --output_dir output/Retrieval_flickr --checkpoint model_file/ALBEF.pth --evaluate True

I have changed the relevant configuration files.

Am I right to test like this？

thx!

Hi, I guess this line should be from models.model_pretrain_nlvr import ALBEF?

Hi, Thanks for releasing ALBEF, it is a clean and well organized repo.

We use pre-extracted resnet image features with texts for multimodal tasks. I have tried to use ALBEF in our project, but the results is not as expected. I wonder if this is because the resnet image features.

I find that you provide ALBEF with resnet model, could you share any results on downstream tasks when using resnet for images?

Problem 1:
Hello, I found "train_file" have "train_file", such as:
train_file: ['/export/home/project/VL/dataset/caption/coco_karpathy_train.json',
'/export/home/project/VL/dataset/caption/vg_caption.json',
'/export/home/project/VL/dataset/pretrain_caption/conceptual_caption_train.json',
'/export/home/project/VL/dataset/pretrain_caption/conceptual_caption_val.json',
'/export/home/project/VL/dataset/pretrain_caption/sbu_caption.json'
]
Could you please provide the above JSON file?

Problem 2:
When I fine-tune NLVR2 task, I need to first run NLVR_pretrain.yaml file, and then run NLVR.yaml?

Hi @LiJunnan1992,

Congrats on your great work, and thanks for releasing the code!! To help reproduce the pretraining experiments, could you release the dataset json files for the pretraining datasets as well? Thanks!

Best,
Jie

"coco.json" is in the "json_pretrain" dir and has 108MB.
"coco_train.json" is in the "data" dir and has 84.6MB.

I am confused about the difference.
By the way, "data/coco_karpathy_train.json" points to "coco_train.json" or "coco.json" ?
I want to retrain your model. Thanks.

Hi thanks for your great work.
I have a minor question about your code.
I saw that the step_size is set to 100, and the warmup_iterations is step_size * warmup_epochs.
May I ask the justification?

Thanks!

Thank you for your sharing work. I got problems when finetuning Visual Grounding
python -m torch.distributed.launch --nproc_per_node=8 --use_env Grounding.py
--config ./configs/Grounding.yaml
--output_dir output/RefCOCO
--gradcam_mode itm
--block_num 8
--checkpoint ./pretrain/refcoco.pth
refcoco.pth was downloaded from your link.
Here is the error message
Traceback (most recent call last):
File "Grounding.py", line 295, in
main(args, config)
File "Grounding.py", line 187, in main
state_dict = checkpoint['model']
KeyError: 'model'

First of all, thanks for the great work.
I use 8 V100 to train the model with 4M images. it takes 5~6 hours for an epoch which is too slow. is it correct?
Could you please share the pretraining log, I want to check the loss of the pretraining process.
Thank you in advance.

For the SBU dataset (http://www.cs.virginia.edu/~vicente/sbucaptions/), a quite portion of the urls are unaccessible now (many downloaded images have 0 size). How did you deal with this case in pretraining?

Hi there, I am trying to evaluate the pretrained coco model you provided. Originally I got the same error as the original poster in #7 . However, setting distill to False does not fix the issue for me. It seems as though the checkpoint is already the model, therefore the line in #7 can be changed from state_dict = checkpoint['model'] to state_dict = checkpoint. However when I do this it gives me another key error here on line 190:

m_pos_embed_reshaped = interpolate_pos_embed(state_dict['visual_encoder_m.pos_embed'],model.visual_encoder_m) 

Inidicating that the key 'visual_encoder_m.pos_embed' does not exist. Is this an error with the code, checkpoint, or am I doing something wrong? For what its worth the key visual_encoder.pos_embed does exist. Any guidance is appreciated.

Also, great work.

Hi, kudos on your work, and thank you for releasing the code. I'm looking to use ALBEF to train on my custom data of about 50k-100k image-text pairs, and I had a few doubts.

• Can you tell me how I should go about changing the pretraining command to support training on a single gpu?
• What is the minimum batch size ALBEF must be trained on to get practical results (since contrastive loss is involved)
• How many epochs is recommended when trained on a single gpu, in order to reach pretraining convergence?
• I noticed in the sample pretrain config file that train_files is a list containing multiple paths to different datasets. For my single dataset json, should the value for this key be a list containing path(str) to json or just the path as a string itself?

I'm fairly new to this, and I appreciate the help. Thank you

I have a problem at pretrain phase, when the program is run by half, such as:
WARNING:torch.distributed.elastic.agent.server.api:Received 1 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6593 closing signal SIGHUP
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6593 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6594 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6595 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6596 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6597 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6598 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6599 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 6600 closing signal SIGTERM
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run
result = self._invoke_run(role)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/agent/server/api.py", line 843, in _invoke_run
time.sleep(monitor_interval)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 60, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 6523 got signal: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 193, in
main()
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 189, in main
launch(args)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/launch.py", line 174, in launch
run(args)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/launcher/api.py", line 252, in launch_agent
result = agent.run()
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/metrics/api.py", line 125, in wrapper
result = f(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/agent/server/api.py", line 716, in run
self._shutdown(e.sigval)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/agent/server/local_elastic_agent.py", line 190, in _shutdown
self._pcontext.close(death_sig)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 330, in close
self._close(death_sig=death_sig, timeout=timeout)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 709, in _close
if handler.proc.poll() is None:
File "/usr/lib/python3.6/subprocess.py", line 875, in poll
return self._internal_poll()
File "/usr/lib/python3.6/subprocess.py", line 1403, in _internal_poll
pid, sts = _waitpid(self.pid, _WNOHANG)
File "/home/ubuntu/.local/lib/python3.6/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 60, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 6523 got signal: 1

Have you ever had a similar problem?

Hi team, I'm trying to use the code to pretrain a model, and I looked at the JSON files contains pretrain dataset, but there is only the image location path. Could you please provide the raw image data, or where can I download that images? Because I found that the image storage names in the JSON files are different with the official version, and the captions are also a little different.

If it is hard to store all the raw images in your servers' hard-disk, I suggest that maybe you can provide a toy dataset for users to testing the codes. Thanks!

Hello! You provide the Pre-trained checkpoint is 4M or 14 M?
link:https://storage.googleapis.com/sfr-pcl-data-research/ALBEF/ALBEF.pth

In the page 3 of the paper, p_m^{i2t} = exp(...)/sum_{i=1}^M(...). I found it is different in code, which is sum_{i=1}^{M+batchsize}, so I think there is a mistake in this detail in the paper.

Hi, Thanks for your excellent work!
I want to ask which datasets are the released checkpoints pretrained on, 4m dataset or 4 + 10m dataset? Since there are two pretrained dataset in your paper.

Thanks!

Can you tell me for your provided "pretrained checkpoints", how many training images did you use, 4M or 14M?

I tested your "pretrained checkpoints" on flickr30k, I got this results with 8 V100 GPUs (32GB):
{'txt_r1': 88.5, 'txt_r5': 98.5, 'txt_r10': 99.2, 'txt_r_mean': 95.39999999999999, 'img_r1': 75.92, 'img_r5': 93.34, 'img_r10': 96.66, 'img_r_mean': 88.63999999999999, 'r_mean': 92.01999999999998}

which is slightly different of your provided results in Table 3 of arxiv paper, especially for TR @1.

Thanks for releasing the code!

I have a quick question about visualizing Grad-CAM.
Do you have any particular reason for using 3rd layer of multimodal encoder?
I've tried other layers using your demo code for visualization, but the results generated from the 4th & 5th layers are quite inaccurate.

Thanks in advance :)

Is the released pre-trained checkpoint trained on the 4M or 14M full dataset?

The fine-tuned checkpoint mscoco.pth provided by https://storage.googleapis.com/sfr-pcl-data-research/ALBEF/mscoco.pth (Finetuned checkpoint for retrieval on MSCOCO) is 800MB, but I get 1.69GB checkpoint model after running the Retrieval.py? Could you tell me why?

Could you explain the code to extract gradient here?
grads=model.text_encoder.base_model.base_model.encoder.layer[block_num].crossattention.self.get_attn_gradients()
(block_num=8)
Why did you use base_model.base_model here, and what is the connection to "3rd layer of the multimodal heads"?

Hi, thanks for sharing. ALBEF is wonderful and your code is well orginized.
What bothers me now is the training process takes too much time. It takes about 10 hours to finetune VQA traning tast per epoch with 3 RTX 3090 GPUs. I'm wondering if there is a way to speed up the training process somehow?
Thanks again for your work.

Thank you a lot for your outstanding work! I'm having problems with Visual Grounding task:

1. How did you get refcoco.pth?
   During Fine-tuning according to this provided procedure, a 3.3G checkpoint_best.pth file, as large as the pretrained model, is generated; However the checkpoint: refcoco.pth you've given is only 800M, Could you explain how you managed to shrink the model size?
   I tried distill :True and distill :False in config file, making no difference to the final size.
   python -m torch.distributed.launch --nproc_per_node=8 --use_env Grounding.py \ --config ./configs/Grounding.yaml \ --output_dir output/RefCOCO \ --gradcam_mode itm \ --block_num 8 \ --checkpoint [Pretrained checkpoint, size 3.3G]

2. How to evaluate refcoco.pth?
   Setting distill: False in config file does not work for me;
   python -m torch.distributed.launch --nproc_per_node=8 --use_env Grounding.py \ --config ./configs/Grounding.yaml \ --output_dir output/RefCOCO_albefpth \ --gradcam_mode itm \ --block_num 8 \ --evaluate \ --checkpoint refcoco.pth
   Drops following KeyError problem:
   Traceback (most recent call last): File "Grounding.py", line 295, in <module> main(args, config) File "Grounding.py", line 187, in main state_dict = checkpoint['model'] KeyError: 'model'

3. How to visualize the 3.3G checkpoint_best.pth file generated by fine-tuning?
   During Fine-tuning, the [val, test_A, test_B] metrics data printed out seems fine. However, the visualization.ipynb only works for refcoco.pth, but not works for the 3.3G checkpoint_best.pth generated by fine-tuning, the heat map is totally mess, not as expected. There seems a gap between checkpoint_best.pth and refcoco.pth.

Hi, when I runned the Retrieval.py, I met this issue. How could I repair it?

Traceback (most recent call last):
File "Retrieval.py", line 382, in
main(args, config)
File "Retrieval.py", line 305, in main
train_stats = train(model, train_loader, optimizer, tokenizer, epoch, warmup_steps, device, lr_scheduler, config)
File "Retrieval.py", line 51, in train
loss_ita, loss_itm = model(image, text_input,alpha=alpha, idx=idx)
File "/home/ma-user/anaconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/anaconda/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/ma-user/anaconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/ma-user/modelarts/user-job-dir/ALBEF/models/model_retrieval.py", line 130, in forward
neg_idx = torch.multinomial(weights_t2i[b], 1).item()
RuntimeError: invalid multinomial distribution (sum of probabilities <= 0)