what size of your A100 gpu's memory? about albef

[LiJunnan1992]

Hi, I used the 40GB gpu.
Data reading was not a major speed bottleneck, but you could try to resize the image on hard drive because some images could have a high resolution.

[shoutOutYangJie]

how about cost time of  pretrain stage ?

…

------------------ Original ------------------ From: Junnan Li ***@***.***> Date: Wed,Dec 8,2021 5:45 PM To: salesforce/ALBEF ***@***.***> Cc: shoutOutYangJie ***@***.***>, Author ***@***.***> Subject: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue#31) Hi, I used the 40GB gpu. Data reading was not a major speed bottleneck, but you could try to resize the image on hard drive because some images could have a high resolution. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[LiJunnan1992]

Using 8 A100 GPUs, it takes 2-3 days with 4M images, and around 7-8 days with 14M images. You could make training faster by reducing the image resolution to 224 and increasing the batch size, the performance would be roughly the same. You can also try some other memory reduction techniques such as zero-redundancy optimizer.

[shoutOutYangJie]

can i use 8 32gb-v100  gpus to reproduce your training result?  by the way, the code of data preprocessing (filter some pairs) will be released?

…

------------------ Original ------------------ From: Junnan Li ***@***.***> Date: Wed,Dec 8,2021 6:05 PM To: salesforce/ALBEF ***@***.***> Cc: shoutOutYangJie ***@***.***>, Author ***@***.***> Subject: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue#31) Using 8 A100 GPUs, it takes 2-3 days with 4M images, and around 7-8 days with 14M images. You could make training faster by reducing the image resolution to 224 and increasing the batch size, the performance would be roughly the same. You can also try some other memory reduction techniques such as zero-redundancy optimizer. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[LiJunnan1992]

Yes I think you could.
The pre-training dataset annotation (image paths and text) is released.

[sunanhe]

Hi, do you get similar performance using 8 V100 GPUs?

can i use 8 32gb-v100  gpus to reproduce your training result?  by the way, the code of data preprocessing (filter some pairs) will be released?
…
------------------ Original ------------------ From: Junnan Li @.> Date: Wed,Dec 8,2021 6:05 PM To: salesforce/ALBEF @.> Cc: shoutOutYangJie @.>, Author @.> Subject: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue#31) Using 8 A100 GPUs, it takes 2-3 days with 4M images, and around 7-8 days with 14M images. You could make training faster by reducing the image resolution to 224 and increasing the batch size, the performance would be roughly the same. You can also try some other memory reduction techniques such as zero-redundancy optimizer. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[shoutOutYangJie]

yes，I got similar result

…

------------------ Original ------------------ From: Richard_He ***@***.***> Date: Wed,Jan 12,2022 4:03 PM To: salesforce/ALBEF ***@***.***> Cc: shoutOutYangJie ***@***.***>, State change ***@***.***> Subject: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue#31) Hi, do you get similar performance using 8 V100 GPUs? can i use 8 32gb-v100  gpus to reproduce your training result?  by the way, the code of data preprocessing (filter some pairs) will be released? … ------------------ Original ------------------ From: Junnan Li @.> Date: Wed,Dec 8,2021 6:05 PM To: salesforce/ALBEF @.> Cc: shoutOutYangJie @.>, Author @.> Subject: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue#31) Using 8 A100 GPUs, it takes 2-3 days with 4M images, and around 7-8 days with 14M images. You could make training faster by reducing the image resolution to 224 and increasing the batch size, the performance would be roughly the same. You can also try some other memory reduction techniques such as zero-redundancy optimizer. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[yangbang18]

@shoutOutYangJie
Hi, it seems that you had reproduced the results with 8 V100 GPUs.

1. Did you use the same configurations as in Pretrain.yaml?
2. How many hours per epoch it took for the training?
3. Have you tried to reduce the image resolution from 384 to 224?

Looking forward to your reply.

[shoutOutYangJie]

1. Yes 2. about several hours. 3. No. I can say the work is solid. very good.

…

------------------ 原始邮件 ------------------ 发件人: "salesforce/ALBEF" ***@***.***>; 发送时间: 2022年6月18日(星期六) 晚上11:01 ***@***.***>; ***@***.******@***.***>; 主题: Re: [salesforce/ALBEF] what size of your A100 gpu's memory? (Issue #31) @shoutOutYangJie Hi, it seems that you had reproduced the results with 8 V100 GPUs. Did you use the same configurations as in Pretrain.yaml? How many hours per epoch it took for the training? Have you tried to reduce the image resolution from 384 to 224? Looking forward to your reply. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>