Possible to use int8 during pretraining stage? about llava

[152334H]

also just an fyi but the references to fastchat/train/train_mem.py in the README should probably be changed to llava/train/train_mem.py

[haotian-liu]

Hi @152334H, thanks for your interest in our work! Thanks for pointing out the script path issue, and I have just corrected it.

Regarding using INT8 during the pretraining stage, there may be issues doing so. The reason is that the Linear layer that is being projected, are optimized with the gradient that is backward propagated all the way from the last layer to the first layer of the LLM. When directly doing this INT8 loading, I guess some gradients will underflow / overflow.

I do know that some Alpaca-LORA projects use INT8 training, and I am interested in integrating these techniques into our project as well. I will also look into this after finishing all related release stuff. Please let me know if you are also interested in this and would like to contribute!

[haotian-liu]

Hi, I am closing this issue, due to the inactivity. Hope your problem has been resolved. If you have further concerns, please feel free to re-open or open a new issue. Thanks!

[152334H]

No problem.