Is there a quantization version support in the future? about phogpt

[datquocnguyen]

Not sure why Bloom is mentioned here.
This issue was fixed for MPT-type models (including PhoGPT): casper-hansen/AutoAWQ#155
I will spend time looking into it.

[datquocnguyen]

Is there any problem of employing: https://huggingface.co/docs/transformers/quantization#quantization and/or: https://github.com/casper-hansen/AutoAWQ with limited computing resources?

[phatjkk]

Yeah I have used AutoAWQ version 0.1.7 (which support Bloom) for quantization PhoGPT-7B5-Instruct to 4 bit GEMM group-size 128 and found this error:

The AutoAWQ working well with vietcuna-7b-v3 (which used Bloom too).

Here is the code I have used:
https://github.com/phatjkk/test_llm/blob/main/FINALL_GENERATOR_Quantization.ipynb

I don't know what the problem is. I tried some previous version of transformer but may not work!
Are you try it yet?
Thank you!