Improve `from_quantized` loading time about autogptq

[PanQiWei]

The warmup used every time when model is loaded can help triton to tune the best configuration that can make inference as fast as possible. But since it's kindly like grid search, hyper-params are manually set (hard coded), it may have no effect for some card. So now I've added warmup_triton to from_quantized api, one can set it to False to skip warmup stage and then model loading can be fast.

I will also find a way to let users to cache the best triton configuration and save as a file so that one can warmup only once.

[PanQiWei]

Will close this issue for loading time problem of .from_quantized has been fixed. Feel free to reopen or raise a new issue if you still encounter similar problem.

I will also find a way to let users to cache the best triton configuration and save as a file so that one can warmup only once.

For this I will add into a backlog as a future work.

[PanQiWei]

Hi, when using triton with from_quantized, there is a auto_tune_warmup is executed. Maybe I should make the warmup as an option for users.

[philschmid]

Is the "warm up" needed after you saved the model once?

[philschmid]

Is the "warm up" needed after you saved the model once?

[larekrow]

Interested in the response to @philschmid's question as well. If so, what are the implications of not using the warm up?

[philschmid]

Thank you will give it a try later this week!