the model quantized is not performant about autogptq

[qwopqwop200]

This was implemented inefficiently due to the complexity of implementing act-order and groupsize at the same time. This is also why I recommend triton in general.

[cxfcxf]

This was implemented inefficiently due to the complexity of implementing act-order and groupsize at the same time. This is also why I recommend triton in general.

thanks thats actually solved my problem, it seems triton moved all model to VRAM which make sense its faster, was not aware that the default cuda version uses VRAM + DRAM, no wonder its slow

i was working on embedding project, be able to load large model in small VRAM really helped, since most of people would not like to feed sensitive data to openai model.

btw, there is maybe a typo on the warning message when i try to load it
WARNING - use_triton will force moving the hole model to GPU, make sure you have enough VRAM.
this means whole right?

[TheBloke]

btw, there is maybe a typo on the warning message when i try to load it
WARNING - use_triton will force moving the hole model to GPU, make sure you have enough VRAM.
this means whole right?

It does, and I've just pushed a PR to fix the typo: #40