Lonnnnnnnnng context load time before generation about gptq-for-llama

[qwopqwop200]

What code did you run?

[USBhost]

I would like to confirm this issue as well. It really becomes noticeable when they're running chat vs normal/notebook. Chat with nothing set runs really fast but once you start putting context etc... start up speed just takes a nose dive.

4bit 65b on my A6000

[plhosk]

In the case of llama.cpp, when a long prompt is given you can see it output the provided prompt word by word at a slow rate even before it starts generating anything new. It's directly evident that it takes a longer time to to get through larger prompts. I guess a similar thing is happening here.

[USBhost]

So I compared B&B 8bit and GPTQ 8bit and GPTQ was the only one that had a start delay. Something is causing a delay before anything starts generating.

[Digitous]

Runs pretty well once it starts.. not sure if it's loading something, reading layers then inferencing. It's definitely got its quirks of new tech, might just be a case of "well that's how it works"

[aljungberg]

Probably fixed now, see #30.

[qwopqwop200]

I think this issue has been resolved.