[Feature Request] Add disk cache and Token streaming about text-generation-webui

[oobabooga]

The --disk option has been implemented. With it, I have been able to load a 30b model on my RTX 3090 with

python server.py --model galactica-30b --auto-devices --disk

The performance is crap (like 1 word every 10 seconds), but it works.

Token streaming would be a great addition!

[oobabooga]

Text streaming has been implemented. It is still experimental, but it seems to work. Can you try it?

[Silver267]

@oobabooga Text streaming did work, however, I think it introduced other bugs - in chat mode, the generate button would sometimes not respond and the input box will not automatically be cleared after generate button is clicked, and input box sometimes clears itself when typing. The other modes seems to be working properly though.

[oobabooga]

Can you git pull and try again? I might have fixed those inconsistencies with the latest commits.

[Silver267]

Your fix worked. I'll still remain this issue as open since the other request (disk cache that generates once-and-for-all) isn't implemented yet, but take your time. Thanks for the work!

[oobabooga]

I am glad to hear that. For now, I have moved the stream work into a separate stream branch because the streaming caused a 50% decrease in performance on my GPU. Once things become more stable I will move it back to the main branch.