Comments (7)
The --disk
option has been implemented. With it, I have been able to load a 30b model on my RTX 3090 with
python server.py --model galactica-30b --auto-devices --disk
The performance is crap (like 1 word every 10 seconds), but it works.
from text-generation-webui.
Token streaming would be a great addition!
from text-generation-webui.
Text streaming has been implemented. It is still experimental, but it seems to work. Can you try it?
from text-generation-webui.
@oobabooga Text streaming did work, however, I think it introduced other bugs - in chat mode, the generate button would sometimes not respond and the input box will not automatically be cleared after generate button is clicked, and input box sometimes clears itself when typing. The other modes seems to be working properly though.
from text-generation-webui.
Can you git pull
and try again? I might have fixed those inconsistencies with the latest commits.
from text-generation-webui.
Your fix worked. I'll still remain this issue as open since the other request (disk cache that generates once-and-for-all) isn't implemented yet, but take your time. Thanks for the work!
from text-generation-webui.
I am glad to hear that. For now, I have moved the stream work into a separate stream
branch because the streaming caused a 50% decrease in performance on my GPU. Once things become more stable I will move it back to the main branch.
from text-generation-webui.
Related Issues (20)
- OpenAI api issues with Huggingface Chat UI
- Llama 3.1 GGUF incompatibility using latest release of llama.cpp and text-generation-webui. HOT 11
- Exllamav2 qcache support (Q4, Q6, Q8) HOT 1
- Exllama v2 crashes when starting to load in the third gpu HOT 4
- VideoLLM-online, CosyVoice, and SenseVoice Integration
- Error at install (Windows - CPU only) <-> ModuleNotFoundError: No module named 'yaml' HOT 3
- When running the start_windows.bat file in the Windows interface, ModuleNotFoundError: No module named 'yaml' appears And I didn't see some important tool files in the download list HOT 1
- Smarter crash diagnosis
- Lorebook/worldbook functionality HOT 1
- API examples in docs outdated HOT 2
- Can't train LoRA for Phi-3-medium-128k-instruct (transformers)
- Help Needed with Local Dataset Support for PPL Evaluation
- Adding files to the context
- Llama-cpp fails to install HOT 1
- git clone on linux is broken HOT 12
- MS are releasing a DeepSpeed WHL for Windows :)
- Text Generation/Responses NOT following user prompts/responses/replies/inputs (ie. Sillytavern + Textgen)
- Only able to generate gibberish (Wizard-Vicuna-13B-Uncensored-GPTQ) HOT 3
- So SLOW with NVidia GPU and Codestral model HOT 3
- "Start Reply with" in the chat API
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from text-generation-webui.