Comments (7)
This is indeed much needed. Will post when this is pushed.
from guidance.
0.0.56 now has a first version of streaming support. We will need to also update library calls to update variables more frequently, but any comments or feedback are welcome over in #129
from guidance.
As a follow up - it would be great if the library allowed doing something like the following without blocking the event loop by internally calling openai.ChatCompletion.acreate
instead of create
.
import asyncio
import guidance
guidance.llm = guidance.llms.OpenAI(model='gpt-3.5-turbo')
program = guidance("""
{{#user~}}
Write A poem about sparkling water
{{~/user}}
{{#assistant~}}
{{gen 'poem'}}
{{~/assistant}}
""")
async def counter():
for x in range(5):
print(x, flush=True)
await asyncio.sleep(1)
async def generate():
# Currently unsupported
async for chunk in program(async_mode=True, stream=True):
print(chunk, flush=True)
async def main():
await asyncio.gather(generate(), counter())
asyncio.run(main())
from guidance.
This is indeed much needed. Will post when this is pushed.
Thank you @slundberg . The guidance
library has tremendous potential for controlling nuanced chatbot behavior vs. langchain and we appreciate your efforts.
from guidance.
I've got a working workaround here https://github.com/andaag/chattergpt/blob/e199ccc99b7d8476275f4ed5720b268d506490c1/shared.py#L41 - it's a bit ugly as it accesses internal classes, but should work fine until something is implemented.
from guidance.
I've got a working workaround here https://github.com/andaag/chattergpt/blob/e199ccc99b7d8476275f4ed5720b268d506490c1/shared.py#L41 - it's a bit ugly as it accesses internal classes, but should work fine until something is implemented.
Thanks for the link, using the internal done function is what I was missing to get hacky streaming working with FastAPI
from guidance.
Closing for now, feel free to reopen if needed.
from guidance.
Related Issues (20)
- Support for case sensitive Literal/Enum and maxLength, minLength for strings HOT 12
- support for transformer LoRA adapters HOT 5
- [Bug] `sequence()` not quite replacement for `zero_or_more()`
- LlamaCpp model crashes with multi-token characters HOT 5
- `ModelStream` should be a drop-in replacement for `Model` or their API differences should be documented HOT 1
- Cannot import guidance w/ vertexai due to protobuf version mismatch HOT 5
- dedent=True doesn't work in python 3.12 HOT 5
- Stream() does not obey the roles function HOT 2
- Unable to connect to Hugging Face Dedicated Endpoints? HOT 3
- OpenAI models does not support multi modal? HOT 1
- Add sample custom handler.py integration with HuggingFace HOT 3
- Telegram or Discord dedicated channels? HOT 2
- Cannot instantiate `Transformers("microsoft/Phi-3-mini-4k-instruct")` HOT 15
- [Bug] Latest Transformers disagrees with GPT2 on MacOS-ARM HOT 2
- Cannot append to a variable created with model.set HOT 2
- Better error messages when failing to load models with sentencepiece based tokenizers HOT 3
- CHAT_TEMPLATE_CACHE keys are not what I expect HOT 1
- LlamaCpp always using gpt2 tokeniser HOT 1
- Irrelevant link
- Transformers `past_key_values` deprecated
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from guidance.