Comments (19)
Funny never spotted, but I've used a lot the streams... Which client?
from spring-ai.
@RestController
@AllArgsConstructor
@RequestMapping("v1/ai")
public class SimpleAiController {
private final OpenAiChatClient chatClient;
@GetMapping("/generateStream")
public Flux<ChatResponse> generateStream(@RequestParam(value = "message", defaultValue = "Tell me a joke") String message) {
Prompt prompt = new Prompt(new UserMessage(message));
return chatClient.stream(prompt);
}
}
Error:
org.springframework.ai.openai.api.OpenAiApi#chatCompletionStream
There's an extra "data:" here.
from spring-ai.
I reviewed the source code at https://github.com/openai/openai-node and saw that the strings have been processed, with 'data' being used as a 'fieldname'.
from spring-ai.
org.springframework.ai.openai.api.OpenAiApiIT#chatCompletionStream
I noticed that in the automated testing section of the spring-ai test code, despite the tests should have already been passed successfully, I still encountered issues when executing the test code myself, and it failed to pass the tests. This is very confusing, as logically, there should be no problems.
from spring-ai.
I've tested against OpenAI service and cannot reproduce the wrong behaviour... can you make a test?
from spring-ai.
@Grogdunn
"/v1/chat/completions" stream mode.
I am not directly using the OpenAI API, but rather accessing it through a proxy service. Debugging revealed that Spring WebFlux selects the appropriate MessageReader based on the "content-type". Since the "content-type: text/plain" did not engage the ServerSentEventHttpMessageReader, it failed to process the "data:" prefix. I suspect this is most likely the case, and I am still confirming this issue.
from spring-ai.
"/v1/chat/completions" stream mode.
The response header from the proxy is "content-type: text/plain; charset=utf-8", but it should correctly be "text/event-stream".
from spring-ai.
@Grogdunn My issue has been resolved; it was because of the response header.
from spring-ai.
Funny never spotted, but I've used a lot the streams... Which client?
OpenAiChatClient,3.5 and 4
from spring-ai.
I've tested against OpenAI service and cannot reproduce the wrong behaviour... can you make a test?
Yes, sometimes it happens, not necessarily
from spring-ai.
I wonder if it is in the chatCompletionStream
method of the OpenAiApi
class that this is used:
.floatMap((mono) -> {
return mono;
})
It might be worth trying to change this method to:
.concatMap((mono) -> {
return mono;
})
from spring-ai.
I have a similar problem
this is my log:
prompt is "1+1="
expect is "1+1=2"
actual is "12+1="(wrong)
2024-05-21 17:05:50.035 | INFO 39472 | reactor-http-nio-4 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content() result()]
2024-05-21 17:05:50.036 | INFO 39472 | task-8 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content(1) result(1)]
2024-05-21 17:05:50.037 | INFO 39472 | task-9 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content(2) result(12)]
2024-05-21 17:05:50.037 | INFO 39472 | task-10 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content(+) result(12+)]
2024-05-21 17:05:50.038 | INFO 39472 | task-11 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content(1) result(12+1)]
2024-05-21 17:05:50.038 | INFO 39472 | task-12 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content() result(12+1)]
2024-05-21 17:05:50.039 | INFO 39472 | task-13 [TID: N/A] c.i.y.m.a.s.impl.AiChatServiceImpl | [xxx][content(=) result(12+1=)]
from spring-ai.
Bug description Calling Fluxstream() returns unordered results Environment Spring AI version 1.0.0-SNAPSHOT , Java version 17,openai
// 注意:Schedulers.immediate() 目的是,避免默认 Schedulers.parallel() 并发消费 chunk 导致 SSE 响应前端会乱序问题
this work for me
from spring-ai.
Bug description Calling Fluxstream() returns unordered results Environment Spring AI version 1.0.0-SNAPSHOT , Java version 17,openai
// 注意:Schedulers.immediate() 目的是,避免默认 Schedulers.parallel() 并发消费 chunk 导致 SSE 响应前端会乱序问题 this work for me
return streamResponse.publishOn(Schedulers.immediate()).map(chunk -> {
from spring-ai.
The LLM expecially the generative one, are not good in math. All model currently in production respond with something "similar" a good response, but not the correct response.
Some models (gemini 1.5 pro I think) reach more accuracy, but not correctness at 100%. So
from spring-ai.
The LLM expecially the generative one, are not good in math. All model currently in production respond with something "similar" a good response, but not the correct response.
Some models (gemini 1.5 pro I think) reach more accuracy, but not correctness at 100%. So
⚠️ BEWARE!
Thank you.
My purpose is mainly to give llm a simple question to test whether spring webflux + Spring-AI will have sse out-of-order problems
from spring-ai.
Bug description Calling Fluxstream() returns unordered results Environment Spring AI version 1.0.0-SNAPSHOT , Java version 17,openai
// 注意:Schedulers.immediate() 目的是,避免默认 Schedulers.parallel() 并发消费 chunk 导致 SSE 响应前端会乱序问题 this work for me
thanks
from spring-ai.
Bug description Calling Fluxstream() returns unordered results Environment Spring AI version 1.0.0-SNAPSHOT , Java version 17,openai
// 注意:Schedulers.immediate() 目的是,避免默认 Schedulers.parallel() 并发消费 chunk 导致 SSE 响应前端会乱序问题 this work for me
thanks
from spring-ai.
Flux stream = chatClient.stream(prompt);
return stream.publishOn(Schedulers.immediate()).map(chunk -> chunk);
from spring-ai.
Related Issues (20)
- Bedrock Titan Embeddings - Malformed input request / 2 validation errors HOT 1
- Generate complex Json schemas with BeanOutputConverter
- MilvusVectorStore need support milvus sdk 'withPartitionName' method when insert or delete
- VectorStoreChatMemoryAdvisor is not working correctly with PGvector
- Why is there no entry for webclient to set up http proxy? I don’t want to set up a proxy for the entire service. 😭 HOT 12
- AWS Bedrock models tools support HOT 2
- Feature Request: Support for Invoking HTTP-based APIs via OpenAPI Spec HOT 1
- Add support for the Amazon Bedrock Converse API HOT 5
- Add Support for get usage tokens HOT 2
- neo4j Vector store does not allow more than 2048 embedding dimensions HOT 1
- SpringAI does not follow the AWS SDK's default region resolution rules HOT 1
- AzureOpenAiChatModel choices is null
- Gemini model doesn't declare dependency on spring-ai-retry
- java.lang.RuntimeException: java.lang.IllegalArgumentException: argument type mismatch
- java.lang.RuntimeException: java.lang.IllegalArgumentException: argument type mismatch
- Get 'The engine PyTorch was not able to initialize' when call TransformersEmbeddingModel's embed method concurrently
- Introduce basic starter dependency for Hugging Face
- odd issue creating embeddings with Ollama HOT 1
- VertexAiGeminiChatModel does not support multiple Function Calls
- Exact similarity match with Redis Vector Store always retuning empty list
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spring-ai.