Word-by-word generation about llm-workflow-engine

[Tecuya]

Oh this is already built, you can use the 'stream' command to enable streaming of responses from chatgpt (or start the program with the -s parameter).

It's not possible to use the markdown renderer to show the output when it streams it, so there's no word wrapping or other markdown formatting but it's still very usable.

Maybe streaming should be turned on by default..

[Robiemaan]

Nice. Do you happen to know why there's such a long delay before output starts showing up? Is it because the playwright browser has to wait for the entire response to be generated on chatgpt and then it starts outputting it to the CLI?

[mmabrouk]

When you use the stream parameter, the only delay you get is the one from openai returning the argument. You can see what exactly happens under the hood by changing the following line

chatgpt = ChatGPT(headless=not install_mode, timeout=60, **extra_kwargs)
to
chatgpt = ChatGPT(headless=False, timeout=60, **extra_kwargs)

and checking what happens in browser

[blz777]

It's not possible to use the markdown renderer to show the output when it streams it, so there's no word wrapping or other markdown formatting but it's still very usable.

After the answer has been completed, I use the /chat command to reload the current chat and then the markdown is properly formatted.