Completions not working with ollama via ollama webui about twinny

[rjmacarthy]

I'll update the readme soon with Ollama web UI support. Many thanks for the help and the donation too!

[kha84]

I'm hitting the same issue, confirmed the bearer token is working with Postman. I've enabled "debug" in Twinny settings but can't seem to find the output anywhere, my Ollama server does not see any requests being made by Twinny.

I struggled with the same question. Here is how to view debug - #72 (comment)

Give it another try with your ollama web ui, but update the Twinny first to the latest version.
@rjmacarthy recently made some changes so Twinny now supports oobabooga, it might be the case that integration with ollama web ui was also fixed.

[kha84]

I'm not the author and no JS programmer myself, but based on a error it looks like the ollama-web-ui OpenAI API endpoint timeouts. Are you able to get responses if you send same request with using curl? See the #72 for examples how I did that for oobabooga/textgen-webui
Paste your output from running curl here, it might help @rjmacarthy to pin that down.

It looks like every inference software who claims to be OpenAI-API-compatible is still doing some things differently from each other.

[kha84]

Yeah, you see, in my example (for Jan.ai) the JSON for streamed responses looked different #72 (comment)

I'm pretty sure that different software wrap streamed responces into JSON differently. To support them all plugin needs to expose some configuration setting, a sort of path inside of JSON, where you can hint it, where exactly to look at for characters in the responded JSON and when the response is over. I hope @rjmacarthy will figure it out how it can be done, so all the users of different inference apps will get their silver bullet

[dlford]

Haha of course, it would be too easy if everyone just agreed on a standard and stuck with it 😂.

Thanks @kha84 for your insights!

@rjmacarthy please let me know if you need anything else from me on this issue.

[rjmacarthy]

Hey I did some debugging, but I am really not sure why it's not working right now, sorry.

[rjmacarthy]

Update: I think it's fixed now...I just tested it and it seems to work as long as the auth credentials are correctly set. Please test on the latest version and let me know, thanks.

[dlford]

@rjmacarthy that did, everything is working now, thank you!

[technovangelist]

why would you use the webui? Just use Ollama directly.

[rjmacarthy]

I'm not sure what could be causing it, but can look into it. Thanks for the report.

[dlford]

I'm hitting the same issue, confirmed the bearer token is working with Postman. I've enabled "debug" in Twinny settings but can't seem to find the output anywhere, my Ollama server does not see any requests being made by Twinny.

[dlford]

why would you use the webui? Just use Ollama directly.

I can't speak for the person who opened this issue, but I use the webui to secure ollama because it's exposed to the internet.

[dlford]

@kha84 oh I see, the logs are in the webview devtools console.

I did some more testing today and I think the bearer token is working, as the wrong token gets a 401 response as it should. But using the correct token this is what I get:

[dlford]

Interestingly, I only see the request in the ollama and webui pods when the token is invalid (401 unauthorized), when using the correct token there is no output from either in the logs.

[dlford]

Even more interestingly, using the Twinny chat sidebar completely hangs the Ollama server, it will not respond to any generate or chat requests after that has been attepmted until I restart the container it's running in.

Here is the output of the curl request, seems to work fine and not cause any hangs:

Same exact results with the code completion as the chat sidebar. curl works, but Twinny times out and hangs the Ollama server without any log output from the container.

@rjmacarthy

[dlford]

Twinny Settings:

{
  "twinny.apiHostname": "ollama.redacted-hostname.tld",
  "twinny.chatApiPort": 443,
  "twinny.fimApiPort": 443,
  "twinny.useTls": true,
  "twinny.apiBearerToken": "REDACTED",
  "twinny.chatApiPath": "/ollama/api/generate",
  "twinny.fimApiPath": "/ollama/api/generate",
  "twinny.enableLogging": true
}

[dlford]

@rjmacarthy any way I can help? I can add a user for you on my Ollama server for debugging?

[rjmacarthy]

Hey @dlford I think it's due to ollama webui backend supporting text/event-stream with stream: false for the content-type request header and all of the other API's which are supported use application/json and stream: true. I ran everything locally even the ollama webui server and requests we're working after I hacked authentication and cors but we're ending pre-maturely due to content-length errors, so I need more time or help.

[dlford]

@rjmacarthy that's weird, I'll try to find some time to explore the code base and tinker a bit. I appreciate you!