llama.cpp api about twinny

[rjmacarthy]

Thanks @kv-gits for the suggestion, it's possible. Do you know which model would support FIM completions using llama.cpp? Edit I am working on this in #47

[rjmacarthy]

Thanks for that I am creating a new release now version 3 which adds support for llama.cpp. You'll need to update some settings in the extension for it to work. FIM is working with the model I tried, but with the same model chat is spitting out garbage, there is an option to use a different endpoint for chat if required.

[rjmacarthy]

I just released version v3.0.0 which includes support for llama.cpp using the --server option. The host, port and endpoints need to be configured in the extension settings. Please let me know how you get on.

Many thanks

[kv-gits]

As I found in issues, llama.cpp supports FIM tokens and all three models of tweeny have their gguf version on hf

https://github.com/ggerganov/llama.cpp/tree/master/examples/infill

[rjmacarthy]

I think there was a small bug which still checked for model availability on Ollamo on v3.0.0 I fixed it and released v3.0.2 to stop this check.

[kv-gits]

Would be nice to turn off "install ollama" window after install, and also provide some example cmd to run llamacpp server for quick start