Support for LM Studio about twinny

[rjmacarthy]

Hey just released version 3.5.0 which includes support for LMStudio, please check it out and let me know your thoughts 😄

#98

Many thanks,

[rjmacarthy]

Hey, thank you for the feature request.

I have decided to work on this in PR #98

I will keep you updated when it's ready. It's looking promising so far, I'd never heard of LM studio before this but it's really awesome, thanks for pointing me towards it.

Many thanks,

[rburgst]

My pleasure. I am glad you like it as much as I do.

[rjmacarthy]

This is ready, closing.

[rburgst]

Hi, i tried using it with
deepseek-coder:base
and getting this log on the server in lm studio

[2024-02-16 09:06:39.982] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-02-16 09:06:39.983] [INFO] Received POST request to /v1/completions with body: {
  "model": "deepseek-coder:base",
  "prompt": "<｜fim▁begin｜>\n\n// Language: Typescript (typescript) \n// File uri: file:///Users/Rainer.Burgstaller/git/emundo/exercises/dt-test-app/src/app/game-of-life.ts (typescript) \nclass Cell<｜fim▁hole｜><｜fim▁end｜>",
  "stream": true,
  "max_tokens": 250,
  "temperature": 0.2
}
[2024-02-16 09:06:39.983] [INFO] Provided inference configuration: {
  "n_threads": 4,
  "n_predict": 250,
  "top_k": 40,
  "min_p": 0.05,
  "top_p": 0.95,
  "temp": 0.2,
  "repeat_penalty": 1.1,
  "input_prefix": "[INST]",
  "input_suffix": "[/INST]\\n",
  "antiprompt": [
    "[INST]"
  ],
  "pre_prompt": "",
  "pre_prompt_suffix": "<</SYS>>[/INST]\\n",
  "pre_prompt_prefix": "[INST]<<SYS>>\\n",
  "seed": -1,
  "tfs_z": 1,
  "typical_p": 1,
  "repeat_last_n": 64,
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "n_keep": 121,
  "logit_bias": {},
  "mirostat": 0,
  "mirostat_tau": 5,
  "mirostat_eta": 0.1,
  "memory_f16": true,
  "multiline_input": false,
  "penalize_nl": true
}
[2024-02-16 09:06:39.983] [INFO] Streaming response..
[2024-02-16 09:06:39.984] [INFO] [LM STUDIO SERVER] Processing...
[2024-02-16 09:06:40.189] [ERROR] [Server Error] {"title":"unordered_map::at: key not found"}
[2024-02-16 09:06:40.191] [INFO] [LM STUDIO SERVER] Finished streaming response
[2024-02-16 09:06:40.191] [ERROR] [Server Error] {"code":"ERR_HTTP_HEADERS_SENT"}
[2024-02-16 09:06:40.195] [ERROR] [Server Error] {"code":"ERR_HTTP_HEADERS_SENT"}

I used the following model

and used the following twinny settings