docs: llama.cpp/GGUF CPU offloading no longer present? about jan

[Van-QA]

hi @mr-september,

1. For model.json, here is how to make modifications to the value of the setting to include ngl:
   

2. For nitro.json, the engines folder no longer exists, we refactored it in the Jan app. and will make modifications to docs shortly

cc: @cahyosubroto @aindrajaya @irfanpena to update 2 points mentioned above into our docs.
Note that for No.2, we will need correction for nitron.json or engines folder in everyplace in our docs, as it no longer exists.

[Van-QA]

hi @irfanpena, all 5 parameters here can be applied to model.json:

  "ctx_len": 2048, 
  "ngl": 100,
  "cpu_threads": 1,
  "cont_batching": false,
  "embedding": false

where

"ctx_len": 2048, 
  "ngl": 100,

are more important (more impact) than the rest
Thank you

[mr-september]

Thanks @Van-QA, that seems to be working on my end. Feel free to close this issue any time the team deems the docs updated.

If I may suggest, could this be added into the GUI as well? Ideally with some kind of general estimation (e.g. jan.ai detects my system has 8GB VRAM and 32GB RAM, and the model size is 12GB - suggest default 50% offload, etc.)

[Van-QA]

Linking the issue to #2208, related to RAM/VRAM utilization.

Thanks @Van-QA, that seems to be working on my end. Feel free to close this issue any time the team deems the docs updated.

If I may suggest, could this be added into the GUI as well? Ideally with some kind of general estimation (e.g. jan.ai detects my system has 8GB VRAM and 32GB RAM, and the model size is 12GB - suggest default 50% offload, etc.)
#2859 (comment)

[irfanpena]

@Van-QA Can the parameters in the https://jan.ai/docs/built-in/llama-cpp for nitro.json be used for the settings parameters in model.json?