Comments (5)
hi @mr-september,
-
For model.json, here is how to make modifications to the value of the setting to include ngl:
-
For nitro.json, the engines folder no longer exists, we refactored it in the Jan app. and will make modifications to docs shortly
cc: @cahyosubroto @aindrajaya @irfanpena to update 2 points mentioned above into our docs.
Note that for No.2, we will need correction for nitron.json or engines folder in everyplace in our docs, as it no longer exists.
from jan.
hi @irfanpena, all 5 parameters here can be applied to model.json:
"ctx_len": 2048,
"ngl": 100,
"cpu_threads": 1,
"cont_batching": false,
"embedding": false
where
"ctx_len": 2048,
"ngl": 100,
are more important (more impact) than the rest
Thank you
from jan.
Thanks @Van-QA, that seems to be working on my end. Feel free to close this issue any time the team deems the docs updated.
If I may suggest, could this be added into the GUI as well? Ideally with some kind of general estimation (e.g. jan.ai detects my system has 8GB VRAM and 32GB RAM, and the model size is 12GB - suggest default 50% offload, etc.)
from jan.
Linking the issue to #2208, related to RAM/VRAM utilization.
Thanks @Van-QA, that seems to be working on my end. Feel free to close this issue any time the team deems the docs updated.
If I may suggest, could this be added into the GUI as well? Ideally with some kind of general estimation (e.g. jan.ai detects my system has 8GB VRAM and 32GB RAM, and the model size is 12GB - suggest default 50% offload, etc.)
#2859 (comment)
from jan.
@Van-QA Can the parameters in the https://jan.ai/docs/built-in/llama-cpp for nitro.json
be used for the settings parameters in model.json?
from jan.
Related Issues (20)
- bug: seams talk to docs not optimized HOT 1
- bug: There is only one model to use via Nvidia NIM API HOT 1
- feat: DeepSeek API support
- feat: Adding keyboard shortcuts to clean and to delete current thread
- The \"readableStream\" argument must be an instance of ReadableStream. Received an instance of PassThrough
- feat: Ability to add files from desktop to the chat HOT 2
- bug: I just upgraded to newer version and it became super slow it taking mintues to open Apple Silicon HOT 3
- feat: Add logos to the rest of the API providers (Groq,Openrouter,...)
- Jan's new starter screen
- bug: AppImage of version v0.5.2 does not show Claude Sonnet 3.5 HOT 2
- bug: Token Retrieval for HTML and CSS is not arriving to the platform or getting rendered correctly with GPT-4o
- feat: Windows on ARM support (with Snapdragon X series NPU)
- bug: local models won't start HOT 3
- add links to the sources of models when i'm looking at them in Jan HOT 1
- feat: Don't hide the vertical scroll bar
- feat: Save sidebars toggled state
- bug: [Manual inported model not visible] HOT 6
- bug: Model Hub is not fully responsive
- Improve Jan's response readability
- bug: Downloading a model using huggingface url doesnt work on mac with the latest build. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jan.