Comments (7)
Are you trying to configure CatAI only to use some CPU/GPU cores?
In general llama.cpp
is the actual engine behind the scenes, you can configure with CatAI how many CPU/GPU cores and memory the model uses.
You can configure CatAI like this:
-
Add custom settings, which you can find here: https://withcatai.github.io/node-llama-cpp/types/LlamaModelOptions.html
from catai.
Hello,
I am trying to get the model to utilize all of the cpu/gpu that I have available...
Looking at the utilization, node
is using 6 cores (of 16) and ~13% of a v100 gpu.
I have tried altering some of the settings but I wasn't able to see a difference.
By chance is it because of the model that I am using?
vicuna-7b-v1.5-q4_1
Thank you.
from catai.
Yes, this may be related. The binding of node-llama-cpp
has some bugs with the configuration.
This binding allows you to use ggmlV3
(legacy format).
The recommended format is gguf
(node-llama-cpp-v2
),
If you want to utilize more VRAM you should use a bigger model, such as
llama-2-13b-chat-limarp-v2-merged-q3_k_s
phind-codellama-34b-q3_k_s
You can check this page for more models: https://huggingface.co/TheBloke
Most GGUF models are supported, you can install a custom model by coping the model link to catai install
The more parameter the model the more resources it utilizes, so this model (70B) https://huggingface.co/TheBloke/ARIA-70B-V2-GGUF
Can be pretty heavy.
If a model split into files like https://huggingface.co/TheBloke/Falcon-180B-GGUF/tree/main
You can install it like this:
canai install model-part-a.gguf,model-part-b.gguf
I have not tried this method yet, so if there is a problem feel free to report it :)
from catai.
Ok. I'll continue to look into it.
For the gif that is in the documentation, would you be able to tell me what model was used and about how long the answer took to generate? I feel that what I am seeing is slower than what is to be expected.
Thank you again.
from catai.
This git was generated from screen recording in mac pro m1. llama.cpp has spesifict optimization for mac silicon.
If I remember right this was from vicuna-13b-v1.5-q4_1
(but older version of the model).
I will soon update the models index to support vicuna with GGUF (gguf have more optimizations).
Are your cuda working when the model generating tokens?
from catai.
I uploaded new models, in my Mac token/s is faster than in the gif, I use vicuna-7b-16k-q4_k_s
Screen.Recording.2023-09-21.at.12.02.31.mov
from catai.
Thank you.
I'll need to go back and see what is happening on my side as it is much slower than what you are showing.
I'll close this out.
Thank you again.
from catai.
Related Issues (20)
- Error on Win 11 HOT 4
- Alternate GUI won't load
- CatAI crashes when quitting a browser or when starting CatAI when a browser is already open. HOT 10
- Catastrophic - CatAI is completely broken (Segmentation fault core dumped fatal error) HOT 11
- Unable to install `catai` onto an ARM development board HOT 3
- Failed to load prebuilt binary for platform "linux" "arm64". HOT 4
- Error: The connection lost, check the server status and refresh the page. HOT 11
- cat dont send answer HOT 17
- configure website down HOT 2
- Does not work on ubuntu 22.04 & node 20 HOT 10
- logo in docs have 2 404 errors HOT 2
- Is there a way to specify a path where to store model ? HOT 3
- How to install and run catai from packages? HOT 1
- Otherbrain HF open data integration HOT 2
- How to use with cuda? HOT 5
- connectionError http://127.0.0.1:3000/assets/index-3673c735.js:39 HOT 1
- System_prompt HOT 2
- remote-catai does not work , should wait ws.on('open')
- Made a Discord bot for your project, still in development tho
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from catai.