marcom / llama.jl Goto Github PK
View Code? Open in Web Editor NEWJulia interface to llama.cpp, a C/C++ library for running language models
License: MIT License
Julia interface to llama.cpp, a C/C++ library for running language models
License: MIT License
It would be great to have https://docs.openwebui.com/ as (one of) the default interface(s) installed through Llama.jl.
I might give it a try at some point, but I am not that good at this sort of stuff.
Expectation: When I run run_chat
, I'd like to terminate the interactive session with CTRL+C (as per the llama.cpp manual).
Problem: When I press CTRL+C, the interrupt control sequence gets consumed by REPL and is not emitted. Ie, I cannot stop it and have to restart the REPL session
MWE
using Llama
model = "/Users/simljx/Documents/llama.cpp/models/rocket-3b-2.76bpw.gguf"
Llama.run_chat(; model, prompt="Say hi!", nthreads=1)
# press CTRL+C to terminate
Versions
julia> versioninfo()
Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: macOS (arm64-apple-darwin22.4.0)
CPU: 8 × Apple M1 Pro
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1)
Threads: 8 on 6 virtual cores
Environment:
JULIA_EDITOR = code
JULIA_NUM_THREADS = 8
trying to follow the instructions. Get stopped at the point of "download_model". Julia says it is not defined.
Thank you for creating this wrapper! I was about to do it myself -- glad I noticed the link in the past jll PRs :)
I wanted to test an idea with you.
People can use Llama.cpp directly if they want the low-level control, but that requires knowledge of prompt templates and compiling source code etc.
What if this package would serve as a Julia-only entry to have an LLM run on your laptop? Ie, no need to install anything else, we'll create a turnkey solution for you to get started.
Ollama is awesome and super user-friendly, but it's a separate application to download and its own limitations (eg, some performance issues, ngl defaults, etc)
There are many others (ooba, ..), but all are separate tools to install...
What do you think? I'm happy to draft a PR.
Objective:
Enhance the onboarding experience for first-time users of LLMs with llama.jl by simplifying the initial setup process.
Just: use Llama; run_server()
Proposal:
list_models
(following the example of MLJ and their models
)Benefits:
Feedback and suggestions are welcome.
Disclaimer: I'm an author of https://github.com/svilupp/PromptingTools.jl, so I'd leverage the API from there and deepen the integration
EDIT: I have other goals/aspirations for llama.cpp jll (eg, data extraction with the grammar), but I think we should first simplify the setup for users.
Unit tests in runtests.jl should use a tiny (as small as possible for testing) language model to run tests on.
A tiny model with all params set to zero should be possible to compress to a very small size and commited directly to this repo. Then running tests doesn't require any kind of download.
Note: 100 MB models seem to be available
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.