vortext / llama.clj

Run LLMs locally. A clojure wrapper for llama.cpp.

llama.clj

Run LLMs locally. A clojure wrapper for llama.cpp.

Quick Start

If you're just looking for a model to try things out, try the 3.6Gb llama2 7B chat model from TheBloke. Make sure to check the link for important info like license and use policy.

mkdir -p models
# Download 3.6Gb model to models/ directory
(cd models && curl -L -O 'https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q4_0.bin')
# mvn-llama alias pulls precompiled llama.cpp libs from maven
clojure -M:mvn-llama -m com.phronemophobic.llama "models/llama-2-7b-chat.ggmlv3.q4_0.bin" "what is 2+2?"

Documentation

Getting Started
Intro to Running LLMs Locally
API Reference Docs

Dependency

For llama.clj with required native dependencies:

com.phronemophobic/llama-clj-combined {:mvn/version "0.8-alpha1"}

For llama.clj only (see below for various alternatives for specifying native dependencies):

com.phronemophobic/llama-clj {:mvn/version "0.8-alpha1"}

Native Dependency

llama.clj relies on the excellent llama.cpp library.

The llama.cpp shared library can either be compiled locally or can be included as a standalone maven dependency.

Precompiled native deps on clojars

The easiest method is to include the corresponding native dependency for your platform (including multiple is fine, but will increase the size of your dependencies). See the mvn-llama alias for an example.

;; gguf dependencies
com.phronemophobic.cljonda/llama-cpp-gguf-linux-x86-64 {:mvn/version "c3f197912f1ce858ac114d70c40db512de02e2e0"}
com.phronemophobic.cljonda/llama-cpp-gguf-darwin-aarch64 {:mvn/version "c3f197912f1ce858ac114d70c40db512de02e2e0"}
com.phronemophobic.cljonda/llama-cpp-gguf-darwin-x86-64 {:mvn/version "c3f197912f1ce858ac114d70c40db512de02e2e0"}

;; ggml dependencies
com.phronemophobic.cljonda/llama-cpp-darwin-aarch64 {:mvn/version "6e88a462d7d2d281e33f35c3c41df785ef633bc1"}
com.phronemophobic.cljonda/llama-cpp-darwin-x86-64 {:mvn/version "6e88a462d7d2d281e33f35c3c41df785ef633bc1"}
com.phronemophobic.cljonda/llama-cpp-linux-x86-64 {:mvn/version "6e88a462d7d2d281e33f35c3c41df785ef633bc1"}

Locally compiled

Clone https://github.com/ggerganov/llama.cpp and follow the instructions for building. Make sure to include the shared library options.

Note: The llama.cpp ffi bindings are based on the 4329d1acb01c353803a54733b8eef9d93d0b84b2 git commit for ggml models and 40e07a60f9ce06e79f3ccd4c903eba300fb31b5e for gguf models. Future versions of llama.cpp might not be compatible if breaking changes are made. TODO: include instructions for updating ffi bindings.

Note: Dual wielding ggml and gguf llama.cpp versions is possible, but not currently supported for locally compiled builds. Please file an issue if you need this

For Example:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git checkout 4329d1acb01c353803a54733b8eef9d93d0b84b2
mkdir build
cd build
cmake -DBUILD_SHARED_LIBS=ON ..
cmake --build . --config Release

Next, include an alias that includes the path to the directory where the shared library is located:

;; in aliases
;; add jvm opt for local llama build.
:local-llama {:jvm-opts ["-Djna.library.path=/path/to/llama.cpp/build/"]}

Obtaining models

For more complete information about the models that llama.clj can work with, refer to the llama.cpp readme.

Another good resource for models is TheBloke on huggingface.

Cli Usage

clojure -M -m com.phronemophobic.llama <path-to-model> <prompt>

Example:

mkdir -p models
# Download 3.6Gb model to models/ directory
(cd models && curl -L -O 'https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q4_0.bin')
clojure -M:mvn-llama -m com.phronemophobic.llama "models/llama-2-7b-chat.ggmlv3.q4_0.bin" "what is 2+2?"

cuBLAS support

For gpu support on linux, cuda must be installed. The instructions for cuda installation can be found in nvidia's documentation.

Make sure to restart and follow the post installation instructions so that the cuda development tools like nvcc are available on the path.

Currently, pre-compiled binaries of llama.cpp with cuBLAS support are not available. The llama.cpp native dependencies must be compiled locally with -DLLAMA_CUBLAS=ON as argument. Something like:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git checkout 4329d1acb01c353803a54733b8eef9d93d0b84b2
mkdir build
cd build
cmake -DBUILD_SHARED_LIBS=ON -DLLAMA_CUBLAS=ON ..
cmake --build . --config Release

More cuBLAS Resources

• https://github.com/ggerganov/llama.cpp#cublas
• https://docs.nvidia.com/cuda/cuda-installation-guide-linux/contents.html

"Roadmap"

• Pure clojure implementation for mirostatv2 and other useful samplers.
• Provide reasonable default implementations for generating responses larger than the context size.
• Update llama.cpp to support gguf format
• More docs!
  • Reference docs
  • Intro Guide to LLMs.

License

The MIT License (MIT)

Copyright © 2023 Adrian Smith

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.