malo94 / fully-local-pdf-chatbot

Yes, it's another chat over documents implementation... but this one is entirely local!

🏠 Fully Local Chat Over Documents

Yes, it's another chat over documents implementation... but this one is entirely local!

It can even run fully in your browser with a small LLM via WebLLM!

It's a Next.js app that read the content of an uploaded PDF, chunks it, adds it to a vector store, and performs RAG, all client side. You can even turn off your WiFi after the site loads.

You can see a live version at https://webml-demo.vercel.app.

Users can choose one of the below options to run inference:

Browser-only mode

You can run the entire stack your browser via WebLLM. The model used is the small, 2.7B parameter Phi-2.

You don't have to leave the window to set this up - just upload a PDF and go!

Note that the first time you start a chat, the app will download and cache the model weights. This download is several GB in size and may take a little while, so make sure you have a good internet connection!

Ollama

You can run more powerful, general models outside the browser using Ollama's desktop app. Users will need to download and set up then run the following commands to allow the site access to a locally running Mistral instance:

Mac/Linux

$ OLLAMA_ORIGINS=https://webml-demo.vercel.app OLLAMA_HOST=127.0.0.1:11435 ollama serve

Then, in another terminal window:

$ OLLAMA_HOST=127.0.0.1:11435 ollama pull mistral

Windows

$ set OLLAMA_ORIGINS=https://webml-demo.vercel.app
set OLLAMA_HOST=127.0.0.1:11435
ollama serve

Then, in another terminal window:

$ set OLLAMA_HOST=127.0.0.1:11435
ollama pull mistral

⚡ Stack

It uses the following:

• Voy as the vector store, fully WASM in the browser.
• Ollama or WebLLM to run an LLM locally and expose it to the web app.
• LangChain.js to call the models, perform retrieval, and generally orchestrate all the pieces.
• Transformers.js to run open source Nomic embeddings in the browser.
  • For higher-quality embeddings, switch to "nomic-ai/nomic-embed-text-v1" in app/worker.ts.

While the goal is to run as much of the app as possible directly in the browser, but you can swap in Ollama embeddings in lieu of Transformers.js as well.

🔱 Forking

To run/deploy this yourself, simply fork this repo and install the required dependencies with yarn.

There are no required environment variables, but you can optionally set up LangSmith tracing while developing locally to help debug the prompts and the chain. Copy the .env.example file into a .env.local file:

# No environment variables required!

# LangSmith tracing from the web worker.
# WARNING: FOR DEVELOPMENT ONLY. DO NOT DEPLOY A LIVE VERSION WITH THESE
# VARIABLES SET AS YOU WILL LEAK YOUR LANGCHAIN API KEY.
NEXT_PUBLIC_LANGCHAIN_TRACING_V2="true"
NEXT_PUBLIC_LANGCHAIN_API_KEY=
NEXT_PUBLIC_LANGCHAIN_PROJECT=

Just make sure you don't set this in production, as your LangChain API key will be public on the frontend!

📖 Further reading

For a bit more on this topic, check out my blog post on Ollama or my Google Summit talk on building with LLMs in the browser.

🙏 Thank you!

Special thanks to:

• @dawchihliou for making Voy
• @jmorgan and @mchiang0610 for making Ollama and for your feedback
• @charlie_ruan for your incredible work on WebLLM
• @xenovacom for making Transformers.js
• And @jason_mayes and @nfcampos for inspiration and some great conversations.

For more, follow me on Twitter @Hacubu!