jasonjmcghee / portable-hnsw

What if an HNSW index was just a file, and you could serve it from a CDN, and search it directly in the browser?

Portable HNSW

To build your own index:

poetry install
poetry run python build_index.py <path to text file> [output folder]

Or you can jump into the code and do more complex use cases.

Then throw it in a GitHub repo and enable GitHub Pages. You can add / edit the index.html or test it by pasting the link to the folder in the "path" input here

Note: rangehttpserver works well as a simple server to support range requests for locally testing duckdb parquet + large indices.

---

So what's going on here?

Yeah - fair question.

So I had this idea.

What if an HNSW index (hierarchical navigable small world graphs - a good way to enable searching for stuff by their underlying meaning) was just a file, and you could serve it from a CDN, and search it directly in the browser?

And what if you didn't need to load the entire thing in memory, so you could search a massive index without being RAM rich?

That would be cool.

A vector store without a server...

So yeah. Here's a proof of concept.

---

There's a Python file called build_index.py that builds an index using a custom hnsw algorithm that can be serialized to a couple of parquet files.

There are very likely bugs and performance problems. But it's within an order of magnitude or two of hnswlib which was fast enough that my development cycle wasn't impacted by repeatedly re-indexing the same files while building the search and front-end bits. I welcome pull requests to fix the problems and make it halfway reasonable.

Then I wrote a webpage that uses transformers.js, duckdb and some SQL to read the parquet files and search it (similar to HNSW approx nearest neighbor search) and then retrieve the associated text.

A big part of the original idea was how this could scale to massive indices.

So, I also tested using parquet range requests and only retrieving what we need from the parquet file, which worked! But since the index is only like 100MB, and each range request added overhead, loading it all into memory was about twice as fast. But, it means you could have a 1TB index and it would (theoretically) still work, which is pretty crazy.

You can try this yourself by swapping out the nodes.parquet bits in the SQL for read_parquet('${path}/nodes.parquet'). And the same with edges. DuckDB takes care of the rest.

---

Anyway, would love feedback and welcome contributions.

It was a fun project!

portable-hnsw's Issues

You should be able to update existing indexes

Currently you have to rebuild indexes any time you want to update them. That's too bad!

A naive solution shouldn't be too difficult...

• add loading of indices (read the parquet files, set the fields, initialize the data)

• when you want to add elements, adjust the max_elements, reinitialize the data matrix to be the new size, copying over existing elements, and add the new items.

Product Quantization

Turns out there are some tricks to wildly reduce the size of the indices.

Product Quantization should be explored.

https://www.pinecone.io/learn/series/faiss/product-quantization/

Should be no problem to do it at index creation time and on front end before search.

Could improve parquet file size, speed, download size, and ram usage.

WASM

We should consider converting the core front end logic to be in webassembly to improve search time.

The distance function and going from duckdb -> js are expensive.

Both of these could likely benefit by using wasm instead.

Note: executing the distance function in duckdb is another potential approach.