gdreamcatcher / vearch

A distributed system for embedding-based retrieval

   

Overview

Vearch is a scalable distributed system for efficient similarity search of deep learning vectors.

Quick start

• Quickly compile the source codes to build a distributed vector search system with RESTful API, please see docs/SourceCompileDeployment.md.

• Vearch can be leveraged to build a complete visual search system to index billions of images. The image retrieval plugin for object detection and feature extraction is also required. For more information, please refer to docs/Quickstart.md.

• Vearch Python SDK enables vearch to use locally. Vearch python sdk can be installed easily by pip install vearch. For more information, please refer to docs/APIPythonSDK.md.

• One-click binary installation, please see docs/BinaryInstallation.md(中文版).

APIs and Use Cases

LowLevelAPI

• docs/APILowLevel.md
• For GPU docs/APILowLevel.md

VisualSearchAPI

• docs/APIVisualSearch.md

PythonSDKAPI

• docs/APIPythonSDK.md

Document

• https://vearch.readthedocs.io/en/latest
• https://vearch.readthedocs.io/zh_CN/latest

Components

Vearch Architecture

Master

Responsible for schema mananagement, cluster-level metadata, and resource coordination.

Router

Provides RESTful API: `create` , `delete` `search` and `update` ; request routing, and result merging.

PartitionServer (PS)

Hosts document partitions with raft-based replication.

Gamma is the core vector search engine implemented based on faiss. It provides the ability of storing, indexing and retrieving the vectors and scalars.

Benchmarks

• benchmarks

Reference

Reference to cite when you use Vearch in a research paper:

@misc{li2019design,
      title={The Design and Implementation of a Real Time Visual Search System on JD E-commerce Platform}, 
      author={Jie Li and Haifeng Liu and Chuanghua Gui and Jianyu Chen and Zhenyun Ni and Ning Wang},
      year={2019},
      eprint={1908.07389},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}

Community

You can report bugs or ask questions in the issues page of the repository.

For public discussion of Vearch or for questions, you can also send email to [email protected].

Our slack : https://vearchwrokspace.slack.com

Known Users

Welcome to register the company name in this issue: vearch#230 (in order of registration)

欢迎在此 issue vearch#230 中登记公司名称

License

Licensed under the Apache License, Version 2.0. For detail see LICENSE and NOTICE.