This repository contains the API server, neural models, and UI client for Covidex, a neural search engine for the COVID-19 Open Research Dataset (CORD-19).
For a description of our system, check out this paper: Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset.
Install CUDA 10.1
- For Ubuntu, follow these instructions
- For Debian run
sudo apt-get install nvidia-cuda-toolkit
Install Anaconda (currently version 2020.02)
wget https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh
bash Anaconda3-2020.02-Linux-x86_64.sh
Install Java 11
sudo apt-get install openjdk-11-jre openjdk-11-jdk
Build the latest Anserini indices
sh scripts/update-anserini-index.sh
Build the latest HNSW index for related article search
sh scripts/update-hnsw-index.sh
Set up environment variables by copying over the defaults and modifying as needed
cp api/.env.sample api/.env
Create an Anaconda environment for Python 3.7
conda create -n covidex python=3.7
Activate the Anaconda environment
conda activate covidex
Install Python dependencies
pip install -r api/requirements.txt
Run the server (make sure you are in the api/
folder)
uvicorn app.main:app --reload --port=8000
The server wil be running at localhost:8000 with API documentation at /docs
Install Node.js 12+ and Yarn.
Install dependencies
yarn install
Start the server
yarn start
The client will be running at localhost:3000
Redirect port 80 to specified port since only root can bind to port 80 (the below command uses port 8000):
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8000
Build the latest Anserini indices
sh scripts/update-anserini-index.sh [DATE]
Build the latest HNSW index for related article search
sh scripts/update-hnsw-index.sh
Start the server (deploys to port 8000 by default):
sh scripts/deploy-prod.sh
Optional: set the environment variable $PORT
:
PORT=8000 sh scripts/deploy-prod.sh
Log files are available under api/logs
, where new files are created daily based on UTC time. All filenames have the date appended except for the current one, which will be named search.log
or related.log
.
To run all API tests
TESTING=true pytest api
@article{zhang2020covidex,
title={Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset},
author={Zhang, Edwin and Gupta, Nikhil and Tang, Raphael and Han, Xiao and Pradeep, Ronak and Lu, Kuang and Zhang, Yue and Nogueira, Rodrigo and Cho, Kyunghyun and Fang, Hui and others},
journal={arXiv preprint arXiv:2007.07846},
year={2020}
}