hahnvanessa / odqa

Open Domain Question Answering System with Joint Training

Open Domain Question Answering System with Joint Training using Reinforcement Learning

Reimplementation of https://www.aclweb.org/anthology/P18-1159/ for the course Advances in Question Answering, WS 19/20, Saarland University by Ekaterina Saveleva, Fabian Goessl and Vanessa Hahn

Installation Requirements

pip install -r requirements.txt

Data preprocessing

Data conversion

Use a variation of this command for every file to convert and save it as a pickled dictionary object:

python3 preprocessing.py -t "searchqa" -f /local/user/output/searchQA  -s "test"

Encoding

For encoding please add a glove file and all converted files in output, then run:

python3 embeddings.py --out /local/user/output 

Preprocessing and Dataset objects

To do the final preprocessing of all the encoded files call:

python dict_to_qas.py --out /local/user/output 

Usage

Collect training and test files in two separate folders. Example of usage:

python run.py --lr 0.0001 --num_epochs 2 --emb /local/user/embedding_matrix.pkl --id2v /local/user/idx_2_word_dict.pkl --input_train /local/user/train_files_folder/ --input_test /local/user/test_files_folder/

Data

SearchQA: https://github.com/nyu-dl/dl4ir-searchqA

Quasar-T: https://github.com/bdhingra/quasar