allenai / dqa-net

Diagram question answering system described in "A Diagram is Worth a Dozen Images"

DQA-Net

This is the original source code for DQA-Net described in ["A Diagram is Worth a Dozen Images"] (http://arxiv.org/abs/1603.07396)

Quick Start

1. Install Requirements

• Python (verified on 3.5.1)
• Python packages: numpy, progressbar2, nltk, tensorflow, h5py, qa2hypo
• Torch (comes with Lua)
• Lua packages: cunn, cudnn, cutorch, loadcaffe, hdf5

Note that the most recent official release of tensorflow (0.7.1) probably won't be compatible with this. You will need to build from a recent commit (verified on e39d8fe). DQA-Net does not use images (VQA baseline does), so you can skip Lua/Torch if you just want to run DQA-Net. See details below.

2. Download Data and Models

At the root folder, run:

chmod +x download.sh
./download.sh

to download DQA data, folds, Glove vectors and VGG-19 model. VGG-19 model is used for images, and as mentioned above, DQA-Net does not use images, so you can comment this line out if you only run DQA-Net.

3. Preprocess Data

Run:

python -m prepro.p05

to preprocess data. You can just use default directories unless you make changes to download directories in download.sh.

If you wish to skip image preprocessing (in case you only run DQA-Net), Run with an additional flag:

python -m prepro.p05 --prepro_images False

Now you will see all preprocessed json and h5 files in data/s3 folder inside the source code's root folder.

4. Train and Test

To train the default model, run:

python -m main.x05 --train

To test the trained model on test data, run:

python -m main.x05

To launch tensorboard, run:

tensorboard --logdir logs/m05/None --port 6006

Here, m05 is the model name, and None is the default configuration. All tensorboard logs will be stored in logs/ folder.

To visualize the attention, run:

python -m vis.list_results 5 None train 1 --port 8000 --open True

Here, 5 is the model number (m05), None is the default configuration, train indicates data type, and 1 is the epoch from which the result will be fetches. See evals/m05/None folder to see possible epochs (result saving frequency can be controlled by "save_period" flag at main/m05.py). After running the script, the script hosts html server at the specified port. --open True flag opens web browser at this address.

In general, use -h flag for the run files to see what kind options there are.

dqa-net's Issues

AttributeError: 'module' object has no attribute 'ServerProxy'

mldl@mldlUB1604:/ub16_prj/dqa-net$ python -m prepro.p05
dumping json files ...
done
N/A%| |ETA: --:--:--Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/mldl/ub16_prj/dqa-net/prepro/p05.py", line 493, in
prepro_questions(ARGS)
File "/home/mldl/ub16_prj/dqa-net/prepro/p05.py", line 323, in prepro_questions
sents = [_tokenize(qa2hypo(ques_text, choice, args.qa2hypo, args.qa2hypo_path)) for choice in d['answerTexts']]
File "/home/mldl/ub16_prj/dqa-net/prepro/p05.py", line 37, in qa2hypo
from qa2hypo import qa2hypo as f
File "qa2hypo/qa2hypo.py", line 7, in
from helper import *
File "qa2hypo/helper.py", line 1, in
from parser import *
File "qa2hypo/parser.py", line 10, in
server = jsonrpc.ServerProxy(jsonrpc.JsonRpc20(), jsonrpc.TransportTcpIp(addr=("127.0.0.1", 8080)))
AttributeError: 'module' object has no attribute 'ServerProxy'
mldl@mldlUB1604:/ub16_prj/dqa-net$

Producing DPG from a new diagram

Hi,

I would like to generate diagram parsing graph (DPG) from pathway diagrams that I extracted from biomedical literature. Can I use this implementation to do that?

I checked the original paper, but could not reproduce the preprocessing steps such as finding blobs and arrows or potential relations (i.e. section 7.1 and 7.2 ) for DSDP-Net.

Best Regards,
Satoshi

would you specify exactly which tensorflow used in this prj?