shubhamagarwal92 / mmd

This repository contains the Pytorch implementation for our SCAI (EMNLP-2018) submission "A Knowledge-Grounded Multimodal Search-Based Conversational Agent"

Hello world! 💻

I am Shubham, rather his Github page 🤖 😎. I don't know how you landed here ¯\_(ツ)_/¯ 🧐 but nice to e-meet you 🤓.

I am passionate about cool research 🔥 💡 🔬. To know more about my recent work, please head to my homepage 🔎 💼.

If you would like to discuss things, feel free to reach out. You can also follow me on Twitter to see the things I am interested in. We can go out for a ☕ / 🍵 .

What do I need?

           

mmd's Issues

Availability of Source Code

When will you publish source code?

Thanks in advance.

model.eval() in train.py

In train.py method evaluate, I find you use model.eval() to disable dropout. However, after evaluate, I didn't find model.train() to enable dropout. It seems that once evaluate called, dropout will be disabled. Does this influence final performance?

Training Time

Thank you for sharing the code.
Can you tell me how much time training takes?

dimensions mismatch error in translate.py

@githwd2016 @shubhamagarwal92
I have trained the model and save the checkpoint. whereas I am trying to run again bash train_and _translate.sh with IS TRAIN to 'No', which means for translation, I got following error: can anybody help me to rectify this:

File "translate.py", line 212, in
main(args)
File "translate.py", line 139, in main
model.load_state_dict(torch.load(args.checkpoint_path))
File "/home/mauajamapcs16/anaconda2/envs/mmd/lib/python2.7/site-packages/torch/nn/modules/module.py", line 721, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for MultimodalHRED:
While copying the parameter named "context_encoder.contextRNN.weight_ih_l0", whose dimensions in the model are torch.Size([1536, 768]) and whose dimensions in the checkpoint are torch.Size([1536, 1024]).

Thanks in advance ..

some issues in data processing

There are some issues in data processing:

• I downloaded dataset and extracted it into the data folder ($PROJECT_DIR/data). For detail, the data folder is like below:

mmd/data
├── dataset
   ├── v1
   │   ├── test
   │   ├── train
   │   └── valid
   └── v2
       ├── test
       ├── train
       └── valid

But when I run dialogue_data.sh, it didn't work. I looked at your code. I think we should fix at line 31.
export DATA_DIR=$PARENT_DIR/data/ --> export DATA_DIR=$PARENT_DIR/data/dataset/$DATA_VERSION

• Besides, in file train_and_translate.sh, at line 54, the default config is _v4, but in your code, you wrote _v5.

Anw, thank you so much for pytorch version of MMD.

annoy.ann file

What exactly does the annoy.ann file hold since it's almost 60GB on unzipping? Is it images or some text?

Global name HRED is not defined in translate.py

Hi,
@shubhamagarwal92 @githwd2016
Can you help me for solving this issues, i have checked the line of error and model_type for opt.model_type, it takes all the parameters but after complete training, the following error raise:

Traceback (most recent call last):
File "translate.py", line 207, in
main(args)
File "translate.py", line 105, in main
model = HRED(src_vocab_size=vocab_size,
NameError: global name 'HRED' is not defined

Thanks for help in advance

import error

Sir I have installed zomatopy using the command :
pip install zomatopy
and after that when I try executing the following commands:
import zomatopy

config={
"user_key":"ZOMATO_API_KEY"
}

zomato = zomatopy.initialize_app(config)
it gives me import error
Traceback (most recent call last):
File "zomato.py", line 1, in
import zomatopy
ImportError: No module named zomatopy

HRED in translate.py

File "translate.py", line 105, in main
model = HRED(src_vocab_size=vocab_size,.....
NameError: global name 'HRED' is not defined

Did anyone else face the HRED issue?

MAX_LEN in dialogue_data.sh

What exactly is the variable 'MAX_LEN'?

Some questions about data processing

In dialogue_data.py, I find this sentence: "if 'search_criteria' in utterance['utterance']:"(line 161).
However, in the data provided by Amrita Saha, utterance['utterance'] doesn't contain the key 'search_criteria' or 'compulsory_fields'. Moreover, I have run the script and find there is nothing in "train_search_criteria.txt".

Thank you in advance.

Guidance for changing image feature extractor

I want to change image feature extractor from VGG-16 to Resnet 50 or something.

Thanks in advance.

What is DATA_STATE_DIR ?

in file train_and_translate.sh
line 23: read -p "Please enter state data directory. Blank for all: " DATA_STATE_DIR
what is DATA_STATE_DIR and what input should I give in cmd line?

Compile errors

Dear @shubhamagarwal92,

There are some compile errors:

• You didn't declare param args for function evaluate at line 64 but called it at line 106.
• At line 87, Bridge class only has 2 params hidden size of encoder and decoder, but you declared 3 arguments: hidden size of encoder, decoder and activation function.
• Besides, I got some troubles with 2 KB Encoders. I tried to run train_and_translate_shah_et_al.sh. It means that I didn't use kb encoders, but they were initialized. However, param vocab_size was assigned None value.

I think you should re-run your code with all configurations to guarantee that your code works well.

Thank you so much for your supporting.

No vocab.pkl file exits

Traceback (most recent call last):
File "train.py", line 327, in
main(args)
File "train.py", line 154, in main
vocab = pkl.load(open(args.vocab_path,'rb'))[1]
IOError: [Errno 2] No such file or directory: '/home1/mauzama/Arunav/mmd_courteous_final/data/v2/dialogue_data//context_2__20//vocab.pkl'

v4 VS v8

Can you tell me what is a performance difference between config version 4 and config version 8 ?

Thanks in advance

annoy.ann file is missing from the directory

I have downloaded the DATA Link, while training the model.

I have checked the directory, the directory contains only FileNameMapToIndex.pkl which is used as annoy_pkl_path. the file annoy.ann file is missing for which I got following error:

File "train.py", line 157, in main
annoyIndex.load(args.annoy_file_path)
IOError: [Errno 2] No such file or directory

I even checked the train_and_translate.sh in which file path is correct but there is no annoy.ann\

ANNOY_PATH = ${PWD}/data
ANNOY_FILE_PATH = $ANNOY_PATH/annoy.ann

Thanks for help in advance