lancopku / amm

The code for "An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation" （EMNLP 2018)

Auto-Encoder Matching Model

The code for "An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation"

Requirements

• Python 3
• Tensorflow >= 1.8
• mlbootstrap == 0.02

Data Preparation

• Get the DailyDialog dataset at http://yanran.li/dailydialog.html
• Unzip the downloaded file
• Move dialogues_text.txt to data/source/daily/dialogues_text.txt

To use your own data, create a folder data/source/<dataset-name>/ and place the original data in the directory. Then write a parsing script (you can refer to daily.py) and update the config.yaml to include the new data path.

Training

python play.py

Evaluation

Change the last line in play.py to bootstrap.evaluate() and run python play.py

Hyperparameters

You can change the hyperparameters in config.yaml according to your needs.

amm's Issues

How can I get the original dataset?

the link

http://yanran.li/dailydialog.html

is invalid.

How can I get responses for test_samples.txt?

I placed test_samples.txt in ./data/processed/daily/ which contains several sentences then run play.py in evaluation mode, but got no response in ./save folder. I followed steps in this issue. How to generate response? Thank you for help!

Could I apply for your pretrained model on DailyDialog?

Thanks in advance!

Why Add beamsearch when decoding, the effect is very poor?

FileNotFoundError: [Errno 2] No such file or directory: 'data/processed/daily/test_samples.txt'

When I execute "python play.py",got FileNotFoundError:

Start processing data for task 'daily'.
Saved full dataset.
Flattening QA pairs in training set: 67%|▋| 48440/71890 [00:00<00:00, 484395.84Flattening QA pairs in training set: 88%|▉| 62972/71890 [00:00<00:00, 207005.67 Flattening QA pairs in testing set: 8%| | 1424/17972 [00:00<00:01, 11103.53it/ Completed processing.
Traceback (most recent call last):
File "play.py", line 22, in
bootstrap.train()
File "/home/zys/anaconda3/envs/AMM/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 48, in train
self._init_model()
File "/home/zys/anaconda3/envs/AMM/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 63, in _init_model
self.preprocess()
File "/home/zys/anaconda3/envs/AMM/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 44, in preprocess
self.__preprocessor.load_processed()
File "/home/zys/anaconda3/envs/AMM/lib/python3.6/site-packages/mlbootstrap/preprocess.py", line 26, in load_processed
self.dataset = self._load_dataset()
File "/home/zys/文档/chatbot/AMM-master/process/process.py", line 177, in _load_dataset
with open(os.path.join(dst, 'test_samples.txt'), 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'data/processed/daily/test_samples.txt'

There is nothing about "test_samples.txt" in your code or datasets.Could you explain this for me?Thank you very much!

The L2 Norm Loss

It seems that there is a L2_norm loss term(J3) in the total loss according to you paper "An Auto-Encoder Matching Model for Learning Utterance-Level Semantic Dependency in Dialogue Generation". But I didn't find it in the code. Is it because that ignoring it does not hurt the performance?

AttributeError: 'Bootstrap' object has no attribute '_Bootstrap__model'

When I execute "python play.py",got FileNotFoundError:
File "E:/AMM-master/play.py", line 22, in
bootstrap.train()
File "F:\Anaconda3\lib\site-packages\mlbootstrap\bootstrap.py", line 57, in train
self._init_model(**kwargs)
File "F:\Anaconda3\lib\site-packages\mlbootstrap\bootstrap.py", line 92, in _init_model
Loaded dataset: 12162 words, 58533 training samples, 11071 testing samples
self.__model.set_config(self.config)
AttributeError: 'Bootstrap' object has no attribute '_Bootstrap__model'
can you help me to solve the problem?
thank you very much!

NameError: name 'task' is not defined

when I ran the play.py, I attain this error:
/home/yangjy/anaconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Traceback (most recent call last):
File "/home/yangjy/PycharmProjects/AMM-master/play.py", line 22, in
bootstrap.train()
File "/home/yangjy/anaconda3/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 57, in train
self._init_model(**kwargs)
File "/home/yangjy/anaconda3/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 83, in _init_model
self.preprocess(**kwargs)
File "/home/yangjy/anaconda3/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 45, in preprocess
self.fetch(**kwargs)
File "/home/yangjy/anaconda3/lib/python3.6/site-packages/mlbootstrap/bootstrap.py", line 41, in fetch
self._fetcher.fetch()
File "/home/yangjy/PycharmProjects/AMM-master/fetch/fetch.py", line 13, in fetch
fetch_fntask
NameError: name 'task' is not defined