zhaoxlpku / knowledgpt

KnowledGPT

This is an implementation of our paper

• Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao and Rui Yan. Knowledge-Grounded Dialogue Generation with Pre-trained Language Models. In EMNLP, 2020 (Main Conference)

Requirements

• Python 3.6
• Pytorch 1.2.0
• CUDA 10.0 supported GPU with at least 12GB memory
• see requirements.txt for more details

Usage

To run our pretrained model on Wizard of Wikipedia:

• Download the data from ParlAI, then preprocess the data

python preprocess.py --in_file /path/to/wizard_of_wikipedia/test_random_split.json --out_file wizard_of_wikipedia/data/test_seen.jsonl
python preprocess.py --in_file /path/to/wizard_of_wikipedia/test_topic_split.json --out_file wizard_of_wikipedia/data/test_unseen.jsonl

• Prepare the BERT/GPT-2 files

python bert_config.py --out_file pretrain-models/bert_base_uncased
python gpt2_config.py --out_file pretrain-models/gpt2

• Download the checkpoint from here, then save to "wizard_of_wikipedia/checkpoints"

• To evaluate the model, run

python evaluate.py --eval_batch_size 2 --gpu_list 0 --exp_name test

Citation

We appreciate your citation if you find our code is beneficial.

@inproceedings{zhao2020knowledgpt,
  title={Knowledge-Grounded Dialogue Generation with Pre-trained Language Models},
  author={Xueliang Zhao, Wei Wu, Can Xu, Chongyang Tao, Dongyan Zhao, Rui Yan},
  booktitle = {EMNLP},
  year = {2020}
}

knowledgpt's Issues

AssertionError in File "evaluate.py", line 170, in main

Terminal command run: python evaluate.py --eval_batch_size 2 --gpu_list 1 --exp_name test

GPU_RATIO=0.85
KNOWLEDGE_TRUNCATE=64
LENGTH_PENALTY=1.0
LOAD_DIS=True
LOAD_GEN=True
LOG=wizard_of_wikipedia/log
LSTM_HIDDEN=256
LSTM_LAYER=1
MAX_LENGTH=30
MIN_LENGTH=15
N_SENT=1
NO_CUDA=False
NO_REPEAT_NGRAM_SIZE=0
REPETITION_PENALTY=1.0
SEED=42
SEGMENT=True
TEST_SEEN_FILE=wizard_of_wikipedia/data/test_seen.jsonl
TEST_UNSEEN_FILE=wizard_of_wikipedia/data/test_unseen.jsonl
TEXT_TRUNCATE=64

Writing to wizard_of_wikipedia/log/test

Create training dataset begain... | 2021-05-23 13:58:07 
Create training dataset end... | 2021-05-23 13:58:07 
Traceback (most recent call last):
  File "evaluate.py", line 312, in <module>
    main(args)
  File "evaluate.py", line 170, in main
    gen_batcher = GenBatcher(args.knowledge_truncate, args.text_truncate, args.gpt2_truncate, args.gpt2_config, args.cuda)
  File "/home1/deeksha/aditya/KnowledGPT/utils.py", line 103, in __init__
    self.tokenizer = GPT2Tokenizer.from_pretrained(gpt2_config)
  File "/home1/deeksha/anaconda3/envs/aditya/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 394, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home1/deeksha/anaconda3/envs/aditya/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 545, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home1/deeksha/anaconda3/envs/aditya/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 149, in __init__
    super().__init__(bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, **kwargs)
  File "/home1/deeksha/anaconda3/envs/aditya/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 338, in __init__
    assert isinstance(value, str)
AssertionError

Line 337 block in .../lib/python3.6/site-packages/transformers/tokenization_utils.py

for key, value in kwargs.items():
      if key in self.SPECIAL_TOKENS_ATTRIBUTES:
          # import pdb; pdb.set_trace()
          if key == "additional_special_tokens":
              assert isinstance(value, (list, tuple)) and all(isinstance(t, str) for t in value)
          else:
              assert isinstance(value, str)
          setattr(self, key, value)

请问能提供一下训练的代码吗

能提供一下训练的脚本吗

目前代码里好像没有提供训练的脚本，不知道能否提供一下

Training Scripts

Hi,
Thanks for your nice work. Could you please provide the training scripts? Its not fully clear to me from the paper about how the training is conducted. I was looking for the training script as I want to test it on a new data

方便提供CMU_DoG上的训练checkpoint吗

看了论文之后，感觉用RL+无监督的方式去做选择与生成的联合很有参考意义，想看下模型在Tmax > 1训练下的效果。看到论文中CMU_DoG的训练用过Tmax = 2的，请问方便提供在这个数据集上训练的checkpoint和对应的infer脚本吗？感谢~~

模型里的batch_decode函数里的self.generate 是在哪里实现的呢，继承的GPT2Pretrained也没有啊

4年前的项目了，NB啊

Very good work, and even better if you can provide the training scripts of the model!

请问能提供一下训练的代码吗？

Could you please provide the training scripts of the model?

it seems that the checkpoints miss some parameters.

Missing key(s) in state_dict: "_transformer.embeddings.position_ids".