dengbocong / prompt-tuning

A pipeline for Prompt-tuning

Prompt-Tuning

• A pipeline for Prompt-tuning
• 集成主流的Prompt-tuning相关方法，以及search template策略
• 提供Prompt-tuning完整的执行pipeline

Requirements

本项目相关的依赖包参考requirements.txt，也可直接使用如下指令安装：

pip install -r requirements.txt

Usage

• core下放置相关prompt-tuning模型
• core/gen_template下是相关template生成方法，执行入口参考run_gen_template.py，执行示例如下：

python3 run_gen_template.py \
    --task_name CoLA \
    --k 16 \
    --dev_rate 1 \
    --data_loader glue \
    --template_generator lm_bff \
    --data_dir data/original/CoLA \
    --output_dir data/output \
    --generator_config_path data/config/lm_bff.json

• 模型实现放在core目录下，执行入口参考run_prompt_tuning.py，执行示例如下：

python3 run_prompt_tuning.py \
    --data_dir data/CoLA/ \
    --do_train \
    --do_eval \
    --do_predict \
    --model_name_or_path bert \
    --num_k 16 \
    --max_steps 1000 \
    --eval_steps 100 \
    --learning_rate 1e-5 \
    --output_dir result/ \
    --seed 16
    --template "*cls**sent_0*_It_was*mask*.*sep+*" \
    --mapping "{'0':'terrible','1':'great'}" \
    --num_sample 16 \

• data放置相关config及datasets，由于数据集比较庞大，可使用scripts下的下载脚本自行下载，如下：

cd data
sh download_clue_dataset.sh
sh download_glue_dataset.sh

• tools放置相关工具方法及数据集处理方法等

Paper

更详细的论文解读和阅读笔记 ☞ 点这里

• Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Making Pre-trained Language Models Better Few-shot Learners
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• GPT Understands, Too
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Noisy Channel Language Model Prompting for Few-Shot Text Classification
• PPT: Pre-trained Prompt Tuning for Few-shot Learning
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer

Reference

• https://github.com/princeton-nlp/LM-BFF
• https://github.com/shmsw25/Channel-LM-Prompting

Dataset

• GLUE：https://nlp.cs.princeton.edu/projects/lm-bff/datasets.tar

sh scripts/download_glue_dataset.sh

• CLUE：https://github.com/CLUEbenchmark/CLUE

sh scripts/download_clue_dataset.sh

prompt-tuning's Issues

运行 run_prompt_tuning 报错

run_gen_template.py has no error when running, but later run_prompt_tuning.py occurs errors(1、2) as follow:

I'm very grateful if you have any advice!

forward() got an unexpected keyword argument 'mask_pos'

已经调试到最后一步了，请作者一定帮忙看看

加载的模型用的 bert-base-uncased，不知道问题出在哪了

lm_bff.json文件中相互关联的参数："inspired_templates", "replace_token_map_list"如何设置？