valkyrialenneth / longformer_zh

中文预训练Longformer模型 | Longformer_ZH with PyTorch

相比于Transformer的O(n^2)复杂度，Longformer提供了一种以线性复杂度处理最长4K字符级别文档序列的方法。Longformer Attention包括了标准的自注意力与全局注意力机制，方便模型更好地学习超长序列的信息。

Compared with O(n^2) complexity for Transformer model, Longformer provides an efficient method for processing long-document level sequence in Linear complexity. Longformer’s attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention.

我们注意到关于中文Longformer或超长序列任务的资源较少，因此在此开源了我们预训练的中文Longformer模型参数， 并提供了相应的加载方法，以及预训练脚本。

There are not so much resource for Chinese Longformer or long-sequence-level chinese task. Thus we open source our pretrained longformer model to help the researchers.

加载模型 | Load the model

您可以使用谷歌云盘或百度网盘下载我们的模型
You could get Longformer_zh from Google Drive or Baidu Yun.

• Google Drive: https://drive.google.com/file/d/1IDJ4aVTfSFUQLIqCYBtoRpnfbgHPoxB4/view?usp=sharing
• 百度云: 链接：https://pan.baidu.com/s/1HaVDENx52I7ryPFpnQmq1w 提取码：y601

我们同样提供了Huggingface的自动下载
We also provide auto load with HuggingFace.Transformers.

from Longformer_zh import LongformerZhForMaksedLM
LongformerZhForMaskedLM.from_pretrained('ValkyriaLenneth/longformer_zh')

注意事项 | Notice

• 直接使用 transformers.LongformerModel.from_pretrained 加载模型

• Please use transformers.LongformerModel.from_pretrained to load the model directly

• 以下内容已经被弃用

• The following notices are abondoned, please ignore them.

• 区别于英文原版Longformer， 中文Longformer的基础是Roberta_zh模型，其本质上属于 Transformers.BertModel 而非 RobertaModel, 因此无法使用原版代码直接加载。

• Different with origin English Longformer, Longformer_Zh is based on Roberta_zh which is a subclass of Transformers.BertModel not RobertaModel. Thus it is impossible to load it with origin code.

• 我们提供了修改后的中文Longformer文件，您可以使用其加载参数。

• We provide modified Longformer_zh class, you can use it directly to load the model.

• 如果您想将此参数用于更多任务，请参考Longformer_zh.py替换Attention Layer.

• If you want to use our model on more down-stream tasks, please refer to Longformer_zh.py and replace Attention layer with Longformer Attention layer.

关于预训练 | About Pretraining

• 我们的预训练语料来自 https://github.com/brightmart/nlp_chinese_corpus， 根据Longformer原文的设置，采用了多种语料混合的预训练数据。

• The corpus of pretraining is from https://github.com/brightmart/nlp_chinese_corpus. Based on the paper of Longformer, we use a mixture of 4 different chinese corpus for pretraining.

• 我们的模型是基于Roberta_zh_mid (https://github.com/brightmart/roberta_zh),训练脚本参考了https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb

• The basement of our model is Roberta_zh_mid (https://github.com/brightmart/roberta_zh). Pretraining scripts is modified from https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb.

• 同时我们在原版基础上，引入了 Whole-Word-Masking 机制，以便更好地适应中文特性。

• We introduce Whole-Word-Masking method into pretraining for better fitting Chinese language.

• Whole-Word-Masking代码改写自TensorFlow版本的Roberta_zh，据我们所知是第一个开源的Pytorch版本WWM.

• Our WWM scripts is refacted from Roberta_zh_Tensorflow, as far as we know, it is the first open source Whole-word-masking scripts in Pytorch.

• 模型 max_seq_length = 4096, 在 4 * Titan RTX 上预训练3K steps 大概用时4天。

• Max seuence length is 4096 and the pretraining took 4 days on 4 * Titan RTX.

• 我们使用了 Nvidia.Apex 引入了混合精度训练，以加速预训练。

• We use Nvidia.Apex to accelerate pretraining.

• 关于数据预处理， 我们采用 Jieba 分词与JIONLP进行数据清洗。

• We use Jieba Chinese tokenizer and JIONLP data cleaning.

• 更多细节可以参考我们的预训练脚本

• For more details, please check our pretraining scripts.

效果测试 | Evaluation

CCF Sentiment Analysis

• 由于中文超长文本级别任务稀缺，我们采用了CCF-Sentiment-Analysis任务进行测试
• Since it is hard to acquire open-sourced long sequence level chinese NLP task, we use CCF-Sentiment-Analysis for evaluation.

Model	Dev F
Bert	80.3
Bert-wwm-ext	80.5
Roberta-mid	80.5
Roberta-large	81.25
Longformer_SC	79.37
Longformer_ZH	80.51

Pretraining BPC

• 我们提供了预训练BPC(bits-per-character), BPC越小，代表语言模型性能更优。可视作PPL.
• We also provide BPC scores of pretraining, the lower BPC score, the better performance Langugage Model has. You can also treat it as PPL.

Model	BPC
Longformer before training	14.78
Longformer after training	3.10

CMRC(Chinese Machine Reading Comprehension)

Model	F1	EM
Bert	85.87	64.90
Roberta	86.45	66.57
Longformer_zh	86.15	66.84

Chinese Coreference Resolution

Model	Conll-F1	Precision	Recall
Bert	66.82	70.30	63.67
Roberta	67.77	69.28	66.32
Longformer_zh	67.81	70.13	65.64

致谢

感谢东京工业大学 奥村·船越研究室 提供算力。

Thanks Okumula·Funakoshi Lab from Tokyo Institute of Technology who provides the devices and oppotunity for me to finish this project.

longformer_zh's Issues

Sequence length should be multiple of 512. It can't directly used for encoding

File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 1068, in forward
return_dict=return_dict,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 591, in forward
output_attentions,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 476, in forward
past_key_value=self_attn_past_key_value,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 408, in forward
output_attentions,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "I:\PycharmProject\zh_efficient-autogressive-EL\model\Longformer_zh.py", line 21, in forward
output_attentions=output_attentions)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\longformer\modeling_longformer.py", line 591, in forward
query_vectors, key_vectors, self.one_sided_attn_window_size
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\longformer\modeling_longformer.py", line 803, in _sliding_chunks_query_key_matmul
), f"Sequence length should be multiple of {window_overlap * 2}. Given {seq_len}"
AssertionError: Sequence length should be multiple of 512. Given 158

did you miss something that pad the sequence to suitbal length?

spelling mistake?

我们同样提供了Huggingface的自动下载
We also provide auto load with HuggingFace.Transformers.
from Longformer_zh import LongformerZhForMaksedLM
LongformerZhForMaksedLM.from_pretrained('ValkyriaLenneth/longformer_zh')

ImportError: cannot import name 'LongformerZhForMaksedLM' from 'Longformer_zh'

LongformerZhForMaksedLM or LongformerZhForMaskedLM ?

文本匹配任务使用longformer能使用交互型模型吗

使用交互型模型感觉会长出长度限制导致后一文本被切断

如直接使用LongformerTokenizer会报此错，是否需要使用BertTokenizer？

I'm not sure if it's able to directly ask you questions in Chinese. If it caused misinterpretations, I can change to English.

您好！我现在正在使用您的预训练模型，文件下载自 https://huggingface.co/ValkyriaLenneth/longformer_zh 。我直接使用AutoTokenizer的话，代码会自动调用LongformerTokenizer，然后会报如下错误：

Traceback (most recent call last):
  File "mypath/trylongformerzh1.py", line 3, in <module>
    tokenizer = LongformerTokenizer.from_pretrained("pretrain_path/longformer_zh")
  File "virtualenv_path/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1744, in from_pretrained
    return cls._from_pretrained(
  File "virtualenv_path/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1872, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "virtualenv_path/lib/python3.8/site-packages/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "virtualenv_path/lib/python3.8/site-packages/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType

我看到您的代码中使用的是BertTokenizerFast，所以请问加载longformer_zh的tokenizer是否也需要使用BertTokenizer？
我直接用BertTokenizer确实是可以运行的。

另：我是使用transformers.LongformerModel.from_pretrained来加载您的模型。我暂时没有测试其他功能，直接加载模型似乎是可行的。

我的transformers版本是4.12.5，我能够运行成功的代码是：

from transformers import BertTokenizer, LongformerModel

tokenizer = BertTokenizer.from_pretrained("pretrain_path/longformer_zh")

model = LongformerModel.from_pretrained("pretrain_path/longformer_zh")

如果您有时间浏览本issue的话，我会非常感谢！

请问原生longformer的Dilated sliding window这些能够使用吗

有注意到hugging face的longformerconfig里面好像没有加载这些参数的地方

请问有没有使用的例子？