valkyrialenneth / longformer_zh Goto Github PK

我们同样提供了Huggingface的自动下载
We also provide auto load with HuggingFace.Transformers.
from Longformer_zh import LongformerZhForMaksedLM
LongformerZhForMaksedLM.from_pretrained('ValkyriaLenneth/longformer_zh')

ImportError: cannot import name 'LongformerZhForMaksedLM' from 'Longformer_zh'

LongformerZhForMaksedLM or LongformerZhForMaskedLM ?

使用交互型模型感觉会长出长度限制导致后一文本被切断

File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 1068, in forward
return_dict=return_dict,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 591, in forward
output_attentions,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 476, in forward
past_key_value=self_attn_past_key_value,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\bert\modeling_bert.py", line 408, in forward
output_attentions,
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "I:\PycharmProject\zh_efficient-autogressive-EL\model\Longformer_zh.py", line 21, in forward
output_attentions=output_attentions)
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\longformer\modeling_longformer.py", line 591, in forward
query_vectors, key_vectors, self.one_sided_attn_window_size
File "D:\Anaconda\envs\torch_1.7\lib\site-packages\transformers\models\longformer\modeling_longformer.py", line 803, in _sliding_chunks_query_key_matmul
), f"Sequence length should be multiple of {window_overlap * 2}. Given {seq_len}"
AssertionError: Sequence length should be multiple of 512. Given 158

did you miss something that pad the sequence to suitbal length?

有注意到hugging face的longformerconfig里面好像没有加载这些参数的地方

I'm not sure if it's able to directly ask you questions in Chinese. If it caused misinterpretations, I can change to English.

您好！我现在正在使用您的预训练模型，文件下载自 https://huggingface.co/ValkyriaLenneth/longformer_zh 。我直接使用AutoTokenizer的话，代码会自动调用LongformerTokenizer，然后会报如下错误：

Traceback (most recent call last):
  File "mypath/trylongformerzh1.py", line 3, in <module>
    tokenizer = LongformerTokenizer.from_pretrained("pretrain_path/longformer_zh")
  File "virtualenv_path/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1744, in from_pretrained
    return cls._from_pretrained(
  File "virtualenv_path/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1872, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "virtualenv_path/lib/python3.8/site-packages/transformers/models/roberta/tokenization_roberta.py", line 159, in __init__
    super().__init__(
  File "virtualenv_path/lib/python3.8/site-packages/transformers/models/gpt2/tokenization_gpt2.py", line 179, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType

我看到您的代码中使用的是BertTokenizerFast，所以请问加载longformer_zh的tokenizer是否也需要使用BertTokenizer？
我直接用BertTokenizer确实是可以运行的。

另：我是使用transformers.LongformerModel.from_pretrained来加载您的模型。我暂时没有测试其他功能，直接加载模型似乎是可行的。

我的transformers版本是4.12.5，我能够运行成功的代码是：

from transformers import BertTokenizer, LongformerModel

tokenizer = BertTokenizer.from_pretrained("pretrain_path/longformer_zh")

model = LongformerModel.from_pretrained("pretrain_path/longformer_zh")

如果您有时间浏览本issue的话，我会非常感谢！