atnlp / torchtext-summary Goto Github PK

在language model中，看到要加载word2vec.6B.100d这个预训练模型，我使用的是glove.6B.50d，但是会报错。求解

Traceback (most recent call last):
File "D:/DesktopBackup/right/MLHomework/AllenNLP/[NLP]Pytorch17_torchTextDemo.py", line 75, in
wvmodel = gensim.models.KeyedVectors.load_word2vec_format(r'D:\DesktopBackup\right\MLHomework\AllenNLP\data\glove.6B.50d.txt', binary=False, encoding='utf-8')
File "C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\keyedvectors.py", line 1476, in load_word2vec_format
limit=limit, datatype=datatype)
File "C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\utils_any2vec.py", line 344, in _load_word2vec_format
vocab_size, vector_size = (int(x) for x in header.split()) # throws for invalid file format
File "C:\ProgramData\Anaconda3\lib\site-packages\gensim\models\utils_any2vec.py", line 344, in
vocab_size, vector_size = (int(x) for x in header.split()) # throws for invalid file format
ValueError: invalid literal for int() with base 10: 'the'

weight[index, :] = torch.from_numpy(wvmodel.get_vector(idx_to_word[word_to_idx[wvmodel.index2word[i]]]))
高版本会报错（如101版）
TypeError: expected np.ndarray (got numpy.ndarray)
将torch.from_numpy()改为torch.Tensor()即可
建议注明

您好，请问我构造了连个数据集
examples = []
fields = [('id', ID_FIELD), ('content', TEXT_FIELD)]
for que_id, content in question.content.items():
example_list = [que_id, content]
example = torchtext.data.Example.fromlist(example_list, fields)
examples.append(example)
question_dataset = torchtext.data.Dataset(examples, fields)
examples = []
fields = [('id', ID_FIELD), ('content', TEXT_FIELD)]
for ans_id, content in answer.content.items():
example_list = [ans_id, content]
example = torchtext.data.Example.fromlist(example_list, fields)
examples.append(example)
answer_dataset = torchtext.data.Dataset(examples, fields)
当我TEXT_FIELD.build_vocab时
#TEXT_FIELD.build_vocab(question_dataset, vectors=pre_vectors)（通过了）
TEXT_FIELD.build_vocab(answer_dataset, vectors=pre_vectors)报错
TypeError Traceback (most recent call last)
in ()
1 #TEXT_FIELD.build_vocab(question_dataset, vectors=pre_vectors)
----> 2 TEXT_FIELD.build_vocab(answer_dataset, vectors=pre_vectors)
3 vocab = TEXT_FIELD.vocab # 词表
4 vectors = TEXT_FIELD.vocab.vectors # 预训练的词向量

C:\ProgramData\Anaconda3\lib\site-packages\torchtext\data\field.py in build_vocab(self, *args, **kwargs)
302 counter.update(x)
303 except TypeError:
--> 304 counter.update(chain.from_iterable(x))
305 specials = list(OrderedDict.fromkeys(
306 tok for tok in [self.unk_token, self.pad_token, self.init_token,

TypeError: 'float' object is not iterable
请问这是什么原因，question和answer的example_list 格式都是一样的，万分感谢

请问torchText中不同field的词向量是共享的吗？似乎要分别build_vocab才行，但是为什么不设置成共享的词向量呢