shark-nlp / cont Goto Github PK
View Code? Open in Web Editor NEW[NeurIPS'22 Spotlight] Data and code for our paper CoNT: Contrastive Neural Text Generation
Home Page: https://arxiv.org/pdf/2205.14690.pdf
[NeurIPS'22 Spotlight] Data and code for our paper CoNT: Contrastive Neural Text Generation
Home Page: https://arxiv.org/pdf/2205.14690.pdf
Really interesting paper! I was wondering, when will the code be made available for it?
我看到代码里有
def form_ngram(self, input_tensor, n=2)
def torch_bleu(self, ref_tensor, sys_tensor, pad_id, n_gram=2):
这俩个函数来计算bleu score.请问为什么不使用nltk这种可以计算bleu的三方库而选择自己实现呢?请问下主要的考量是什么?
Hi,
While playing around, I have found an error in the function torch_bleu
which is used to rank batch-negatives and beam-positives.
In model.model.CoNTGenerator.torch_bleu
(line 47-70),
there is an severe mistake which results in wrong BLEU scores, certainly when n_gram == 1
and possibly when n_gram >= 2
(rare case where token indices are propotional; i.e. 2-gram [4, 8] and [34, 68]).
Current line 66-67:
sim_matrix = torch.cosine_similarity(input_tensor2_4gram.unsqueeze(3), input_tensor1_4gram.unsqueeze(2),
dim=-1) >= 1.0
Suggestion:
sim_matrix = torch.norm( # Calculate L2 norm to find if N-gram in `sys`` is present in `ref``
input_tensor2_4gram.unsqueeze(3) - input_tensor1_4gram.unsqueeze(2),
p=2,
dim=-1
) == 0.0
您好,我正打算使用CoNT进行文本摘要相关的对比学习,数据格式是CSV,两列内均为str类型文本。经过对代码进行一定修改后,在process_data函数内遇到了一些问题,包括:
fastNLP.core.collators.padders.exceptions.EleDtypeUnsupportedError: Fail to get padder for field:target_outp.
TorchNumberPadderonly supports padding python numbers or numpy numbers or torch.Tensor but get
<class 'str'>. To view more information please set logger's level to DEBUG.
是否可以提供一下数据格式的说明,以便更好的应用代码?感谢您的贡献。
File "xxx/Consistency_model/CoNT/model/model.py", line 192, in generate encoder_feature = self.affine_transformation(encoder_hidden_states, attention_mask) # batch x h File "/xxx/Consistency_model/CoNT/model/model.py", line 103, in affine_transformation trans_tmp = trans_tmp * padding_mask.unsqueeze(-1).float() RuntimeError: The size of tensor a (128) must match the size of tensor b (16) at non-singleton dimension 0
使用的模型为fnlp/cpt-large
,在运行训练时,如果在训练过程中需要eval,则会出现如上的错误,请问是什么问题?
As described in the article 2205.14690: "The feature representations come from pooling the output of the encoder (source sequence) or decoder (target sequence) ." However, the Transformer decoders contain cross attention modules, wouldn't this lead to the information leakage of the target sequence feature representation?
您好,很有趣的工作!
我想请问一下,CoNT-main/model/model.py在CoNTGenerator类的forward函数中
distance_mask = (actual_distance < 0.48)
这一行代码是什么意思?0.48是超参数吗?
感谢您的回复!
你好,对您的工作十分感兴趣,可以告知下关于运行配置和训练时长方面的问题吗
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.