separius / awesome-sentence-embedding Goto Github PK

Hello @Separius ,
We've leveraged SIF and AraVec for computing sentence embeddings for Arabic, which we termed it as AraSIF. The paper is yet to appear in official ACL-2019 proceedings. However, please feel free to add it to this list. You can find the bibtex in AraSIF repo.

Thanks!

ul and li tags are not rendered in the github page, we should somehow put unordered lists in table cells with pure markdown(using 2 or more spaces didn't work)

https://github.com/facebookresearch/StarSpace

I recently found a recent sentence embedding model that isn't on this list. If you think it's interesting, it might be worthwhile to include it 🙂

• Name: LaBSE
• Blog Post: https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html
• Summary: For LaBSE, we leverage recent advances on language model pre-training, including MLM and TLM, on a BERT-like architecture and follow this with fine-tuning on a translation ranking task. A 12-layer transformer with a 500k token vocabulary pre-trained using MLM and TLM on 109 languages is used to increase the model and vocabulary coverage. The resulting LaBSE model offers extended support to 109 languages in a single model.
• Pretrained Model: https://tfhub.dev/google/LaBSE/1

Thanks for creating this resource! 😄

add this when they released their codes

Where do you see that the code linked to this paper is indeed a implementation of the paper? I can't see anything of this in the LASER repo of facebookresearch.
Can you help me? I'm just searching that code.
Thanks for your great job!

make sure to add XLM-R

I just found a recent sentence embedding model that doesn't show up on this list. If you think it's interesting enough, it might make sense to include here 🙂

• Name: BERT-Flow
• Paper: https://arxiv.org/abs/2011.05864
• Summary (from abstract): Transforms anisotropic sentence embedding distribution from BERT to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective. Experimental results show that our proposed BERT-flow method obtains significant performance gains over the state-of-the-art sentence embeddings on a variety of semantic textual similarity tasks.
• Code: https://github.com/bohanli/BERT-flow

Hi there,

Not sure if this list is still being maintained but if so, might I (shamelessly) recommend adding DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations. It is an unsupervised method for learning high-quality sentence embeddings that we recently developed. It is similar to Sentence Transformers in that it pre-trains a transformer-based language model, but because it is unsupervised, you do not need any labels!

• GitHub.
• Paper.
• Pretrained models: small and base.

add this when the code is ready.

I'm curious how CLIP performs when treated simply as a sentence embedding. Is it competitive?

• adding date is easy for arxiv papers and hard for the others
• citation count seems easy via scholar.py (it's as easy as python scholar.py -t -p "title")

add InferLite when it has codes

Hi Separius,
As you have described how the sentence embedding work, I have some questions about applying a model into the framework. For example, for doc2vec, what is the encoder to generate contextualized embeddings, and what is the pooling method for those embeddings to build the sentence embedding? Also, for CNN encoder, what output of the encoder can be viewed as the contextualized embeddings? If the framework does not apply for these models, may I ask why ?
Thanks.

include magnitude somewhere

make sure to add ELECTRA

Great work and would you mind to add https://github.com/hanxiao/bert-as-service/ into the list?

add this when they released their code

add MT-DNN when they released their codes