Comments (3)
ymcui
commented on June 16, 2024
这个目录里面的并不是严格意义上的RoBERTa，因为模型是从BERT做初始化，会保留CLS那部分的权重。
from chinese-bert-wwm.
wangruichens
commented on June 16, 2024
谢谢解释。想再请教一下，单纯讲RoBERTa， 为什么它也能在分类任务上跑出高分呢？ 我看原paper里没有提到关于 [CLS] 的训练方法，如何得到它们的embedding呢？
from chinese-bert-wwm.
wangruichens
commented on June 16, 2024
想通了，因为attention机制的存在,CLS本身并不需要训练。
from chinese-bert-wwm.
Related Issues (20)
- 预训练阶段的loss问题 HOT 2
- 请问一下训练用了几块tpu呀 HOT 2
- 预训练分词是会用到中文vocab后面带“##”的token吗？如果是，整词掩码预训练出来的语言模型，用在下游任务中，下游任务可以不分词吗？ HOT 4
- 有点晕：『开源版本不包含MLM任务的权重』,这句话是什么意思？ HOT 3
- RoBERTa-wwm-ext-large应用到全新领域不收敛 HOT 4
- How to download chinese-roberta-wwm-ext.pt ? HOT 1
- CJRC有预训练的模型么？ HOT 1
- How WWM stratege works in code HOT 2
- 链接失效求助 HOT 2
- RoBERTa-wwm-ext-large ft的时候loss飞了 HOT 2
- 继续预训练 HOT 2
- 如何抽取特定layer的词向量？ HOT 2
- 预训练数据 HOT 2
- 请问可否使用wwm系列的模型做词级别的fill-mask预测？ HOT 2
- NER问题 HOT 2
- Is there any sharing about phoneme-BERT pretrained? HOT 2
- Is onnx model available HOT 2
- 请问每个模型推理时需要多大的显存去加载？ HOT 1
- 请问pytorch版本的chinese-roberta-wwm-ext-large在哪里下载稳定而且快？huggingface上太慢了还总断。 HOT 1
- 从BERT换成此模型后跑不通，应该怎么解决，显示缺少参数 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
这个初始token是如何得到的呢？from chinese-bert-wwm.